I was contracted to lead the modernization effort of an outdated health claims data aggregation application built using .NET that has become increasingly difficult to maintain and support. The application had been in use for several years and had undergone multiple updates and patches, leading to a brittle, complex and convoluted codebase. The technology stack used in the application was also outdated, making it difficult to integrate with other modern tools and technologies. As a result, the application's performance had declined, and it was becoming increasingly challenging to fix bugs and add new features. In this post, I will take you through the process of modernizing a .NET application and share my experience of leading a .NET modernization effort. As this post is more detailed than other project overviews, I will provider links to the three main sections:

1. Background

2. Primary Issue and Fixes

3. The Next Phase

Claims Aggregator

The claims Aggregator is a system designed to streamline and consolidate the process of managing healthcare claims from various source systems within the enterprise. This was a complex healthcare landscape where multiple claims systems were used, it included COTS software, homegrown applications, applications acquired through the company’s various acquisitions, as well as interfaces to outside 3rd parties. By acting as a centralized hub, the application collects, processes, and organizes claims data from different sources, providing users with a unified view of customer information. The Aggregator interfaced with systems in numerous ways - from real time API’s to batch processing.

Once the claims data is collected, the Aggregator performs data normalization and harmonization, ensuring consistency and uniformity of the data regardless of the system of record. It maps and translates different data formats, codes, and terminology used by the various claims systems, enabling a unified view of customer information. This unified view allows users, such as employees and members, to access a comprehensive and accurate representation of the claims data, irrespective of the source system. By eliminating the need to navigate through multiple systems, they system greatly simplifies the process of managing and analyzing claims, ultimately improving efficiency and decision-making.

The application had been developed using older versions of .NET, ASP and C#, which limited its capabilities and hindered its performance. It was running primarily on .NET Framework 4.0 and utilizing C# 4.0, missing out on the advancements and optimizations introduced in subsequent versions. The decision was made to upgrading to a more recent version, such as .NET Framework 4.8 and transition full to .NET Core, this would provide numerous benefits, including improved performance, enhanced security features, and access to a wider range of libraries and frameworks. The code was developed by numerous teams over the years so it drifted from the original architecture and lacked a consistency in implementation.

Project Objectives

The goals of the project were focused on improving the overall performance and stability of the existing system with the ultimate objective of getting on a modern platform. The first priority was to stabilize the system by clearing critical defects that were causing disruptions in the workflow. Once these defects were addressed, the team focused on improving the system's performance by identifying areas where it could be optimized for better speed and efficiency. We had to carefulhly consider the tradeoff between the time to delivery of the shorter term fixes and the ultimate objective of fully modernizing the platform. Additionally, the team began designing a new modern web app that would be compatible with mobile devices, providing a more user-friendly experience for customers. This new web app was designed with scalability in mind, allowing for future expansion and growth. Finally, the team prepared for system migration, ensuring that all data and systems were properly transferred to the new platform. Overall, the project was a success in meeting its goals, resulting in a more stable, efficient, and user-friendly system for all stakeholders involved.

The First Steps

The initial steps taken to address the project's objectives involved gathering extensive data to determine the root causes of the issues at hand. This data-driven approach aimed to identify the underlying factors responsible for the system's instability and critical defects. By analyzing various metrics, error logs, user feedback, and performance reports, the project team gained valuable insights into the specific areas that required immediate attention. To accurately assess the expectations and define success for the existing system, gathering user feedback played a crucial role. By reaching out to the user community, we endevored to understand their experiences, pain points, and desired outcomes. This feedback provided valuable insights into their expectations, history of issues with the system and helped shape the definition of success for the project. By actively listening to users and incorporating their perspectives, we ensured that the modernization efforts aligned with their needs and delivered tangible improvements. Ultimately, this user-centric approach fostered a more customer-oriented system that could meet and exceed their expectations, resulting in a successful outcome for the project.

Once the data was gathered and analyzed, the next crucial step was to prioritize the issues based on their severity and impact on the system. By assigning priority levels, the team could allocate resources and focus on resolving the most critical issues first, ensuring that the most impactful problems were addressed promptly. This approach enabled the team to efficiently utilize their time and effort, maximizing the effectiveness of the project.

In summary, the project's initial steps involved a thorough data analysis to determine the root causes of the system's issues. This analysis enabled the team to prioritize the problems and allocate resources accordingly. Additionally, assembling the right teams with the required skills and knowledge ensured that the appropriate expertise was applied to each issue. These steps laid a solid foundation for effectively addressing the project's goals and moving towards system stabilization and improvement.

Organization and System Performance

Once we collected and analyzed the the data it pointed us to two main problems: an organizational problem and a major performance problem.

The organization problem centered around work allocation, communication to the user community, and defining/communicating service levels.

Firstly, improving work allocation, it was essential to assemble the right teams with the necessary expertise to tackle each identified issue. Different problems often require different skill sets and knowledge areas. By carefully selecting and assigning the appropriate teams to specific issues, the project aimed to ensure that the individuals working on each problem had the relevant expertise and experience to provide effective solutions. I had to bring in external specialized talent for the performance related problems - this consisted of a small SWAT team I worked with in the past. We dug in quickly and produced results. This approach promoted efficiency, collaboration, and specialization, allowing teams to work in parallel on multiple fronts. Additionally, defect work was not getting assigned to correct groups in a timely fashion. For example, defects were assigned to the Aggregator team when it was a defect in the upstream system - tickets could languish in this state leading to poor customer satisfaction.

Secondly, effective communication with the user community is vital for building trust and ensuring customer satisfaction. Establishing clear communication channels, such as regular newsletters, dedicated forums, or a robust ticketing system, allows organizations to proactively update users about any issues, fixes, or system updates. Timely and transparent communication regarding problem resolutions and anticipated timeframes demonstrates a commitment to customer service. Additionally, soliciting user feedback and actively addressing concerns creates a sense of partnership, enabling organizations to continuously improve their products or services based on user needs and expectations.

Lastly, defining Service Level Agreements (SLAs) is essential for setting clear expectations regarding response times, issue resolution, and system availability. We discovered that there was never an agreed upon definition of acceptable performance or a fact based discussion of the trade-offs involved. The result was, users of the system were operating under one set of assumptions and the technical community under a different set. We collaborated with stakeholders to define realistic SLAs that align with their requirements, the capabilities of all systems involved and available resources. Clearly outlining the scope of support, communication channels, escalation procedures, and target resolution times in SLAs helps manage user expectations and provides a framework for measuring performance. Regularly monitoring SLA compliance and actively addressing any deviations allows organizations to identify areas for improvement and enhance their service delivery to meet or exceed user expectations.

Fixing the organizational issues did two things for the teams - we were able to allocate and fix most of the non-performance related defects and bought us some time to fix the more difficult issue of system performance.

System Performance

Our data analysis clearly pointed to three primary performance related issues: memory management, data access/query optimization and C# code optimization.

Memory Management:

Efficient memory allocation and deallocation plays a pivotal role in the performance of any C# .NET application. Prior development teams didn’t give deep enough thought to memory management as the app grew in size and usage. Garbage Collection (GC), while automatic and convenient, introduced performance costs in overhead due to its non-deterministic nature. By analyzing performance profiles and leveraging tools like the .NET Memory Profiler, performance counters and PerfMon, trace tools, and BenchmarkDotNet (to start building a benchmark history) we discovered that excessive memory allocations and frequent garbage collections were slowing down our application.

By optimizing memory usage, we achieved remarkable results. By reducing the number of temporary objects created, we managed to cut down garbage collection cycles by 30%. This led to a 20% improvement in overall application responsiveness, resulting in faster load times and smoother user interactions. Furthermore, adopting object pooling techniques for frequently used objects reduced memory churn and significantly enhanced our application's scalability, enabling it to handle more concurrent requests without sacrificing performance.

API Calls to External Systems

Another issue with the application was its inefficient handling of API calls to external systems. The code had multiple hard-coded endpoints and lacked a centralized and reusable approach for making API requests. This led to redundant code and increased maintenance efforts. By implementing a modern API integration strategy, utilizing Microsoft’s recommended approach for using IHttpClientFactory, dependency injection, and proper encapsulation of API calls, the team significantly enhanced the application's efficiency and maintainability. Additionally, incorporating industry-standard practices like caching, asynchronous programming, and implementing appropriate error handling mechanisms would further optimized the application's interaction with external systems.

Database Access and Queries:

Database interactions were causing performance challenges for the application. This was an area where leveraging my network of specialist talent in SQLServer paid major dividends. Slow queries, inefficient data retrieval, and inadequate caching mechanisms were impacting the responsiveness of the application. Through careful analysis of our Aggregators performance metrics, we discovered that our database access layer was a major bottleneck.

To address this, we focused on optimizing the database queries. By utilizing proper indexing techniques, reducing unnecessary joins, and caching frequently accessed data, we managed to reduce query execution time by 40%. This optimization resulted in a 15% decrease in average response time, allowing the aggregator app to easily handle higher traffic loads. Furthermore, leveraging tools like the Entity Framework Profiler helped us identify and eliminate N+1 query issues, by implementing eager loading, further improving the efficiency of our data access layer.

Furthermore, the application exhibited poor use of stored procedures. Many business logic operations were being performed directly within the application's C# code instead of leveraging the power of stored procedures. This resulted in scattered and duplicated code across different layers, making it challenging to maintain and update. By refactoring the code to utilize stored procedures effectively, we could centralize the business logic within the database, improving performance, data integrity, and maintainability. Leveraging SQL Server features like stored procedure parameters, transactions, and optimized query execution plans resulted in streamlined database operations and reduced network overhead as well.

Code Optimization:

In addition to memory management and database access, suboptimal coding practices caused two problems - slowed the delivery of new product features and system performance suffered. Poorly written algorithms, excessive resource usage, unnecessary computations, poorly organized and redundant code all impacted the application’s and development team’s performance. We implemented performance profiling tools including Visual Studio Profiler, as we refactored the code base.

Through code refactoring and algorithmic improvements, we achieved significant performance gains. Our analysis identified several hotspots within the codebase, where we optimized critical sections by utilizing more efficient algorithms and data structures. This led to a 25% reduction in overall CPU usage, resulting in a 30% improvement in application throughput. Additionally, by adopting asynchronous programming patterns where possible, we were able to eliminate blocking operations, providing a more responsive and scalable application experience.

By addressing memory management, database access, and code optimization, we significantly improved the applications and the development teams performance.

Next Steps

The work performed above greatly simplified the claims aggregation process, fixed the performance issues and eliminated the backlog of enhancements/bug fixes. It also paved the way for a pivot in the evolution of the system. The data aggregation was the most valuable process within the application so next steps were focused on moving front end access to a COTS system using the database of aggregated claims data as a source. We began planning for the following:

• Selection of and migration to a COTS package

• Claims systems consolidation

• Streamlining RESTful API access by eliminating the use of OData and moving to GraphQL