Data Design and Microservices

As companies move to a microservices architecture a few areas of data design need to be considered, they are: data sovereignty and database complexity including use of foreign keys and joins. Data sovereignty and its implications on your microservices architecture is sometimes a contentious topic that comes up during the design phase. One of the rules or at least strong guidelines for microservices design is that each microservice owns its domain data and its logic. The services should be loosely coupled - a change in one service does not require a change to another service — and highly cohesive — services that perform a similar function reside together and services that meet a different functional requirement reside elsewhere. Related to loose coupling is data sovereignty, which is a difficult concept to grasp when you are used to dealing with large monolithic systems that have one large data store with foreign key constraints.

Data Sovereignty

Data sovereignty is a design choice where each microservice or a set of services that are highly cohesive own their own database - these is not one large dB as there typically is with a monolithic system and the loss of a “single source of truth” has to be addressed. This is a more difficult design to implement because you have data split across databases, communications costs have to be managed, sometimes a single ACID transaction can not be made across databases (eventual consistency needs to be considered).

Data design for microservices - note the separate data stores for each app area based on functionality.

Queues can be used for async transactions across multiple services.

Foreign Keys & SQL Joins

Foreign keys are another consideration when building out microservices. FK are usually a given in monolithic systems but this is not necessarily so with microservices. I want to be clear here, you definitely can use FK with microservices but its not always a default decision. SQL joins are another area of complexity when using a RDBMS, they are a powerful tool but can be difficult to debug and scale. Microservice teams have often opted to limit SQL joins in favor of a caching mechanism to offload work from the dB.

If you are reading this you already know what foreign keys and a SQL joins are and why they are used so the question is why would you not always employ them with microservices? When teams are very small and operating without a dedicated database staff, simplicity of logical/physical database design is often a priority so constraints are moved out of the dB layer. As discussed above, designing for data sovereignty means encapsulated data and microservices tends to operate with small more nimble teams often times without a full-time traditional DBA role. Scalability is another reason to move away from a more complex dB designs. The dB layer can be the most difficult area to scale so making this less complex and moving the workload to a caching tool is another design option. To improve dB simplicity, ease maintenance, improve performance and scalability, and to lighten the dB load you can do key-value lookups using an in memory data store. An early choice for this was Memcached and remains a viable option for ease of use but Redis seems the better option these days. Redis is feature rich and it’s part of the portfolio in Amazon Web Services and Microsoft Azure making it an easy choice for most cloud deployments. I have seen benchmarks where caching can improve performance time of key-value lookups from from O(n) to O(1).

A microservices architecture does not prevent you from using the dB as you would with a monolithic system; if you do just make sure you have a strong dB staff on the team however if you are making the move to to microservices you should consider all options.

Microservices Tooling

Previously in this three-part series, I wrote about how to introduce microservices in a legacy environment and provided an overview of domain driven design (DDD) and how this development philosophy can be used to represent the real world in code, while also being well-suited to a microservices implementation. This time, I will cover some of the tools and frameworks that can be used when implementing microservices. 

I deliberately saved the tools discussion for last because I often find clients like to jump into tools before they have completely thought through why, and if, a new architectural approach can or should be implemented in their environment. After you have decided that you should move to microservices then it’s appropriate to think about how it can be done and with what tools. 

Containerization

Microservices architectures do not require containerization, but it is one set of tools that will make your life easier on a number of fronts. 

The metaphor of a shipping container is used because through standardization containers allow goods to be transported by different shippers via ship, rail, or truck regardless of items in the container, knowing that the contents of one container will not affect the others. These containers can be stacked and moved easily.  

In software, containerization facilitates the packaging and deployment of services across environments with no modifications. Additional benefits of containers include:

• Application isolation

• Application scalability; containers allow you to easily scale your application services up/down as needed

• Application portability

• Allows for, but does not require, microservices implementation

• Continuous integration/continuous delivery (CI/CD)

As you can imagine, there are tools in the market place that allow you to containerize your applications and manage/orchestrate those containers in your environment and in the cloud.  The three leading products are Docker, Kubernetes, and Mesos, with Docker having become the industry de-facto standard. There are some use cases where one of the other tools may make more sense, such as Mesos for extremely high scalability (e.g. Twitter and Apple).

Developing microservices is only the beginning; the effort and thought necessary for testing, deploying, running, and maintaining services should not be underestimated. To a large extent, the types of tools and techniques you will select will be driven by your existing technical infrastructure. If, for example, you are a heavy user of IBM products then you will be looking to them for solutions. 

I have come to believe that the cost of adopting and developing with new tools and frameworks, or in a new language, outweighs the benefits so you should be very careful if you plan to move from your primary vendor’s ecosystem to new one.  There is, however, a rich set of tools for microservices deployment and maintenance to choose from that can be used to complement or create your DevOps environment. Here’s an overview of some of these tools and frameworks below.

For CI/CD build management and automation the most widely used tool is Jenkins. Jenkins has a rich library of plug-ins that extends Jenkins to almost any external tool in support of the CI/CD process. Jenkins runs on multiple operating systems; Windows, OS X, or Linux. Configuration of Jenkins is not difficult and has strong support in the industry, so hiring staff -- which should always be a consideration -- is manageable.  

There are, of course, other tools. I have worked with companies that are 100 percent Microsoft and wish to stay in that environment whenever possible. Microsoft offers a product called Team Foundation Server (TFS), which also has garnered strong support in the industry. TFS is tailored for Visual Studio IDE but will work with Eclipse and other integrated development environments. In addition to support for the usual CI/CD functionality, it offers integration with Microsoft’s cloud platform and tools such as Azure Service Fabric and Azure API Management (for gateway and portal services). Combining Docker and CI/CD tools can have a profound change on your microservices implementations and deployments. 

Testing of microservices poses some unique challenges due to the distributed and the inter-connected, but loosely coupled nature, of their design. The basic principles of testing should be familiar – in 2009, Mike Cohn descried a testing pyramid that has, with some modifications, been widely adopted. This approach is applicable for microservices testing and the need for automation may even be greater now.  

Here again, vendor frameworks can help. Microsoft Team Foundation Server offers a full suite of testing tools and services that work on Azure. Jenkins, too, comes with microservices test tools. In addition, some specialized testing tools are available, such as Rest-Assured for Java testing of REST services and WebInject for testing.

Lastly, I want to touch on using microservices for front-end UI development.  It is typical for companies to start their journey with server-side microservices.  As you move up the maturity curve it will become apparent that a monolithic UI becomes a bottleneck in your development/delivery process as the front-end becomes more unweidly.  To solve this I have used a composite UI with microservices.  You can develop composite UI with traditional tools such as ASP.net and you can augment your development process with a with services and libraries like Project Mosaic (https://www.mosaic9.org). 

I hope this blog series has helped you gain a better understanding of how to begin your journey to microservices in a legacy environment, how domain-driven design can be used to jumpstart your design efforts, and the tools that are available to facilitate your work.

From Domain Driven Design to Microservices

As many of you may recall, the software design and architecture style known as service-oriented architecture (SOA) emerged in the mid 1990’s. Since then, we have discovered better ways to build systems, including advances in cloud-based virtualization, continuous integration and delivery, and microservices. In the process, these technologies have made SOA and all the associated benefits a reality. 

In March, I published the first post of this three-part blog series – How to Introduce Microservices in a Legacy Environment – explaining how microservices can be introduced into a large organization with well-established legacy systems. In this post, I will cover domain-driven design (DDD) and how this development philosophy can be used to represent the real world in code while being well-suited to a microservices implementation.

Domain-driven design 

 Cohesion is an early tenet of software design and refers to the degree of functional relatedness that exists inside a module or class. In this context, cohesion was first described in the late 1970’s by Tom DeMarco and has come to mean grouping and keeping together those things that change for the same reasons and separate the functionality that changes for different reasons.

DDD provides a method to facilitate the development of highly cohesive systems through bounded contexts. Microservices is an implementation approach that encourages you to focus your service boundaries on the business domain boundaries. DDD and microservices can be used together as you move your organization to a service-oriented design and reap the benefits of continuous integration and delivery.

 The seminal work in DDD was defined in a 2003 book by Eric Evans called Domain-Driven Design: Tackling Complexity in the Heart of Software. The overarching philosophy of DDD is to use the notion of bounded contexts which form protective layers around models that define the business domain. Bounded contexts are analogous to departments in a company – the legal department has certain specific responsibilities (contexts) that are different than the IT department and those responsibilities are enforced by rules (boundaries) for interaction and obtaining services from the departments.  

 This is the same for bounded contexts that we model using DDD. To facilitate a common understanding of the problem domain and translate that domain knowledge into a computer system the business and technical team must develop a common language. In DDD this common language is called the ubiquitous language (UL). As the technical staff develops their models and code they use the UL to decrease the risk of misunderstanding between the business analysts and the engineering staff as the project progresses.

This also serves to provide an additional layer of documentation of the systems and enhances the organization’s understanding of how a system was designed and intended to work. The analysis models that are used to understand and define the domain are tied to the code models that are used to create software by the UL.

Other key principles of DDD include:

• Iterative creation of the analysis and code models. As the team learns more about the domain they iterate on their analysis and code models, keeping both in sync. DDD does not specify tools, databases or languages, but I have used UML (universal modeling language) to create analysis models and my code or implementation models were done in C++ and Java.

• Collaboration of the business and technical teams requires close, face-to-face collaboration to create the relevant models. This is a heavy commitment for all parties to develop the UL, use it to define the domain, iterate through the definition of the domain, and focus on the problem instead of jumping directly to a solution.

• Focus on the core domains – the core domains are those domains that will make the product a success. It is a core domain if it is absolutely essential to the success of the business. You should be asking yourselves how this domain increases revenue, cut costs, or increase efficiency, and why and how this domain is critical to the business.

 The problem you are solving must be substantial. There is no use in implementing DDD for problems that are insignificant, won’t move the needle for the business or are better solved with a COTS (commercial off-the-shelf) solution.

After you have begun to understand the business problem and developed models to define it you will have to think about how to integrate bounded contexts. In their book Enterprise Integration Patterns (Addison Wesley Signature Series) Gregor Hohpe and Bobby Woolf define four integration styles: file transfer, shared database, remote procedure invocation, and messaging. In most applications of substantial size and for reasons of cohesion your DDD and technical team will most likely settle on remote procedure invocation and/or messaging for integration.

From domain-driven design to microservices, pairing these two approaches to solve large and complex problems makes good business sense.

The third and final post in this series is an overview of microservices tooling and can be found here.

Introducing Microservices into a Legacy Environment.

While currently no consensus exists on how to define microservices it’s generally agreed that they are an architectural pattern that is composed of loosely coupled, autonomous, fine-grained services that are independently deployable and communicate using a lightweight mechanism such as HTTP/REST.  Now is the time for companies -- particularly enterprises that need to make frequent changes to their systems and where time to market is paramount -- to be investigating how best to introduce microservices in their legacy environments if they expect to realize a digital transformation that drives tangible business results.

The benefits and potential hurdles associated with adopting microservices are well documented. On the plus side, the modular and independent nature of microservices enables improvements in efficiency, scalability, speed and flexibility. Detractors, however, frequently point to management and security challenges, especially when they pertain to customer-facing applications and services. 

Like virtually all technology decisions, it’s critical to balance risk with reward and, when it comes to microservices, embracing an evolutionary approach and process. After all, lessons can be learned from both success and failure, and the same is true for implementing microservices that can increase product and service quality, ensure systems are more resilient and secure, and drive revenue growth. In the first of this three-part series, I’m going to explain how business and technology leaders can smoothly and successfully introduce microservices in a legacy environment.

It’s all about the monkey

A key requirement of microservices design is to focus service boundaries around application business boundaries. A keen awareness and understanding of service and business boundaries helps right-size services and keeps technology professionals focused on doing one thing and doing it very well. 

In my experience, I’m finding that the larger the organization the greater value microservices architecture can deliver, but only if executed in a systematic, enterprise-wide fashion. Fortune 500 organizations tend to have a significant proliferation of legacy technologies and should strive to simplify deployment, along with applying continuous integration and delivery of microservices. All too often, enterprises focus their efforts on buying tools, implementing a small proof-of-concept or other “quick wins” that likely aren’t the most effective place to initiate microservices strategies.

Astro Teller, the “Captain of Google Moonshots” has a humorous anecdote about where to begin when solving a large and complex problem and advocates that companies should avoid allocating all of their resources on the easy stuff and instead start by addressing the hard problems; he calls it “tackling the monkey first.”  The monkey, when deploying microservices in a large, established environment, is understanding and decomposing the legacy systems.

Decompose the legacy environment by identifying seams

In the second part of this series I’ll cover domain-driven design (DDD), but for now it’s important to understand two concepts found in DDD: bounded contexts and domain models. 

Any problem domain is composed of a number of bounded contexts with models sitting inside them. The bounded contexts provide a level of protection and isolation of the models. In addition, the bounded context provides an interface to the model and controls what information is shared with other bounded contexts. For example, in an e-commerce application some of the bounded contexts may be ordering, pricing, or promotions.   

Years ago I enjoyed reading “Working Effectively with Legacy Code” by Michael Feathers. In his book, the author presented the idea of a seam as a way to identify portions of code that can be modified without affecting the rest of the code base. This notion of seams can be extended as a method to divide a monolithic system into bounded contexts from which services can be quickly and seamlessly created.  

Uncovering seams in applications and building bounded contexts is an important first step in breaking down the monolith. Seam identification can be accomplished by reviewing the current code base, interviewing domain experts, and understanding the organizational structure. A few suggestions:

• Review the current code base. When reviewing the current code base and any artifacts it’s critical to realize that this is only a starting point. The code is often redundant and difficult to understand. 

• Interview domain experts. This is a key step to learning where the seams are and identifying bounded contexts. Having domain experts that understand what the business should be doing not just what the system currently does is critically important.

• Understand the organizational structure – Often, organizational structure will provide clues to where the seams can be found.

Once bounded contexts are identified, along with the programming language and environment that support them, creating packages and sub-packages that contain these bounded contexts should closely follow. This approach will afford a careful analysis of package usage and dependencies, which are paramount to fully and quickly understanding and ensuring that testing and instrumenting code is being done properly.  In addition, there are some standard design patterns that should be followed:

•  Open Host Pattern - Exposing legacy systems via JSON/HTTP service.  Here the isolated legacy system is exposed through an API that returns JSON.

• Anti-Corruption Layer (ACL) Pattern – A translation layer or sometimes called bridge layer is built between the legacy environment and the microservices code.  This pattern can be effective for short durations but it can be costly to maintain over time. We have also called this layer “scaffolding” it’s needed during the transition but will be taken down after the job is done.

 That’s how microservices should be introduced in a legacy environment.

Read the second post in this series here