DevOps and System Administration Blogs
Explore DevOps and system administration blog posts. Stay current with best practices, tools, and insights into efficient IT management strategies.
cancel
Showing results for 
Search instead for 
Did you mean: 
eliyahuniv
Associate
Associate

microservices! the promise of scalability, independence, and a developer's dream. But what happens when that dream turns into a nightmare? That's exactly what happened to us a year and a half ago. As our product grew, so did our repositories, leading to a tangled web of dependencies that felt like hacking through a jungle.

A bit about the situation we were in

We started small, with just a couple of repositories for our microservices and libraries. However, as our library collection grew, so did the number of repos. Soon, we found ourselves in a dependency maze.

Look at these dependencies tree!

eliyahuniv_0-1716064644474.png

*All the services were updating the same schemas of the database. While we had other microservices updating different schemas of the database, they weren't relevant to this story.

Here’s where the fun (and frustration) began:

  • Branch Tree updating: Imagine working on a feature in a library buried deep in the dependency tree. Every change required a domino effect of updates across all the service branches that depended on it.
  • Leaf Level updating: Sometimes, a change in a common library (not has to be an infrastructure or deep library, even a logic library) needed to be reflected across multiple services – online services, offline services, the whole gang. Updating them all felt like an endless game. feature after feature.
  • Build errors: Skipping leaf-level updates for some services might seem tempting, but it came back to haunt us later. Developers would encounter mysterious build errors stemming from changes made weeks or months ago, with no clear path to a fix. Not exactly a recipe for developer happiness.
  • E2E Earthquake: And then there were the E2E tests. Updating a service late in the game could trigger cascading failures, leaving us scrambling to identify the culprit. It could take days of detective work, involving multiple developers and a deep dive into commit history. All this while, our shiny new service remained unreleased, gathering dust.
  • Deployments- Fixing a bug or adding new logic in a common library required deploying all dependent services together. This meant keeping potentially buggy code in production environments until all services are deployed.

We knew something had to change. We needed a solution that streamlined development, fostered collaboration, and didn't require spelunking through a code cavern.

Exploring Solutions

We didn't just dive headfirst into a mono repo. We meticulously evaluated our options:

  • Automatic merge request for updating dependencies: This involved the CI/CD automatically opening merge requests in dependent applications/libraries whenever new code was merged into a library. While easy and fast to implement, it wasn't atomic, required manual intervention, and left some issues like build errors and the E2E unaddressed.
  • Common logic libraries can work as-a-service: This classic approach is aligned with microservice principles and addressed most of our issues. However, it meant a significant architectural change, which we weren't keen on due to potential performance drawbacks for our big data processing applications.
  •  Mono Repo: All relevant code in single repository - this seems as the most promising solution, potentially fixing all our issues without altering our product architecture.

Mono Repo

We opted for the mono repo to achieve a fully automated approach that addressed our pain points.

We wanted to put together all the libraries and apps that connected each other in one place to achieve benefits in the development flow:

  • Instantaneous Impact: Updates to a library are instantly reflected across all dependent branches and services. No more time-wasting branch updates, just a faster, more efficient workflow.
  • Build Error Bonanza: Identifying and fixing build errors becomes a breeze. Developers can pinpoint issues right away, preventing them from becoming time bombs down the line.
  • E2E Efficiency: Our CI/CD pipeline acts as a safety net for E2E tests. Any change that breaks a service's E2E test simply won't get merged. This saves us from those multi-day debugging marathons and keeps our releases rolling smoothly.

Implementing the Mono Repo

Initially, we explored tools like Bazel that could identify changes in specific apps/libraries upon every merge and trigger independent CI/CD pipelines for them. This approach, while maintaining separate pipelines, would still utilize the single codebase of a mono-repo.

However, this approach had its challenges:

  1. Version Visibility: Maintaining separate version tags for each application within the mono-repo would make it difficult to ensure all components were aligned and in sync.
  2. Interdependency Woes: Independent pipelines wouldn’t capture the need for a unified installation process, which was crucial for our core common logic library. we can’t deploy a fix for a bug in one service and leave unfix the same bug in other service. (We weren't aiming to migrate all repositories into the mono-repo, only the tightly coupled applications and libraries that formed the tangled dependency tree within our team that are running the main logic of the product).

What we learn? we need an atomic installation and single version.

These considerations led us to the Helm umbrella chart pattern, which leverages Helm's dependency feature (https://helm.sh/docs/helm/helm_dependency/). In essence, this pattern allows us to create a single installation command for all applications within the mono-repo.

Here's a breakdown of the key technical details:

  • Unified Codebase: We consolidated all relevant libraries and services into a single, well-organized Git repository, forming the foundation of the mono-repo.
  • Centralized CI/CD Pipeline: A single CI/CD pipeline was implemented to govern the entire mono-repo. This pipeline streamlines the development process by managing builds, tests, and deployments for all interconnected components.
  • Helm Umbrella Chart: We utilized Helm's dependency feature to define the relationships between the various applications within the mono-repo. This umbrella chart acts as a master control panel, specifying the individual charts (applications) and their dependencies. When deploying the umbrella chart, Helm automatically resolves all dependencies and installs the applications in the desired order.
  • Java Build Tool Integration: For building and managing our Java-based applications within the mono-repo, we leveraged sbt (Simple Build Tool) (https://www.scala-sbt.org/1.x/docs/Multi-Project.html), a popular build tool for Scala and Java projects that also excels in handling multi-project builds. This integration ensures efficient compilation, testing, and packaging of all our Java libraries and services within the mono-repo framework.

This approach ensures a single version/tag for the entire codebase and facilitates a streamlined single-command installation process, all while maintaining our microservice architecture. The changes are primarily confined to the development and CI/CD phases; the production environment continues to leverage independent, containerized microservices running on Kubernetes.

just to feel a bit how our code base is now looks like:

eliyahuniv_1-1716064644488.png

you can see that we have

  • apps folder – where we put all the services.
  • libs folder – where we put all the libraries.
  • Helm folder - a folder with the same name of the project name with all the helm files. The main Chart.yaml configuring the sub charts that will be installed when we install the main Chart.yaml that under the charts folder.

Hold on, isn't this a monolith?

Not quite! Unlike a monolith, where everything runs in a single service, our microservices remain independent and containerized, running separately on Kubernetes in production. The mono-repo solution only affects the development and CI/CD phases, bringing all the tightly coupled services and libraries together for a smoother development experience. It's like having all your tools neatly organized in one toolbox, but each tool is still used independently for its specific job.

Results and Learning

Transitioning to a mono-repo wasn't without its challenges, but the impact on our team has been undeniably positive. Streamlined workflows and instant dependency updates have fostered a more efficient development process. While it's difficult to pinpoint an exact increase in velocity, the anecdotal evidence is clear: developers are demonstrably happier. They spend less time wrestling with dependency issues and have more confidence in the stability of their code thanks to the robust CI/CD pipeline ensuring E2E test success.

By sharing our journey, we hope to offer insights and inspiration to others facing similar challenges.

Labels in this area
Top kudoed authors