Is there a point in having a monorepo if you're all in on the microservices approach? I'm a big microservices skeptic, but as far as I understand it, the benefit of microservices is independence of change & deployment enforced by solid API contracts—don't you give that all up when you use a monorepo? What does "Monorepo with microservices" give you that a normal monolithic backend doesn't?
(Obviously e.g. an image resizer or something else completely decoupled from your business logic should be a separate service / repo anyway—my point is more along the lines of "If something shares code, shouldn't it share a deployment strategy?")
> Is there a point in having a monorepo if you're all in on the microservices approach?
Monorepos are excellent for microservices.
- You can update the protobuf service graph (all strongly typed) easily and make sure all the code changes are compatible. You still have to release in a sensible order to make sure the APIs are talking in an expected way, but this at least ensures that the code agrees.
- You can address library vulns and upgrades all at once for everything. Everything can get the new gRPC release at the same time. Instead of having app owners be on the hook for this, a central team can manage these important upgrades and provide assistance / pairing for only the most complex situations.
- If you're the one working on a very large library migration, you can rebase daily against the entire fleet of microservices and not manage N-many code changes in N-many repos. This makes huge efforts much easier. Bonus: you can land incrementally across everything.
- If you're the one scoping out one of these "big changes", you can statically find all of the code you'll impact or need to understand. This is such an amazing win. No more hunting for repos and grepping for code in hundreds of undiscovered places.
- Once a vuln is fixed, you can tell all apps to deploy after SHA X to fix VULN Y. This is such an easy thing for app owners to do.
- You can collect common service library code in a central place (eg. internal Guava, auth tools, i18n, etc). Such packages are easy to share and reuse. All of your internal code is "vendored" essentially, but you can choose to depend on only the things you need. A monorepo only feels heavy if you depend on all the things (or your tooling doesn't support git or build operations - you seriously have to staff a monorepo team).
- Other teams can easily discover and read your code.
Monorepos are the best possible way to go as long as you have the tooling to support it. They fall over and become a burden if they're not seriously staffed. When they work, they really work.
Should have also mentioned: all those changes to cross cutting library code will trigger builds and tests of the dependent services. You can find out at once what breaks. It's a superpower.
None of this addresses my question—what benefits do you get from having monorepo-with-microservices over a monolithic backend? All of the things you mentioned would be even easier with a monolithic backend.
- Monoliths force you to coordinate deploys, which can be a pain. It's difficult to verify your changes in concert with others.
- Bugs from one team can cause a rollback for everyone.
- One N+1 query, nonlinear algorithm, or GC-contentious endpoint can send the entire system into a spiral and cause non-linear behavior.
- You don't need to scale everything the same (services, databases, etc.) Your authentication pieces need massive scale: 10,000 QPS in an active-active configuration. Your random endpoint for the "XYZ promotion" does 0.5 QPM.
- Engineers that build platforms can operate in Java/Go/Rust, engineers that build product can build in Ruby/Python/Node.
- Payment data growing faster than user data? Scale your payment pieces. Shard their databases. They're safely cordoned off and isolated from anything else. The payments team(s) can focus on this one really hard problem without anyone else knowing or caring at all.
- Engineers can have privileged access to the resources their team consumes and needs. You don't want your customer loyalty team having access to payment team data. Or your appointments team having access to user accounts and authentication.
- You can hire for teams/concerns, and you have an interesting metric: number of services per engineer or team. It's actually pretty useful.
- Mistakes in one corner can sometimes be completely isolated and not take down the entire company. This isn't always the case. Critical infra (auth, service discovery, networking) can take everything out. But your appointments or customer success app won't have any impact to the other business areas. Also, it's great to have those super mission-critical folks focused on only the extremely important function they serve.
- Tech debt can live "under the covers". External teams calling your services might not know about the mess or the ongoing migrations under the hood. They don't need to know. (Granted, if your data model at scale sucks, this doesn't really afford much. But it is an easier puzzle to decompose.)
tl;dr - teams are focused on things they own, scope is controlled, blast radius is limited, pieces scale independently as needed.
They solve different problems, some of which may overlap I suppose.
For one thing you get clear ownership of deployed code. There isn't one monolithic service that everyone is responsible for babying, every team can baby their own, even if they all share libraries and whatnot.
You also get things like fault isolation and fine grained scaling too.
Monorepo with microservices gives you the ability to scale and perform SRE-type maintenance at a granular level. Teams maintain responsibility for their service, but are more easily able to refactor, share code, pull dependencies like GraphQL schemas into the frontend, etc. across many services.
So basically each team has to reinvent devops from the ground up, and staff their own on call rotation, instead of having a centralized devops function that provides a stable platform? That sounds horrendous.
Although, that said, I can at least see the benefits of the "1 service per team" methodology, where you have a dedicated team that's independently responsible for updating their service. I'm more used to associating "microservices" with the model where a single team is managing 5 or 6 interacting services, and the benefits there seem much smaller.
Different teams can make their own decisions, but as a developer on a team that ran our own SRE, I found it came with many advantages. Specifically, we saw very little downtime, and when outages did occur were very prepared to fix it as we knew the exact state of our services (code, infrastructure, recent changes and deploys.) Additionally, we had very good logging and metrics because we knew what we'd want to have in the event of a problem.
And I'm not sure what you mean "from the ground up." We were able to share a lot of Ansible playbooks, frameworks, and our entire observability stack across all teams.
But I think you may also be missing the rest of my post. This is only one possible advantage. Even if the team doesn't perform their own SRE, these services can be scaled independently - both in terms of infrastructure and codebase - even while sharing code (including things like protocol data structures, auth schemes, etc.)
A service that receives 1 SAML Response from an IdP (identity provider) per day per user may not need the same resources as a dashboard that exposes all SP (service providers) to a user many times a day. And an administration panel for this has still different needs.
Yet, all of these services may communicate with each other.
Yeah, I've seen it used to allow teams to use consistent frameworks and libraries across many different microservices. Think of authentication, DB clients, logging, webservers, grpc/http service front ends, uptime oracle -- there's lots of cross cutting concerns that are shared code among many microservices.
So the next thing you decide to do is create some microservice framework that bundles all that stuff in and allow your microservice team to write some business logic on top. But now 99% of your executables are in this microservice framework that everyone is using, and that's the point where a lot of companies go the monorepo route.
Actually most companies do some mix -- have a lot of stuff in a big repo and then other smaller repos alongside that.
(Obviously e.g. an image resizer or something else completely decoupled from your business logic should be a separate service / repo anyway—my point is more along the lines of "If something shares code, shouldn't it share a deployment strategy?")