Shopify’s core monolith has over 2.8 million lines of Ruby code and more than 500,000 commits. This makes it one of the largest Ruby on Rails codebases in the world. Though we spoke about it briefly in an earlier post about different types of Monolithic systems, we are going to take a more detailed look into the development of Shopify’s Modular Monolith approach in this particular post.
The Shopify app has a ton of diverse functionality spanning various domains.
- Billing merchants
- Managing 3rd party development apps
- Updating product information
- Handle shipping processes
Since the beginning, Shopify was built as a monolithic application. In other words, all of the distinct functionalities were within one code-base with no boundaries.
What did this really mean in the context of Shopify’s monolith?
As an example, it meant that the code that handled calculation of shipping rates lived with the code handling checkouts. There was no barrier between the two pieces of functionality in terms of calling each other. Basically, there was a tight coupling between the two different parts of the system.
This led to a bunch of disadvantages that are true for any monolithic application:
- The application had become extremely fragile with any new code causing unexpected repercussions.
- Small changes triggered a cascade of test failures. Moreover, test suites had become quite slow.
- Onboarding new developers had become tougher. A new developer had to know a lot more about the system and the overall context of the monolith in order to become effective. For example, a developer working in the shipping team should know about orders, payments and so on. Everything was intertwined in such a way that the learning curve had become quite steep.
Transforming the Monolith
Out of the box, Rails does not provide special patterns and tooling to manage inherent business complexity.
With the view to make the system more friendly for developers, Shopify formed an internal team to find ways of transforming the monolith.
Some of the goals for this team were as follows:
- Make it easier to onboard new developers to relevant parts of the system.
- Make it possible to run test suites on smaller subset of components affected by some change. Also, make the test suites faster to run.
- Reduce impact of changes and cut down feature implementation time.
Basically, the idea was to make the developers feel like they were working on a smaller application rather than a big monolith.
One of the obvious solutions was a move towards a microservices architecture. However, the Shopify team felt that transforming to a microservices system would create its own set of challenges. Instead, the Shopify team chose to pursue a Modular Monolith approach.
The key features of this approach are:
- All of the code powers a single application.
- Boundaries between various parts of the system are clearly defined.
Basically, Modular Monoliths gave Shopify advantages of both the monoliths and microservices without the downsides associated with either of the approaches.
However, moving in this direction required a number of concrete steps.
Shopify’s Implementation of Modular Monolith
Shopify decided to move towards a Modular Monolith in the year 2017. To achieve this transformation, a team was put together. Shopify’s Modular Monolith project was called ‘Break-Core-Up-Into-Multiple-Pieces’, and was eventually changed to being called ‘Componentization’.
The overall task was broken up into multiple steps as follows:
1 – Code Re-Organization
The first issue to tackle was of code re-organization. Code of the existing monolith was based more on software concepts such as models, views and controllers.
Instead, the team decided that going forward, the code base should be modelled around real-world concepts such as orders, shipping, inventory and billing. This change would make it easier to locate code for future changes. Also, it will make it easier to locate people who understood the code for a particular concept really well.
With this thought process, each component was structured as its own mini rails app with the goal of name-spacing them as separate Ruby modules. An automated script was created to move the files into the newly decided structure. The main downside to this activity was losing a lot of commit history in Github.
Of course, this was a big-bang step taken by the team. But lot of internal discussions between various SMEs and developers had taken place to make the right decisions.
2 – Isolating Dependencies
The next objective was to decouple the various business domains or real-world domains from each other.
To achieve this, each component was provided a clean dedicated interface with domain boundaries expressed using a public API. Ownership of data was linked to the various components.
Of course, this was not something that could be done in a big-bang manner. Each subsystem needed time to evolve towards the target architecture. Therefore, an in-house tool was created to track the progress of each component towards it goal of isolation. Basically, the tool highlighted any violation of domain boundaries (such as direct access) or data coupling.
The tool also provided a dashboard view that could be seen by all developers and project owners to track their progress.
3 – Enforcing Boundaries
The long term objective was to move forward with this approach and implement boundary enforcement on a programmatic level.
For example, each component only loads the other components it explicitly depends upon. There would be a runtime error if we try to access code within a component that is not declared a dependency. Also, there would be runtime errors if components are accessed through anything other than their public API.
Achieving complete isolation and enforcing boundaries is an ongoing task, but due to several awareness sessions and workshops, all the developers within the organization are now onboard this change. Already in many places, teams are starting to see expected benefits of this approach.
The main conclusion from Shopify’s Modular Monolith system is that no architecture is often the best architecture during the early days of the system.
In the initial days of the systems, speed of adding features and functionality is usually more important. Monoliths are much better suited in this regard. There is no practical use in spending weeks and months attempting to architect a complex system that you don’t yet know. It is better to trade off absolute design quality for time to market. Once the speed at which you can add features and functionality begins to slow down, that’s when we should think about investing in good design.
The best time to refactor and re-architect is as late as possible. As you work on the system, you are constantly learning more about your system and business domain. Designing a complex system of microservices before you have complete domain expertise is a risky move that too many software projects fall into. Good software architecture is a constantly evolving task and the correct solution for your application depends on the scale of your operation.
What do you think about Shopify’s Modular Monolithic approach? Are monolith systems good or bad? If you have used Monolithic systems, what have been the challenges and how you have overcome them?
Do share your views in the comments section of this post. And if you liked this post, please do share it among your friends and colleagues.
0 Comments