My Software Architecture Journey


Some stories start at the beginning. My story begins in a Telco company in Portugal.

 

 

Armando Rodrigues

 

Article by Armando Rodrigues
Software Engineer

 

Some stories start at the beginning. My story begins in a Telco company in Portugal.

At that time, most web software used a monolithic architecture. You had one backend, one database and… that’s it! As you can see, it’s pretty simple to develop, deploy and scale.

Well, it started off simple, then we began to add feature after feature and a simple application became a very complex application. (In other words, a large code base that was hard to maintain and intimidating for new developers). IDE’s became  overloaded. Continuous integration took hours. We had a long-term commitment to a technology stack; and changing that meant rebuilding most of the application which amongst many other reasons , just isn't cost effective. Availability was also an issue because sometimes one small problem could compromise the entire application.

But can’t you have multiple instances?

You can, but remember that this was before Cloud was a thing. Back then, “auto-scaling” was more “manual-scaling” and the start time for these applications could easily reach several minutes. Enter the cloud. Cloud solved one of the problems with Infrastructure as a Service, allowing your application to auto-scale and your company would pay only what you consume, you didn’t have to spend money buying bare metal in advance and have employees to support that infrastructure.

For the following years, I focused on solving some of the above-mentioned  monolithic  problems, and most companies saw the answer in a new architecture design: Microservices. When designed properly, you will have decoupled microservices with business logic that will allow you to easily scale them.

So how do you design Microservices?

To be honest, I don’t know yet. But it has been my odyssey since then, and this article is about that adventurous journey.

My first attempt to build a microservices architecture was quite an interesting experience. As a young and foolish software developer I thought I could read a few articles and definitions of microservices architecture and implement a fully functional software, just like that! Well, it turns out I couldn’t. It was a complete disaster!

I read somewhere that each microservice should have their own database, or at least schema, but I decided to ignore that and use what I thought was my brilliant mind (It turns out it was just a foolish mind) and use the same database for all microservice… if you also did that you’ll know what a spectacular disaster that turned out to be, but in case not, allow me to tell you: We deployed and tested the application without any problem, everything was working as expected. With that amazing news we went live. First few days were calm, then the first weekend arrived and chaos struck, all microservices stopped responding and it took us hours to understand that there was a limit of database connections and as the application traffic started to increase, the instances per service and the number of database connections increased too. 

On that day I learned two very valuable lessons:

 1. Never go live with a software without stress tests to identify bottlenecks and limits of your application.

 2. Never ignore advice from those who have way more experience than you, no matter how smart you think you are, experience is more important than that. 

Later on we had more problems regarding shared databases between microservices, it’s hard to maintain and deploy. I could write a few more paragraphs about that but my only advice is: DON'T DO IT! :)

A year later, I was building new software using microservices with individual databases for each one. This time, before designing anything I decided to read books, articles and talks from authors like Robert C. Martin (aka Uncle Bob), Michael Amundsen, Chris Richardson, Eric Evans. I read so much that I began to understand one simple thing: there was no perfect recipe to design microservices. At least, nobody has found one yet. But I did find some patterns that will at least help guide us in the right direction.

I was now conscious of how important it was to have the services completely decoupled but was not aware of all the challenges there would be  to achieve that - which my next project allowed me to explore.

The project was an online store. Like any online store, it sells products to customers (obviously) so this means that you need: Orders, products, and customers, right? So, the first question I asked: Should I have a microservice for each business capability? Some sort of Domain-Driven Design (DDD)? Or can I have two or more business capabilities in the same microservices? The short answer: Both are acceptable! There is no rule that says that each business capability or domain must have their own microservice, that’s a decision that you need to make based on application needs. There are advantages and disadvantages to both and actually it is more a trade off between consistency and scalability. 

Let me give you a more detailed explanation.

You have an order microservice and a customer microservice, each one has their own database (or at least an independent schema). You receive a request to create an order and you need to verify if the customer has a balance for that order and if so, you need to deduct the order amount from the balance. Traditionally I used a transaction where I would create the order, update the balance and if there is no error I would commit the transaction. In case of an error I would rollback everything inside that transaction. That’s not an option for this case because we have the data in two different databases. Also, traditionally the order table will have a foreign key(FK) to the customer table, right? You can’t have a FK between two databases (there is a way of doing that but it’s very specific and not recommended).

The solution was to use aggregates and use the aggregate ID as reference, in this case a customer ID.  And here is where we have the trade off between consistency and scalability: since you are using references to the other aggregates it will be harder for you to keep the data consistent and it will require extra complexity to do that. On the other hand you will have a complete decoupled domain that will be very easy to scale and maintain.

Monolith to Microservices with Quarkus.io


So, we had microservices with individual databases that needed to communicate between them. Our solution at that time was to call the microservice API directly. It worked… kind of, but the result of the stress tests were not what we expected (and not in a good way). With that in mind we decided to implement Event-based communication using RabbitMQ, which solved the majority of the issues (there were still some minor problems, but mostly from lack of experience).
With that problem solved we had a new goal: make the system resilient.

There were multiple ways to do that, but our approach was to use the stress tests to identify bottlenecks and system failures and even the system limits. With that information we were able to add fault tolerance, monitoring and alarming to prevent system failures.

The last pain I wanted to share was CI/CD. For these projects we just reused the existing CI/CD (most of the time with Jenkins) and thought that would work fine. It’s not that it didn't, the pipelines were working and deploying, but working with microservices involves a mindset change that took us some time to accept.  We spent a lot of time trying to adapt the pipelines and maintain them. Be aware that the number of pipelines will increase exponentially and in time, you will want to have controlled deployments using Blue-Green or Canary deployments or have multi versions of a service.

On my more recent projects I’ve been more successful, but there is always something that doesn't work as expected and as I mentioned before, I don’t know (yet) how to design and build microservices, but I am definitely experienced in what not to do!

You're probably wondering if, after all these problems, I would still recommend microservices? 
My answer: It depends on what you need to build.

If you're building software for internal usage of an organization with a low number of users and low complexity, then a monolith architecture should work fine. On the other hand if you’re building software that has to scale and you don’t have a clear view of the future or complexity, I would advise you to use microservice because In the end, and if you build it right, you will have a more flexible solution easy to scale and to maintain.

Imagine that I needed to hire a new developer to join the team, instead of facing a monstrous application he could start with one microservice. As a developer that had to learn very complex monolithic applications I know how hard and intimidating it is to learn an application like that. If this new developer is working only on one microservice it will be faster to compile and test right? He is only changing the code of one microservice and that’s what you need to test. Also, you don’t need to run hundreds or thousands of tests each time you change your code, you just need to run the tests of that particular microservice. If you change one microservice and that change affects other parts of the application you are doing something wrong. Same thing for deployment, if you are changing the customer service to add some new features you only need to deploy those changes. It’s also easy to scale the development team because you can have multiple teams working on the same application without any entropy. Each team will be responsible for one or more microservice and the only thing that they need to agree on is the interface that will be used to communicate. Your application will be more fault tolerant, if something unexpected happens to one microservice it will not break the entire application. 

You can use different tech stacks in different microservies. And last but not least, you can easily scale each microservice with different rules.

My journey with microservices has now taken me to serverless architecture, and like the microservices road it’s off to a rocky start, but let's see where I end up. The important lesson that I wanted to share with you is that software architectures are like technology; you need to understand their advantages, disadvantages and how they work before you use them.
 

If you like this article:

Don't miss our next BOLD-X masterclass. "Modern Observability - Logging, Request Tracing, and Metrics Collection"

Register
\