Strategies to integrate the transactional outbox pattern in our code


Strategies to implement the transactional outbox pattern

 

 

Joao_antunes

 

Article by João Antunes
Microsoft MVP and .NET Expert

 

How to integrate the transactional outbox pattern in your code

In this article, we're assuming we need to go the manual route, which, ideally, we'd avoid by using an existent framework or library capable of abstracting some of these challenges. If you're not familiar with this pattern, my presentation in this year's BOLD-X Digital Routes event or check my introductory post on this pattern, so if you want to get up to speed, it should be a good place to start.

Explicitly do it along with the other database orchestration

Let's start with what's probably the most obvious strategy, add/update/delete the entity, add the event, then commit. In C# like pseudo-code, it could look something like the following:

         csharp
         var order = db.Orders.GetById(orderId);
         order.Deliver();
         db.Orders.Update(order);
         db.Outbox.Add(new OrderDeliveredEvent());
         db.Save();

It's a pretty obvious solution and should achieve our objectives. Maybe not the prettiest thing ever, but gets the job done.
 

Hide it behind an abstraction

Another approach we can consider, pretty much in line with the previous one, just slightly tweaked, is to hide the portion of the code that stores things in the outbox behind an abstraction.

Take the following sample code:

      csharp
      var order = db.Orders.GetById(orderId);
      order.Deliver();
      db.Orders.Update(order);
      messagePublisher.Publish(new OrderDeliveredEvent());
      db.Save();

Where the code for `messagePublisher.Publish` would be basically the same as in the previous approach.

This gives it a feeling like we're actually publishing a message at that moment, while we actually aren't. It's not really better than the previous approach, most probably a matter of preference.

It does, however, have a couple of potential pitfalls:

      1. The abstraction falls apart if the programmer needs to remember to call `Save` after `Publish`, to ensure it happens in a transaction.
      2. To avoid pitfall 1, we'd need to wrap the whole code in a transaction.

For a manually implemented strategy, this one doesn't feel like the best one. For a framework or library though, it might make more sense, as the code could potentially be already running in a context the library controls, so it could easily handle the transactional requirements.

For example, imagine a library provides an interface `IMessageHandler`, which we implement to handle messages. Before the library passes the message on for our code to handle, it would start a transaction, then commit it when our code returns.

It could look something like:

         // our code
         class DeliverOrderHandler : IMessageHandler<DeliverOrderCommand>
   {
             void Handle(DeliverOrderCommand command)
             {
                 var order = db.Orders.GetById(command.OrderId);
                 order.Deliver();
                 db.Orders.Update(order);
                 messagePublisher.Publish(new OrderDeliveredEvent());
                 db.Save();
             }
   }
         // library code
         db.BeginTransaction();
         messageHandler.Handle(message);
         db.CommitTransaction();
 

Infer events from the ORM change tracker

When using an ORM, we can take advantage of its change tracking capabilities and extensibility points to infer events and add them to the database in a single place.

Consider the following code for our domain logic:

         var order = db.Orders.GetById(orderId);
         order.Deliver();
         db.Orders.Update(order);
         db.Save();

It's free of worries with message publishing, so that means we need to move it somewhere else. Hooking up to extensibility points of the ORM, we could for example override the `Save` method (a common approach in EF Core, different ORMs could have different ways of exposing extensibility points).

         override void Save()
         {
             this.InferAndAddMessagesToOutbox();
             base.Save();
         }

         void InferAndAddMessagesToOutbox()
         {
             foreach (var entry in this.changeTracker)
             {
                 if (entry.State == EntityState.Modified
                     && entry.Entity is Order order
                     && entry.OriginalValues.Status == Status.Pending
                     && entry.CurrentValues.Status == Status.Delivered)
                 {
                     this.Outbox.Add(new OrderDeliveredEvent());
                }
            }
         }

This strategy might be interesting if we want to centralize all the logic related to creating and adding the events to the outbox. It probably doesn't scale very well though, depending on the amount of events we need to infer, as well as how complex it is to infer them, the code will grow fast.
 

Include it in the domain entities logic

The final strategy in this post consists in moving the event creation task into the domain entities logic.

The orchestration code would look the same as in the previous strategy:

         var order = db.Orders.GetById(orderId);
         order.Deliver();
         db.Orders.Update(order);
         db.Save();

But in this case, instead of inferring events, we'd go back to creating them explicitly, just in a different place. To do this, we could, for example, make our entities inherit from a base entity class, which would provide it with a way to store the events.

         abstract class EntityBase
         {
             List<OutboxMessage> Messages;

             void AddMessage(OutboxMessage message)
            {
                Messages.Add(message);
            } 
         }

         class Order : EntityBase
         {
            OrderStatus Status;

             void Deliver()
            {
                 Status = OrderStatus.Delivered;
                this.AddMessage(new OrderDeliveredEvent());
            }
         }

With this in place, we could again resort to an ORM extensibility point, to grab the messages from the entity being saved and put them in the outbox table.

         override void Save()
         {
            this.AddMessagesToOutbox();
             base.Save();
         }

         void AddMessagesToOutbox()
         {
             foreach (var entry in this.changeTracker)
            {
                if (entry.Entity is EntityBase entity)
                {
                     this.Outbox.Add(entity.Messages);
                 }
            }
         }

In this case the code we attach to the ORM extensibility point doesn't need to change for each new event we have, it's generic and will always grab all events from the entities.

From all the strategies discussed in this post, this is my favourite one.

First and foremost, because we're putting event creation logic where it belongs, in the domain entities that should centralize as much of the application domain knowledge as possible, and domain events should be part of it.

Additionally, it feels like a good fit for different types of data persistence approaches.

If we're using a SQL database, this approach works just as well as the others we've looked at previously.

If we instead use a NoSQL database, such as [Azure Cosmos DB](https://azure.microsoft.com/en-us/services/cosmos-db/), as we can't have transactional guarantees if we store the entities and the events separately, the events should be stored within the entity that triggered them, so we're already in a good place to achieve it.

Even looking at a third possibility, using event sourcing, it fits well, as it does in this case, the entities should not only create the events that represent their state transitions, they also need to handle those events when they're loaded from the database.
 

A word on finally publishing the messages

I focused the post on the portion of the transactional outbox pattern that gets more intertwined with the rest of our code, adding the messages to the outbox. Of course, implementing the pattern isn't completed until we've actually published the messages to the message broker.

We have two main ways of achieving this: polling the outbox or tailing the transaction log.

Polling the outbox, consists in having some background process that will check the outbox for new messages every once in a while. This works with SQL databases, where we can centralize the outbox in a table we can easily look at from time to time.

Tailing the transaction log, consists in hooking up to the database transaction log and checking for new messages being pushed into the outbox. This works with SQL databases, which have the traditional transaction log, but also with NoSQL databases that expose ways to consume the changes that have been applied (e.g. Azure Cosmos DB Change Feed).

The last couple of times I implemented this pattern, I went with a slight variation of the polling approach. In addition to checking the outbox every few seconds, added a way to trigger a check immediately upon a new message being pushed into the outbox table, in order to reduce the latency between operation completion and message publishing. If you're interested in seeing a sample of this approach, check the "Event-driven through cloudy skies" presentation repository.
 

Consider using an existing framework/library

As interesting and fun as all this talk is, it's clear it's not trivial to implement a robust solution. As such, I'd recommend starting by looking at available frameworks or libraries that abstract some of these challenges.

In .NET land, there's no shortage of options, like NServiceBus, MassTransit or Brighter, just to name a few.

Of course, don't just assume the libraries will "just work". Depending on your use case, they might, in fact, just work, but in others you might need to adapt to the library's recommended patterns and practices. In other cases the library might actually not work for you at all, so it's time to look for another one or eventually roll up your sleeves and implement something that fits your needs.

Taking the transactional outbox pattern as an example, from my research on some of the available libraries, it seems that it's well implemented in general when publishing messages from a message handler (a consumer of commands or events), but if we want to publish messages from outside of a message handler (e.g. an HTTP request handler), not so much.

Another reason why the frameworks/libraries don't work for a given use case, is if they don't support the infrastructure being used (i.e. type of database or message broker).

Again, first try to see if there are available frameworks or libraries that support your requirements, as that'll abstract many challenges and save a ton of work, while giving you the extra peace of mind of using something thoroughly tested and used by others. If it doesn't fit, then there's no other option than rolling your own.

If you found this article useful and want to discuss it further, get in touch with us!

\