Mark Makary Blog

Prerequisites

For more info on Reactive design patterns such as CQRS, event sourcing, command sourcing and Saga please refer to the references section.

Introduction

Reactive architecture is an architecture approach aims to use asynchronous messaging or event driven architecture to build Responsive, Resilient and Elastic systems.

Reactive Microservices is capitalizing on the Reactive approach while supporting faster time to market using Microservices.

Reactive Microservices is using asynchronous messaging to minimize or isolate the negative effects of resource contention, coherency delays and inter-service communication network latency.

By using an event driven architecture we can have both agile development and build responsive systems.

To rephrase the question within this context, why Kafka is a better fit for an event-driven architecture and consequently reactive microservices than the traditional messaging systems?

Kafka Architecture

Some would argue that; traditional messaging systems are capable of doing everything Kafka can do in the event driven architecture space. However, even if this is true, still, this will require a lot of workarounds and it won’t come naturally to those traditional messaging systems. Meanwhile, Kafka architecture is a natural when it comes to reactive architecture, so how Kafka architecture differs from the traditional messaging systems in that space?

Reactive Architecture design patterns

To minimize the effect of resource contention, state synchronization (consistency) and latency; Reactive adopts a set of design patterns such as CQRS, event sourcing, command sourcing and sharding to trade off eventual consistency, availability and scalability for strong consistency (CAP Theorem). Kafka is a perfect fit for those design patterns in the following regard,

Kafka event log architecture

Kafka is a distributed, structured and sequential event log, messages are not removed when they are read as other messaging systems do. This architecture has a positive outcome when it comes to the Reactive design patterns as highlighted below.

Persistence is a first class citizen in Kafka

Durable messages supports both consistency and availability, enable us to time travel and replay commands or events in case those events or commands didn’t make it to the event store. It’s also, used for audit and correct failed transaction. It’s helpful for invoking compensation actions in distributed Saga.

Persistence is a first class citizen in Kafka. Kafka is using distributed HDFS to store events and retain those events for a configured period of time, other messaging treat persistence as a second class citizen.

Kafka Stream Processing

Event and Command sourcing requires stream processing engine and event handler. Kafka streaming is a very low latency event handling and stream processing engine, you can utilize using Java or Scala.

Guaranteed order per partition

In event and command sourcing the order of the messages might impact the state. The wrong order of messages can lead to a corrupted state. Kafka guarantees the order of messages per partition by default, we can combine with a specific partition key to solve a distributed guaranteed order problem. For example, all commands related to a specific order ID or key will land on the same partition.

Exactly once message delivery

Duplicate messages can cause many problems to the reactive design patterns, imagine how much harm would duplicate messages cause in a distributed Saga. It’s impossible to achieve exactly once semantics in a partitioned network, at least for now, please check the two generals’ problem for more info.

However you can simulate exactly once delivery using at least once delivery and idempotence.

Kafka does this very well since version 0.11.

Native sharding for strong consistency and more powerful availability

Sharding is a technique we use in Reactive to support strong consistency, by isolating contention to a specific partition. Kafka supports sharding in a form of partitions and uses partition key to make sure a message will land on a specific partition. Those partitions can then be replicated for more availability and that is an amazing balance between strong consistency and availability.

Native Distributed architecture and unparalleled throughput

As you can see Kafka covers many distributed concerns out of the box including distributed persistence, availability and scalability. However it does all that while maintaining unparalleled performance.

Do we still need Reactive toolkits such as Akka?

The short answer is yes.

While Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.

References

Confluent Blog and learning resources

Lightbend blog and learning resources

Lightbend free Reactive Courses

Disclaimer

Logic Keepers is part of the Confluent partner network and a consulting partner with Lightbend.

Those are my personal views and do not represent the people I have worked with, the companies I worked for, our consulting partners, or my/our past and present customers in any shape or form. Any resemblance to real life use cases or situations is accidental and not intentional in any way, shape or form.

I hope this is helpful to some, and again I understand other’s experience and views could be completely different than mine and I respect that.