Mark Makary Blog

Productivity is the queen and logging is the king.

I was fortunate to attend the Reactive Summit last week (October 4th-5th). Lightbend was the main host and they were gracious to make this a celebration of the reactive movement versus creating another vendor marketing event, hats off to Lightbend on that front.

Austin, TX is a beautiful city and it was a perfect choice for such a venue.

This is a summary of the highlights, in case you missed it and you’re looking for the expresso summary.

The focus was on two main tracks, the first one is Microservices and reactive versus monolithic and the second is fast data processing or data streams processing.

Reactive versus Monolithic and the Microservices Architecture

Well, -we all know- for Enterprise Architects and the IT Business Executives, the term “Cool Technologies” is a dreaded and hated one, what I’m going to do with the cool technology is really the main driver. The answer on this one, -among few other things- is “productivity”.

This is back in the realm of the illusive componentization and the service composition architecture, I’m sure most of us tried to get development agility out of service oriented architecture or SOA and how that end up slowing down development and productivity instead of agility?

The Reactive Movement and Microservices is an attempt to improve productivity and agility while still providing scalability, elasticity, responsiveness and resilience.

Ideas to business services

Hypothetically, assuming a company is similar to Uber and another company is similar to Amazon; are looking into expanding on a new revenue stream that is generated from home grocery delivery service, both of them have the infrastructure, maybe one has more on the driver integration and the other has more on the shopping cart; which one will get there first?

The typical answer usually would be; agility and productivity, productivity is the queen as highlighted earlier.

Actors

Actors are at the heart of Microservices, independent, isolated, atomic, exclusively responsible for its own state, the system is comprised of actors, actors come in systems though and not alone.

Ship bulk-heading was highlighted few times as an example of flooding one compartment wouldn’t sink the ship, for a simpler picture, please refer to my write up on Microservices.

Service Scope

It was brought up couple of times, Microservices night not be the proper name; it is misleading in a way that it makes people focus more on size than purpose, instead the Unix model was used extensively,

“Write programs that do one thing and

do it well. Write programs to work together”

It is not about the size, it is about the business purpose, for instance, if you’re building a pricing service, you can logically break down the services into micro services, such as, individual pricing service, bulk pricing service, custom pricing service, etc…

Communication

Service-to-Service Communication

Asynchronous communication, message driven architecture and Event driven architecture. Historically all of those design patterns suffer from persistence latency, for instance if you’re using the fire-and-forget design pattern, you have to persist to desk and you have to have a transaction coordinator to make sure the message is processed and processed only once before removing it from the messaging queue, that introduced high cost, latency and complexity, however; recently, hardware advances overcame those obstacles, solid state drives and network advances allowed us to use commodity hardware to store and manage massive amount of data for relatively long time -days or weeks-.

Look for new technologies in that space, such as Apache Kafka and Lightbend Akka running in production and solving the fire-and-forget problem in a reactive and responsive way.

Consumer-to-Service Communication

Rest is not an evil thing, it’s an acceptable Reactive consumer-to-service protocol, there are situations where you can use Rest or asynchronous messaging, it depends on the situation.

Also, you can use Rest in Asynchronous and non-blocking IO fashion.

Audit trail and Logging

File system logging is the new king.

Now, you can use logging and audit trail to solve a variety of problems, as highlighted in the previous section, writing to file system on a disk is not a bad thing anymore, you can also avoid memory congestion and GC issues associated with big heaps.

Distributed Computing and Microservices Containers

So, we have messaging and message-driven architecture. How about message routing, and keeping track of those messages, how about service discovery and monitoring, this is Microservices containers.

You can look into Lightbend Akka and Lagom, they focus on production ready Microservices containers.

State Management

Can you avoid state management in commercial applications?

Don’t ignore the state, instead, the common wisdom is to separate the stateless functions from the stateful ones, for instance use Functional programming for stateless operations and benefit from the associated scalability, a good example is orchestration.

Transactions

In a distributed model, we will create bottlenecks if we use high-latency patterns, such as two-phase commit, instead, use the compensation model, retries and timeout and logging and audit trail for rollbacks to a consistent state.

Deployment and DevOps

Containers and logical data centers such as Apache Mesos took the center stage, you can’t avoid them and focus on VMs only any longer.

Introduce Failure to your system

In a massive distributed system, you need to start introducing failure scenarios by looking into successful invocation paths and introduce failure or disruption to those successful paths to test your ability to self-heal and recover, don’t wait to test this in production.

Elasticity vs. Scalability

It was brought up few times, Scalability is a subset of Elasticity, it’s important to scale down too and save on the computing cost and reach efficient utilization with Elasticity.

API Management

API management is essential part of a reactive and responsive system, the gateway will provide the SLA, orchestration in some use cases, security and responsiveness.

Fast Data

Near Real-time data streams processing

Let’s say you have a company producing a smart fridge, where it can notify the consumer of the need to refill certain grocery items, imagine the amount of data streams that are coming down the pipes to analyze and take actions upon, it is huge, add to it a recommendation on where to buy or refill those grocery items, based on the data that company receives on the prices and the consumer reviews on providers that physically located within the vicinity of the consumer.

Batch data processing

Now, that smart fridge company is going to look into the consumers’ usage trends and analyze this to provide useful information to the consumers on how much they spend on different categories and what are the kids’ favorite items.

All this is done by just opening and closing the fridge and opening and closing certain drawers.

To accomplish this, we need to look into historical data and run batch processing.

Mixing the two

Apache Kafka, Akka streams, Apache Spark and many more open source technologies are covering those areas and use cases, Lightbend has a fast data production ready platform that compile some of those successful technologies into one platform.

Conclusion

This is of course a quick summary and please don’t expect that to cover all the items discussed in the summit, for a more detailed coverage, please visit the Summit blog and pages.

References

Reactive Microservices Architecture by Jonas Boner

Fast Data Architecture for Streaming Applications by Dean Wampler

Developing Reactive Microservices by Markus Eisele

The Reactive Summit (2016) pages

Disclaimer

Those are my personal views and not representing the people I worked with, the companies I worked for, or my/our past and present customers in any shape or form. Any resemblance to real life use cases or situations is accidental and not intentional in any way, shape or form.

Hope this is helping some and again I understand other’s experience and views could be completely different than mine and I completely respect that.