The evolution of business
There have been massive shifts in how services are consumed by customers over the last few years which is necessitating a change to event-driven architectures in order to effectively deliver rich customer experiences and facilitate back-end operations. Where it was once the norm to queue at a bank, customers now expect a fully digital banking experience, either online or through an app. They want instant notifications when transactions occur or if there are potential fraudulent activities happening on their account. They also want to apply for new banking products through an app and expect immediate approval or some interaction to confirm that something indeed happened to their request.
Another such example is grocery shopping where there is a definite shift in consumer behaviour with many people wanting to order their groceries online. They want delivery within an hour, live tracking of their order once it is placed, live tracking of the driver once they are on their way and they want to be notified when their groceries are arriving.
Do you notice something similar in these two examples? Both the rich front-end experiences and backend operations which support this are driven by events happening in real-time.
In this post, I will discuss event streaming, the paradigm many businesses are turning to make this all possible. I will also detail why this approach has become so immensely popular and is quickly replacing or supplementing the more traditional approaches in delivering on real-time back-end operations and rich front-end customer experiences.
What is Event Streaming?
To understand this, I will start by looking at what is meant by an event. This is pretty straightforward as a concept. An event is simply something that has happened. A change in the operation of some piece of equipment or device is an event. The change in the location of a driver reported by a GPS device is an event. Someone clicking on something on a web app or interactions on a mobile device are events. Changes in business processes, such as an invoice becoming past due is an event. Sales, trades, and shipments are all events. You get the idea.
Having the ability for all lines of business to harness, reason on top of and act on all events occurring in a business in real-time, can solve a multitude of problems, as discussed in our introduction, and is exactly what event streaming is designed to do.
A note before I continue, Apache Kafka is recognised as the de facto standard for event streaming platforms and many of my assertions in this post are based on this technology.
Event streaming uses the publish/subscribe approach to enable asynchronous communications between systems. This approach effectively decouples applications that send events from those that receive them.
In Apache Kafka we talk about producers and consumers, where your producers are apps or other data sources which produce events to what we refer to as topics. Consumers sit on the other end of the platform and subscribe to these topics from where they consume the events.
The decoupling of producers from consumers is vital in mitigating delays associated with synchronous communications and catering for multiple consumers or clients as opposed to point-to-point messaging.
The topics we mentioned above are broken up into partitions. The events themselves are written to these partitions. Simply put partitions are append-only logs which are persisted. This persistence means the same events can be consumed by multiple applications, each having its own offsets. It also means apps can replay events if required.
There are also a number of other aspects that characterise an event streaming solution, such as the scalability, elasticity, reliability, durability, stream processing, flexibility and openness afforded to developers.
This is a very high-level depiction of what characterises an event streaming platform but when you bring these aspects together, you have the building blocks for a fully event-driven architecture.
Some of the concepts I have discussed above may be familiar to some and there certainly are many technologies that fall into the integration or middleware space. The next section will detail what some of the key requirements are for an event streaming platform along with what differentiates event streaming from other similar technologies.
Event Streaming key requirements and differentiators
There are 4 key requirements for enabling event-driven architectures. I have detailed these below along with the main differentiators to traditional tools used in the middleware/integration space.
- The system needs to be built for real-time events. This is in contrast to a data warehouse for example which is good at historical analysis and reporting. When considering databases, the general assumption is that the data is at rest, with slow batch processes which are run daily. Databases also only typically store the current state and not all events which occurred to reach that state. When considering ETL tools, they are traditionally good for batch processing but not real-time events.
- The system needs to be scalable to cater for all event data. This goes beyond just transactional data, which a database, for example, has been historically used for. As we have described above, events stretch across the whole business and can include data from IoT devices, logs, security events, customer interactions, etc. The volumes of data from these sources are significantly more than transactional data. Message Queues (MQs) have traditionally been used as a form of asynchronous service-to-service communication but struggle to scale to efficiently handle large volumes of events and remain performant.
- The system needs to be persistent and durable. Data loss is not acceptable for mission-critical applications. Furthermore, mission-critical applications typically have a requirement for events to be replayed meaning persisting events. MQs are transient in nature and messages are not persisted, meaning once the event is consumed, it is dropped. This has some obvious downsides if the intention is for events to be replayed or for multiple domains in the business to subscribe to the same events.
- The system needs to provide flexibility and openness to development teams and have the capability to enrich events in real-time. Although not relevant for all use cases, having the capability to enrich data in flight has a multitude of benefits and can alleviate the efforts of multiple consumers having to build enrichment logic into their applications. The important aspect is the flexibility event streaming architectures provide where the system can either provide real-time enrichment pipelines or simply dumb pipelines where events can be passed through, enriched or directly consumed by the application. There have historically been many systems used in the application integration or middleware space, including ETL tools for batch pipelines, ESBs to enable SOAs and many more which fulfil very similar functions to each other. I am not going to fully unpack all the pros and cons across the technologies, suffice it to say that businesses are moving away from “black box” type systems such as ESBs where there are massive dependencies on critical skills which often cause bottlenecks in productivity. In contrast, event streaming provides businesses with flexible and open architectures which decouple applications, alleviate these dependencies and allow any kind of application or system to be integrated regardless of the technologies used.
When considering the 4 key elements above, Event Streaming is the only technology that ticks all the boxes and is quickly being entrenched as a critical component in the data infrastructure of businesses.
What is the end goal?
To circle back to the opening of my post, the expectations on businesses are high to continuously improve on the experiences delivered to customers as well as the back-end operations which support this.
The more focused a business is on delivering real-time event-driven architectures which are scalable, persistent, durable, open, flexible and remove critical dependencies, the more likely they are to succeed and rise above the competition.
There is a very distinct maturity curve when it comes to the adoption of event streaming. Becoming a fully event-driven organisation means that event streaming has been deployed across all lines of business and serves as the central nervous system of all business events. Getting to this point is a journey that often requires a shift in how things have traditionally been done.
This is luckily a journey that you do not need to go on alone. Please feel free to contact us and see how we can help and also stay tuned for my next post where I will go a bit more in-depth on use cases and benefits of adopting event streaming.
