Building micro services through Event Driven Architecture part13 : Consume events from Apache KAFKA and project streams into ElasticSearch.

16 January 202116 January 2021 Gora LEYE 0 Comment Azure Devops, Container, Continuous integration, CQRS, DDD, docker, Domain Event, Event Sourcing, Microservices, TDD, Unit Test

Building microservices through Event Driven Architecture part13 : Read model projection.

This tutorial is the 13th part of a series : Building microservices through Event Driven Architecture.

The previous step is about Building microservices through Event Driven Architecture part12 : Produce events to Apache KAFKA

In this tutorial, I will show how to read streams from KAFKA and project streams into ElasticSearch.

I have to consume messages from KAFKA, the messages that I read from KAFKA are event streams. So I a have to project these streams into a structural representation. Then I will index the projections into ElasticSearch.

So I will build a consumer that subscribes to KAFKA and listen to events. If it receives an event , it will use projections to create a structural presentation of the event. And finally store it to a nosql database ElasticSearch.

Projecting events

In practice, reading thousands of events will take too long, Instead we could precalculate the current state and store it to a nosql database.
The projection can be defined as the current state derived from a sequence of events

I define a base generic class Entity, so each projection will derive from it.

I define a abstract generic class Projection, that takes a list of events and applies them to the concrete class ( SpeechProjection in our case).

SpeechProjection is a class that represent the entity that I want to rebuild its state from the events (SpeechCreatedEvent, SpeechTitleChangedEvent, SpeechDescriptionChangedEvent, SpeechUrlChangedEvent and SpeechTypeChangedEvent ).

So for each event related to the given entity ( speech), I have to apply the event to the entity.

Introduction to ElasticSearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease. https://www.elastic.co/elasticsearch/

Go to the following link to install elasticsearch : https://www.elastic.co/downloads/elasticsearch

You can verify if the installation is OK by running the following command curl http://localhost:9200/ or Invoke-RestMethod http://localhost:9200 with PowerShell

The following code create a generic repository to connect to elastic search, and perform CRUD operations.

Create a worker service

The ASP.NET Core Worker Service template provides a starting point for writing long running service apps.

We can use worker service to build applications which do not require user interaction or perform periodic and long-running workloads.

https://docs.microsoft.com/fr-fr/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-5.0&tabs=visual-studio

I will use Worker Service to build a consumer service that consume events from APACHE KAFKA and index them into ElasticSearch

ConsumerHostedService

ConsumerHostedService is a background service that host ConsumerService

ConsumerService

ConsumerService calls a service bus that receives notifications from a Kafka when a new event is produced .

Service Bus

KafkaClient

The KafkaClient implements ReceiveAsync of IServiceBusProvider. It subscribe to a Kafka topic, so when an event is published to that topic, it notifies a mediator service.

ElasticSearchNotifier implements INotificationHandler. The responsability of this class is to deserialize the input event and index it to elasticsearch.