Building micro services through Event Driven Architecture part13 : Consume events from Apache KAFKA and project streams into ElasticSearch.

Building microservices through Event Driven Architecture part13 : Read model projection.

This tutorial is the 13th part of a series : Building microservices through Event Driven Architecture.

The previous step is about Building microservices through Event Driven Architecture part12 : Produce events to Apache KAFKA

In this tutorial, I will show how to read streams from KAFKA and project streams into ElasticSearch.

I have to consume messages from KAFKA, the messages that I read from KAFKA are event streams.  So I a have to project these streams into a structural representation.  Then I will  index the projections  into ElasticSearch.

So I will build a consumer that subscribes to KAFKA and  listen to events.  If it receives an event , it will use projections to create a structural presentation of the event. And finally store it to a nosql database ElasticSearch.

Projecting events

In practice, reading thousands of events will take too long, Instead we could precalculate the current state and store it to a nosql database.
The projection can be defined as the current state derived from a sequence of events

I define a base generic class Entity, so each projection will  derive from it.

I define a abstract generic class Projection, that takes a list of events and applies them to the concrete class ( SpeechProjection in our case).

SpeechProjection is a class that represent the entity that I want to rebuild its state from the events (SpeechCreatedEvent, SpeechTitleChangedEventSpeechDescriptionChangedEvent, SpeechUrlChangedEvent and SpeechTypeChangedEvent ).

So for each event related to the given entity  ( speech), I have to apply the event to the entity.

Introduction to ElasticSearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease.

Go to the following link to install elasticsearch :

You can verify if the installation is OK  by running the following command curl http://localhost:9200/ or Invoke-RestMethod http://localhost:9200 with PowerShell

The following code create a generic repository to connect to elastic search, and perform CRUD operations.

Create a worker service

The ASP.NET Core Worker Service template provides a starting point for writing long running service apps.

We can use worker service to build applications which do not require user interaction or perform periodic and long-running workloads.

I will use Worker Service to build a consumer service that  consume events from APACHE KAFKA and index them into ElasticSearch 


ConsumerHostedService is a background service that host ConsumerService 


ConsumerService calls a service  bus that receives notifications from a Kafka when a new event is produced .

Service Bus


The KafkaClient implements ReceiveAsync of IServiceBusProvider. It subscribe to a Kafka topic, so when an event is published to that topic, it notifies a mediator service.

ElasticSearchNotifier implements  INotificationHandler. The responsability of this class is to deserialize the input event  and index it to elasticsearch.


Start zookeeper

zookeeper-server-start.bat config\

Start Kafka

kafka-server-start.bat config\

Start ElasticSearch



Start the following  projects :

  • LogCorner.EduSync.SignalR.Server
  • LogCorner.EduSync.Speech.Producer
  • LogCorner.EduSync.Speech.Consumer

Start the following  project :

  • LogCorner.EduSync.Speech.Presentation

Start Postman and post a new command

You should see the following output on the comsumer console, consuming the command posted on postman

Code source is available here : 

Thanks for reading, if you have any feedback, feel free to post it



I'm a microsoft most valuable professional (MVP) .NET Architect and Technical Expert skills located in Paris (FRANCE). The purpose of this blog is mainly to post general .NET tips and tricks, Gora LEYE

Support us

BMC logoBuy me a coffee