Skip to main content

What is Apache Kafka?

Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that sends messages between applications, processes, and servers. It is composed of three important components called producer (which acts as Sender), Topics (topic or category of messages), and Consumer (which acts as a Reciever).

Kafka aims to provide high-throughput, low latency for real-time data feeds. It is widely used for real-time streams analytics, ingestion data into Spark, Complex Event Processing, Log aggregation, etc.


Before understanding Kafka Architecture, lets first understand what Kafka Broker is

Kafka Broker:

 As Kafka is a distributed framework, Kafka's cluster consists of different servers called Brokers running Kafka. Producers publish the message to Kafka Topics within this Broker and consumer consumes this message from Topics.

Kafka Architecture:

Kafka Architecture is comprised of mainly 3 components


  1. Kafka Producer
  2. Kafka Topics
  3. Kafka Consumer


Each Topic has Partitions, which contains messages in non-changing sequences. Each message is identified using a unique offset. This helps multiple consumers to read from topics at the same time. Each Topics partition can replicate the messages over several Kafka Brokers in the cluster.

Kafka also has a Consumer Group in which different consumers are assigned to consume messages from different partitions of specific Topics. They will receive the messages from a different subset of the partitions in Topic.

In the next posts, I will discuss more Kafka Producers and Consumers and provide JAVA code with explanations for writing Kafka Producer and Consumer.

Also, check out more posts below:



Labels:

Comments

Post a Comment

Popular posts from this blog

Tricky Questions or Puzzles in C ( Updated for 2026)

Updated for 2026 This article was originally written when C/C++ puzzles were commonly asked in interviews. While such language-specific puzzles are less frequent today, the problem-solving and logical reasoning skills tested here remain highly relevant for modern Software Engineering, Data Engineering, SQL, and system design interviews . Why These Puzzles Still Matter in 2026 Although most Software &   Data Engineering interviews today focus on Programming, SQL, data pipelines, cloud platforms, and system design , interviewers still care deeply about how you think . These puzzles test: Logical reasoning Edge-case handling Understanding of execution flow Ability to reason under pressure The language may change , but the thinking patterns do not . How These Skills Apply to Data Engineering Interviews The same skills tested by C/C++ puzzles appear in modern interviews as: SQL edge cases and NULL handling Data pipeline failure scenarios Incremental vs ...
23 comments
Read more

Decorators in Python

Decorators provide additional functionality to the functions without directly changing their definition of them. Basically, it takes the functions as an argument, adds functionality to them, and returns it.  Before diving deep into the concept of Decorators, let's first try to understand  What are functions in Python and what is an inner function? In Python everything is Objects, be it Class, Variables, and Functions . So functions are python first-class objects that can be used or passed as an argument. You can store the functions in variables, you can pass a function to another function as parameters, and you can also return the function from the function. Below is one simple example where we are treating functions as objects . def make_me_lowercase ( str ): return str .lower() print (make_me_lowercase( "HELLO World" )) copy_of_you = make_me_lowercase print (copy_of_you( "HELLO World" )) The output of the above calls for both the functio...
2 comments
Read more

Program to uncompress a string ie a2b3c4 to aabbbcccc

Below is the program to uncompress a string #include<stdio.h> #include<conio.h> #include<stdlib.h> int main() { char str[100]="a2b3c4d8u7"; for(int i=0;str[i]!='\0';i++) { if(i%2!=0) { for(int j=0;j<atoi(&str[i]);j++) { printf("%c",str[i-1]); } } } getch(); } Want to become a Data Engineer? Check out below blog posts  1.  5 Key Skills Every Data Engineer needs in 2023 2.  How to prepare for Data Engineering Interviews 3.  Top 25 Data Engineer Questions
2 comments
Read more