Skip to main content

What is Apache Kafka?

Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that sends messages between applications, processes, and servers. It is composed of three important components called producer (which acts as Sender), Topics (topic or category of messages), and Consumer (which acts as a Reciever).

Kafka aims to provide high-throughput, low latency for real-time data feeds. It is widely used for real-time streams analytics, ingestion data into Spark, Complex Event Processing, Log aggregation, etc.


Before understanding Kafka Architecture, lets first understand what Kafka Broker is

Kafka Broker:

 As Kafka is a distributed framework, Kafka's cluster consists of different servers called Brokers running Kafka. Producers publish the message to Kafka Topics within this Broker and consumer consumes this message from Topics.

Kafka Architecture:

Kafka Architecture is comprised of mainly 3 components


  1. Kafka Producer
  2. Kafka Topics
  3. Kafka Consumer


Each Topic has Partitions, which contains messages in non-changing sequences. Each message is identified using a unique offset. This helps multiple consumers to read from topics at the same time. Each Topics partition can replicate the messages over several Kafka Brokers in the cluster.

Kafka also has a Consumer Group in which different consumers are assigned to consume messages from different partitions of specific Topics. They will receive the messages from a different subset of the partitions in Topic.

In the next posts, I will discuss more Kafka Producers and Consumers and provide JAVA code with explanations for writing Kafka Producer and Consumer.

Also, check out more posts below:



Labels:

Comments

Post a Comment

Popular posts from this blog

Tricky Questions or Puzzles in C ( Updated for 2026)

Updated for 2026 This article was originally written when C/C++ puzzles were commonly asked in interviews. While such language-specific puzzles are less frequent today, the problem-solving and logical reasoning skills tested here remain highly relevant for modern Software Engineering, Data Engineering, SQL, and system design interviews . Why These Puzzles Still Matter in 2026 Although most Software &   Data Engineering interviews today focus on Programming, SQL, data pipelines, cloud platforms, and system design , interviewers still care deeply about how you think . These puzzles test: Logical reasoning Edge-case handling Understanding of execution flow Ability to reason under pressure The language may change , but the thinking patterns do not . How These Skills Apply to Data Engineering Interviews The same skills tested by C/C++ puzzles appear in modern interviews as: SQL edge cases and NULL handling Data pipeline failure scenarios Incremental vs ...
23 comments
Read more

Program to uncompress a string ie a2b3c4 to aabbbcccc

Below is the program to uncompress a string #include<stdio.h> #include<conio.h> #include<stdlib.h> int main() { char str[100]="a2b3c4d8u7"; for(int i=0;str[i]!='\0';i++) { if(i%2!=0) { for(int j=0;j<atoi(&str[i]);j++) { printf("%c",str[i-1]); } } } getch(); } Want to become a Data Engineer? Check out below blog posts  1.  5 Key Skills Every Data Engineer needs in 2023 2.  How to prepare for Data Engineering Interviews 3.  Top 25 Data Engineer Questions
2 comments
Read more

Programs and Puzzles in technical interviews i faced

I have attended interview of nearly 10 companies in my campus placements and sharing their experiences with you,though i did not got selected in any of the companies but i had great experience facing their interviews and it might help you as well in preparation of interviews.Here are some of the puzzles and programs asked to me in interview in some of the good companies. 1) SAP Labs I attended sap lab online test in my college through campus placements.It had 3 sections,the first one is usual aptitude questions which i would say were little tricky to solve.The second section was Programming test in which you were provided snippet of code and you have to complete the code (See Tricky Code Snippets  ).The code are from different data structures like Binary Tree, AVL Tree etc.Then the third section had questions from Database,OS and Networks.After 2-3 hours we got the result and i was shortlisted for the nest round of interviews scheduled next day.Then the next day we had PPT of t...
4 comments
Read more