What is Apache Kafka?

-
Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that sends messages between applications, processes, and servers. It is composed of three important components called producer (which acts as Sender), Topics (topic or category of messages), and Consumer (which acts as a Reciever).

Kafka aims to provide high-throughput, low latency for real-time data feeds. It is widely used for real-time streams analytics, ingestion data into Spark, Complex Event Processing, Log aggregation, etc.


Before understanding Kafka Architecture, lets first understand what Kafka Broker is

Kafka Broker:

 As Kafka is a distributed framework, Kafka's cluster consists of different servers called Brokers running Kafka. Producers publish the message to Kafka Topics within this Broker and consumer consumes this message from Topics.

Kafka Architecture:

Kafka Architecture is comprised of mainly 3 components


  1. Kafka Producer
  2. Kafka Topics
  3. Kafka Consumer


Each Topic has Partitions, which contains messages in non-changing sequences. Each message is identified using a unique offset. This helps multiple consumers to read from topics at the same time. Each Topics partition can replicate the messages over several Kafka Brokers in the cluster.

Kafka also has a Consumer Group in which different consumers are assigned to consume messages from different partitions of specific Topics. They will receive the messages from a different subset of the partitions in Topic.

In the next posts, I will discuss more Kafka Producers and Consumers and provide JAVA code with explanations for writing Kafka Producer and Consumer.

Also, check out more posts below:



Comments

Post a Comment

Popular posts from this blog

Tricky Questions or Puzzles in C

-
This post is about the Tricky Questions   or code snippet in C or C++ asked in most of the Interviews   (See Interview Experience ) by some of the good companies. You may know probably the right concept but it will not strike you at the interview and to crack all those Interview Questions  you should know some of them beforehand. If you are applying for Job related to JAVA then checkout the blog post  Tricky Questions in JAVA . 1) what will be the output of the following Printf function.   printf("%d",printf("%d",printf("%d",printf("%s","ILOVECPROGRAM")))); Ans-ILOVECPROGRAM1321 The above printf line gives output like this because printf returns the number of character successfully written in the output. So the inner printf("%s","ILOVECPROGRAM") writes 13 characters to the output so the outer printf function will print 13 and as 13 is of 2 characters so the next outer printf function will print 2 and then
Read more

Program to uncompress a string ie a2b3c4 to aabbbcccc

-
Below is the program to uncompress a string #include<stdio.h> #include<conio.h> #include<stdlib.h> int main() { char str[100]="a2b3c4d8u7"; for(int i=0;str[i]!='\0';i++) { if(i%2!=0) { for(int j=0;j<atoi(&str[i]);j++) { printf("%c",str[i-1]); } } } getch(); } Want to become a Data Engineer? Check out below blog posts  1.  5 Key Skills Every Data Engineer needs in 2023 2.  How to prepare for Data Engineering Interviews 3.  Top 25 Data Engineer Questions
Read more

Number series arrangement puzzles

-
In this post I am going to give the number series arrangement puzzles in which you have to find the next element or missing element in the series of numbers given. These questions are an important part of any technical interviews where interviewer wants to check how fast you can think and deduce connections between numbers. 1. What is the next number in the series? 27  101  91 82 18 17 __ 2. What is the next number in the series? 83  99   1027    1181    12243  __ 3. What is the missing number in the series? 132   3134    5236    7338   __   115312 4. What is the missing numbers in the series? 31 34 __ 37 37 __ 40 40 40 __ 40 40 40 5. What is the next number in the series? 58   1024   2040    4056   __ Click  here  for answers
Read more