Kafka is a distributed event-streaming platform used for real-time data pipelines and event-driven systems. Interviews test whether you understand partition-based architecture and how delivery guarantees behave in real applications.
A framework for answering Kafka questions
Explain the mechanism, state the guarantee and its scope, describe the trade-off, and give a production example. Separate broker durability, producer delivery, consumer processing, and downstream side effects.
1. How does Kafka achieve scalability and fault tolerance?
Short answer: Kafka scales by splitting topics into partitions distributed across brokers. It protects data using replicated partitions and leader election, subject to the replication and acknowledgement configuration.
- Partitions: A topic is divided into partitions so reads and writes can be distributed across brokers.
- Parallel consumers: Different partitions can be processed concurrently by consumers in a group. A partition is assigned to at most one consumer in a group at a time.
- Horizontal growth: Additional brokers can increase capacity, but partition reassignment and workload balance matter.
- Replication: Each partition can have a leader and follower replicas on other brokers. If the leader fails, Kafka can elect an eligible in-sync replica.
- Durability settings: Replication factor,
min.insync.replicas, and produceracksaffect the durability guarantee.
replication.factor=3
min.insync.replicas=2
# Producer configuration:
acks=all
enable.idempotence=trueThese settings are commonly used to strengthen durability and producer retry behavior, but they must be configured together appropriately. With acks=all, the broker acknowledges after the required in-sync replicas have acknowledged the write, subject to topic and broker configuration.
Limitation: Replication is not a substitute for backups and does not prevent every possible data-loss scenario. Durability depends on replica placement, ISR behavior, configuration, and operational practices.
2. Why is ordering guaranteed only within a partition?
Kafka stores each partition as an ordered log. Offsets establish record order within that partition. A multi-partition topic has no single built-in total order across all its records.
- Within a partition: Consumers read records in offset order.
- Across partitions: Partitions advance independently and can be processed at different speeds.
- Why this design: Independent partitions provide parallelism and throughput.
If events for one entity must remain ordered, use a stable key such as order_id or account_id. With a consistent partitioning setup, records with the same key normally route to the same partition.
producer.send(
"orders",
key=order_id.encode("utf-8"),
value=event_bytes
)Interview nuance: Ordering depends on keeping related records on the same partition and on producer behavior. Increasing the partition count can change key-to-partition mapping for common partitioners, so plan expansion carefully when per-key ordering is critical.
3. How would you choose a partition key?
Choose a key that preserves the ordering scope required by the business while distributing traffic and data reasonably evenly.
- Ordering requirement: Choose the entity whose events must be ordered, such as
customer_id,order_id, ordevice_id. - Cardinality: More distinct keys usually provide more opportunities for distribution than a low-cardinality key.
- Traffic distribution: A hot key can overload one partition even if the topic has many partitions.
- Consumer parallelism: The number of partitions bounds the number of consumers in one group that can actively own a partition at a time.
- Null/default values: Avoid routing a large share of events to one default key.
- Future growth: Consider throughput, retention, consumer count, and the implications of increasing partitions.
| Use case | Potential key | Watch out for |
|---|---|---|
| Order lifecycle | order_id | A burst for one order stays on one partition |
| Device telemetry | device_id | A small number of unusually chatty devices |
| Country-level metrics | Potentially a composite or higher-cardinality key | Country alone may create hot partitions |
Rule of thumb: Start with ordering semantics, then validate key distribution against real or representative traffic. Do not choose a key only for convenience.
4. What causes consumer lag, and how would you troubleshoot it?
Consumer lag is the gap between the latest available offset and a consumer group's committed or processed position, depending on the monitoring tool. It is usually measured per partition.
Common causes:
- Incoming rate exceeds processing throughput.
- Slow databases, APIs, object storage, or other downstream dependencies.
- CPU-heavy transformations, inefficient deserialization, synchronous I/O, or oversized batches.
- Insufficient partitions/consumers or uneven partition assignment.
- Hot partitions, frequent rebalances, long garbage-collection pauses, repeated errors, or retries.
- Producer traffic spikes or consumers that are stopped or not progressing offsets.
How to investigate:
- Plot lag over time per partition; determine whether it is growing, stable, or recovering.
- Compare producer input rate with consumer throughput and processing latency.
- Inspect errors, retries, rebalances, CPU, memory, and downstream latency.
- Look for partitions with much higher lag than others.
- Estimate catch-up capacity. Adding consumers helps only if there are enough partitions and the bottleneck is parallelizable.
- Fix the root cause and verify that lag decreases without harming correctness or downstream systems.
5. How do consumer groups and rebalancing work?
A consumer group is a set of consumers sharing the work of reading a topic. For a topic partition, at most one consumer in the group owns it at a time; separate groups can independently read the same topic.
- Kafka assigns topic partitions among group members.
- Consumers track progress using offsets, commonly committed to Kafka's internal offsets topic.
- When a consumer joins, leaves, fails, or subscriptions/partitions change, Kafka may redistribute partitions.
- During a rebalance, consumption may pause or assignment changes may introduce delays, depending on the protocol and strategy.
Common triggers: Deployments, autoscaling, crashes, long processing pauses, heartbeat/session timeouts, subscription changes, or partition-count changes.
Reduce disruptive rebalances:
- Keep processing within the configured poll interval, or design appropriate pause/resume and processing behavior.
- Use graceful shutdown and stable deployments.
- Choose session, heartbeat, and poll settings based on the client and workload.
- Consider cooperative-sticky assignment or supported newer group protocols to reduce unnecessary movement.
- Make processing safe for replay because uncommitted work may be processed again.
A rebalance is normal coordination, not necessarily a failure. Frequent or prolonged rebalances can, however, reduce throughput and increase lag.
6. What is the difference between log compaction and retention?
Both are Kafka log-cleanup policies, but they preserve data differently.
| Aspect | Time/size retention | Log compaction |
|---|---|---|
| Goal | Keep records within configured age/size limits | Retain the latest value for each key over time |
| Cleanup | Older or excess segments become eligible for deletion | Older records for a key can be removed as segments are compacted |
| Typical use | Event history, telemetry, replay window | Latest entity state, changelog, state reconstruction |
| History | History limited by configured retention | Does not preserve every historical update |
For example, a customer-state topic may contain several records for key customer-42. Compaction can eventually remove older values and retain the latest value. A tombstone—a record with a key and null value—marks a key for deletion; its removal is governed by compaction and tombstone-retention behavior.
Nuance: Compaction is asynchronous, so consumers may temporarily see older records. A compacted topic is not guaranteed to contain exactly one record per key at every moment. Topics can also use a combined policy such as compaction plus deletion retention.
7. How do you handle duplicate messages?
First identify where duplicates originate: producer retries, consumer restarts before offset commit, rebalances, sink retries, or upstream systems publishing the same business event more than once.
Producer-side controls:
- Enable producer idempotence to prevent duplicate log entries caused by supported retries within the producer protocol.
- Configure acknowledgements and retries appropriately.
- Remember that producer idempotence does not deduplicate two separate application-level sends of the same business event.
Consumer and sink controls:
- Give each business event a stable unique identifier, such as
event_id. - Make downstream writes idempotent with upserts, uniqueness constraints, deduplication tables, or transactional sink behavior.
- Commit offsets consistently with the processing and sink strategy. Committing too early risks loss; committing after processing can allow replay after a crash.
- For streaming pipelines, deduplicate using a stable key and a defined time window where appropriate, accounting for late events and state growth.
# Conceptual sink pattern:
# event_id remains stable across retries.
# Upsert or enforce uniqueness so replay is safe.
upsert_event(
event_id=event["event_id"],
payload=event
)Strong interview answer: Duplicate handling is an end-to-end concern. Kafka producer idempotence helps with retry duplicates, but application-level duplicates and downstream side effects need their own strategy.
8. How would you evolve an Avro schema without breaking consumers?
Schema evolution changes a message structure while preserving compatibility between producer and consumer versions. Avro schemas describe fields and types; a schema registry can assign schema IDs and enforce compatibility policies.
- Add fields with defaults. A default can let a reader using the new schema read older records that lack the field. Verify Avro resolution and registry policy.
- Avoid incompatible type changes. A changed type or meaning can break consumers.
- Do not casually rename or remove fields. Consider aliases or a staged migration where appropriate.
- Test compatibility in CI. Validate candidate schemas against policy and representative old/new payloads.
- Deploy in a compatible order. A common approach is to deploy consumers that tolerate both forms before producers emit the new field.
- Preserve semantic compatibility. A structurally valid field can still break business logic if its meaning changes.
{
"type": "record",
"name": "OrderEvent",
"namespace": "com.example.events",
"fields": [
{"name": "order_id", "type": "string"},
{"name": "status", "type": "string"},
{"name": "source_system", "type": "string", "default": "unknown"}
]
}Here, source_system has a default so a reader using this schema can resolve older records without that field. Compatibility depends on the exact old/new reader and writer schemas and registry policy.
Know the modes: Backward compatibility means a new reader can read data written with an older schema. Forward compatibility means an older reader can read data written with a newer schema. Full compatibility aims to support both directions. Transitive modes check against more than the latest registered version.
9. What does exactly-once processing mean, and what are its limitations?
Exactly-once processing aims to ensure each input record's effects are reflected once in the defined result, even when failures and retries occur. The term is meaningful only when the processing boundary and guarantees are clear.
- Idempotent producer: Prevents duplicate log writes caused by supported producer retries within the Kafka producer protocol.
- Transactions: Can atomically write to multiple Kafka partitions and commit consumed offsets as part of a Kafka transaction, supporting consume-process-produce workflows with read-committed consumers.
- Offset management: Coordinating offsets with output writes is critical; independent commits can create gaps or duplicate processing after failures.
- External systems: A database, API, object store, or email service is not automatically included in a Kafka transaction. Additional transactional integration, idempotency, or deduplication is needed.
| Delivery model | Meaning | Main concern |
|---|---|---|
| At-most-once | Processed zero or one time | Committing before processing can lose work |
| At-least-once | Retry until success under the design's assumptions | Reprocessing can duplicate effects |
| Exactly-once | Defined result reflects each input effect once within a supported boundary | External effects may need separate controls |
For example, Kafka Streams can use transactions to atomically commit output records and input offsets. If an application consumes from Kafka and updates an unrelated database, Kafka transactions alone do not make that database update exactly-once.
Interview takeaway: Exactly-once is an end-to-end design property, not a magic broker setting. Specify the source, processing framework, output sink, transaction boundary, and failure scenarios.
Quick Revision Cheat Sheet
| Topic | Remember |
|---|---|
| Scalability and fault tolerance | Partitions distribute work; replicas and acknowledgements affect durability |
| Ordering | Offsets establish order within a partition, not across a whole topic |
| Partition key | Balance ordering, key cardinality, skew, and consumer parallelism |
| Consumer lag | Compare input rate, throughput, per-partition lag, errors, and downstream latency |
| Consumer groups | Partitions are assigned within a group; membership changes can trigger rebalances |
| Compaction vs retention | Retention deletes by age/size; compaction preserves latest values per key over time |
| Duplicates | Use producer idempotence, stable event IDs, and idempotent sinks |
| Avro evolution | Defaults, compatibility direction, registry checks, and CI tests |
| Exactly-once | Define the boundary; Kafka transactions do not automatically cover external systems |
Scenario-Based Interview Practice
Try answering each scenario out loud before expanding the suggested approach.
Scenario 1: Consumer lag is rising, but CPU usage is low. What would you investigate?
Check downstream I/O, retries, partition assignment, per-partition lag, input rate, processing latency, rebalance frequency, and downstream health before scaling consumers.
Scenario 2: All events for one customer must be processed in order. How would you design the topic?
Use a stable customer identifier as the key, inspect hot-customer traffic, preserve producer ordering assumptions, and plan partition-count changes carefully because key mapping can change.
Scenario 3: A consumer writes to a database and crashes before committing its Kafka offset. What can happen?
The record may be delivered again, duplicating the database effect. Use idempotent upserts or unique event IDs, or an appropriate transactional integration, and coordinate offset handling with sink commit semantics.
Scenario 4: A producer wants to add a required field to an Avro event. How do you roll it out safely?
Check registry compatibility and old/new reader-writer resolution. Roll out consumers that tolerate both forms before producers rely on the new field, use a default where appropriate, and test old/new payloads in CI.
Chapter Summary
Strong Kafka engineers understand the partitioned log model and the application-level choices needed for reliable processing.
- Use partitions for parallelism and replicas for resilience.
- Choose keys based on ordering requirements and real traffic distribution.
- Troubleshoot lag with per-partition metrics and end-to-end latency.
- Design for duplicate delivery and safe retries at the sink.
- Validate schema compatibility and define the boundary of exactly-once guarantees.
Practice Challenge
Design a Kafka pipeline that ingests order events, validates them with Avro, processes them with a consumer group, and writes results to a database. Explain partition-key selection, replication and acknowledgement settings, lag monitoring, duplicate protection, schema rollout, offset handling, and what exactly-once means for your sink.
Comments
Post a Comment