Stream processing continuously handles events as they arrive. The difficult part is not simply processing each message quickly: production systems must also deal with delayed and out-of-order events, duplicate delivery, state recovery, uneven traffic, and accurate results after failures. Interview framework: Discuss event-time semantics, delivery guarantees, state and checkpoints, windowing, late-data policy, observability, and the trade-off between result completeness and result latency. 1. How does stream processing differ from batch processing? Batch processing works on a bounded dataset, usually on a schedule. Stream processing handles an unbounded sequence of events continuously, updating results as new events arrive. Dimension Batch Stream Input Bounded files, tables, or time ranges Continuous events or records Output timing After scheduled processing completes Continuously or at short intervals State Often scoped to a run or part...
Batch processing remains central to data engineering, even in organizations that also use streaming. Interviews often test how you balance throughput, cost, freshness, reliability, and recovery when workloads process millions or billions of records. Interview framework: Explain the workload and SLA first, then cover partitioning, parallelism, data skew, idempotency, observability, and recovery. For performance questions, identify the bottleneck before proposing a fix. 1. When would you choose batch processing over streaming? Batch processing handles a bounded collection of data on a schedule or when a defined amount of data is available. Streaming continuously processes events as they arrive, often with lower latency but more complex state and operational requirements. Factor Batch Streaming Latency Minutes to hours, depending on schedule and duration Often seconds or milliseconds when designed for it Operational model Bounded runs, retries, a...