Data Engineering & AI Interview Playbook

Stream processing continuously handles events as they arrive. The difficult part is not simply processing each message quickly: production systems must also deal with delayed and out-of-order events, duplicate delivery, state recovery, uneven traffic, and accurate results after failures. Interview framework: Discuss event-time semantics, delivery guarantees, state and checkpoints, windowing, late-data policy, observability, and the trade-off between result completeness and result latency. 1. How does stream processing differ from batch processing? Batch processing works on a bounded dataset, usually on a schedule. Stream processing handles an unbounded sequence of events continuously, updating results as new events arrive. Dimension Batch Stream Input Bounded files, tables, or time ranges Continuous events or records Output timing After scheduled processing completes Continuously or at short intervals State Often scoped to a run or part...

Batch processing remains central to data engineering, even in organizations that also use streaming. Interviews often test how you balance throughput, cost, freshness, reliability, and recovery when workloads process millions or billions of records. Interview framework: Explain the workload and SLA first, then cover partitioning, parallelism, data skew, idempotency, observability, and recovery. For performance questions, identify the bottleneck before proposing a fix. 1. When would you choose batch processing over streaming? Batch processing handles a bounded collection of data on a schedule or when a defined amount of data is available. Streaming continuously processes events as they arrive, often with lower latency but more complex state and operational requirements. Factor Batch Streaming Latency Minutes to hours, depending on schedule and duration Often seconds or milliseconds when designed for it Operational model Bounded runs, retries, a...

Change Data Capture (CDC) is a common way to keep analytical platforms synchronized with operational databases without repeatedly extracting entire tables. CDC interviews test whether you understand transaction logs, ordering, deletes, checkpoints, recovery, and how to make target state correct despite retries and failures. Interview framework: When discussing CDC, cover the capture mechanism, event metadata, ordering guarantees, checkpointing, delete semantics, idempotent target application, and recovery strategy. 1. How does log-based CDC work? Log-based CDC reads a database's transaction log—such as a write-ahead log or binary log—to capture committed inserts, updates, and deletes. Instead of repeatedly querying application tables, a CDC connector reads the change stream and converts log entries into ordered change events. Typical flow: The source database records transactions in its transaction log. A CDC connector reads the log from a saved log ...

Data engineering interviews often focus on what happens when pipelines encounter real-world problems: late-arriving records, retries, partial failures, schema changes, historical backfills, and mismatched counts. Strong answers show how you make ingestion reliable, observable, recoverable, and reusable. Interview framework: For every pipeline design question, cover the source, extraction strategy, checkpointing, idempotency, failure recovery, data quality, observability, and security. 1. When would you choose ETL over ELT? ETL means Extract, Transform, Load: data is transformed before being loaded into the target. ELT means Extract, Load, Transform: raw data is loaded first, then transformed in the destination platform. Factor ETL ELT Transformation location Before loading into the target Inside the destination platform Common reason to use Sensitive-data filtering, strict target constraints, specialized transformation engines Scalab...