Data Engineering & AI Interview Playbook

Reliable data platforms need more than job-success notifications. A pipeline can complete successfully and still publish incomplete, duplicated, stale, or incorrect data. Data observability combines operational telemetry with data-level checks and lineage so teams can detect problems, understand their impact, and recover quickly. For an interview, connect monitoring to outcomes: can the business trust the data, is it available when expected, and can the team diagnose issues before downstream consumers are affected? Answer framework Describe the signal, define the expected behavior or SLO, explain how you detect a deviation, show how you trace impact, and finish with alerting and recovery. Separate system health from data correctness. 1. How would you design observability for hundreds of data pipelines? Short answer: Build a standardized observability layer that collects pipeline execution telemetry, data-quality measurements, freshness signals, lineage, ...

A data lake can store enormous volumes of inexpensive data, but raw files alone do not automatically provide the reliability and query behavior expected from a database. Once multiple jobs read and modify the same datasets, teams need consistent snapshots, safe writes, metadata management, schema controls, and a way to recover earlier versions. Transactional table formats add these capabilities over data stored in object storage. They are a key building block of the lakehouse architecture , which aims to combine data-lake flexibility with warehouse-like management and reliability. Interview framework For each answer, explain the problem first, describe the mechanism, discuss operational trade-offs, and finish with how you would choose or validate a solution in production. 1. Why do data lakes need a transactional table format? Short answer: Object storage stores files, but it does not by itself provide table-level transactions, consistent snapshots, atom...

Modern cloud data warehouses offer different ways to scale, optimize, secure, and price analytical workloads. Strong interview answers connect platform architecture to query plans, data layout, workload patterns, recovery requirements, and cost. A framework for answering warehouse questions Explain the architecture, mechanism, and trade-offs. For performance and cost questions, use query history, execution plans, bytes scanned, queue time, spills, and workload-specific SLAs rather than generic rules. 1. Compare Snowflake, BigQuery, and Redshift architectures. All three are cloud data warehouse platforms, but they differ in compute abstractions, storage organization, workload scaling, and operational controls. Aspect Snowflake BigQuery Redshift Core model Managed storage and virtual warehouses for compute Serverless or provisioned capacity over managed storage Provisioned clusters or serverless capacity Workload isolation Separate virtual warehouses Reservations, slots, editions...

Apache Airflow is a workflow orchestration platform used to author, schedule, execute, and monitor data pipelines. Strong interview answers show that you understand both the Airflow control plane and the operational design choices that make workflows reliable. A framework for answering Airflow questions Explain the scheduler or task lifecycle, state the failure and retry behavior, then describe how you make the workflow observable, repeatable, and safe to operate. Mention that some details vary by Airflow version and executor. 1. How does Airflow schedule and execute a DAG? Short answer: A DAG defines tasks and their dependencies. The scheduler evaluates DAG runs and task dependencies, creates runnable task instances, and sends eligible tasks to the configured executor, which runs them on workers or execution infrastructure. DAG definition: Python code describes tasks, schedules, dependencies, retries, and other settings. A DAG represents workflow structure rather than th...