Reliable data platforms need more than job-success notifications. A pipeline can complete successfully and still publish incomplete, duplicated, stale, or incorrect data. Data observability combines operational telemetry with data-level checks and lineage so teams can detect problems, understand their impact, and recover quickly. For an interview, connect monitoring to outcomes: can the business trust the data, is it available when expected, and can the team diagnose issues before downstream consumers are affected? Answer framework Describe the signal, define the expected behavior or SLO, explain how you detect a deviation, show how you trace impact, and finish with alerting and recovery. Separate system health from data correctness. 1. How would you design observability for hundreds of data pipelines? Short answer: Build a standardized observability layer that collects pipeline execution telemetry, data-quality measurements, freshness signals, lineage, ...
A data lake can store enormous volumes of inexpensive data, but raw files alone do not automatically provide the reliability and query behavior expected from a database. Once multiple jobs read and modify the same datasets, teams need consistent snapshots, safe writes, metadata management, schema controls, and a way to recover earlier versions. Transactional table formats add these capabilities over data stored in object storage. They are a key building block of the lakehouse architecture , which aims to combine data-lake flexibility with warehouse-like management and reliability. Interview framework For each answer, explain the problem first, describe the mechanism, discuss operational trade-offs, and finish with how you would choose or validate a solution in production. 1. Why do data lakes need a transactional table format? Short answer: Object storage stores files, but it does not by itself provide table-level transactions, consistent snapshots, atom...