Reliable data platforms need more than job-success notifications. A pipeline can complete successfully and still publish incomplete, duplicated, stale, or incorrect data. Data observability combines operational telemetry with data-level checks and lineage so teams can detect problems, understand their impact, and recover quickly.
For an interview, connect monitoring to outcomes: can the business trust the data, is it available when expected, and can the team diagnose issues before downstream consumers are affected?
Describe the signal, define the expected behavior or SLO, explain how you detect a deviation, show how you trace impact, and finish with alerting and recovery. Separate system health from data correctness.
1. How would you design observability for hundreds of data pipelines?
Short answer: Build a standardized observability layer that collects pipeline execution telemetry, data-quality measurements, freshness signals, lineage, and business-impact metadata. Use common instrumentation and service-level objectives rather than designing monitoring independently for every DAG or job.
Reference architecture
- Standard instrumentation: Every pipeline emits structured events such as run ID, pipeline ID, source and target, start/end time, status, records read/written, bytes processed, retry count, and error category.
- Central metadata store: Store run history, asset metadata, quality results, ownership, schedules, and dependencies in a queryable repository.
- Metrics and logs: Export operational metrics to a monitoring platform and retain searchable structured logs with correlation IDs.
- Data-level checks: Validate freshness, volume, schema, uniqueness, null rates, referential integrity, and business rules at critical boundaries.
- Lineage and impact analysis: Track upstream-to-downstream relationships between sources, transformations, tables, and business metrics.
- Alert routing: Route actionable incidents to the owning team, with severity based on impact and time remaining before an SLA breach.
- Dashboards and runbooks: Provide platform-wide health views, team-level views, pipeline drill-downs, and documented recovery steps.
{
"pipeline_id": "orders_daily",
"run_id": "2026-10-13T02:00Z_01",
"status": "failed",
"source": "raw.orders",
"target": "mart.orders",
"records_read": 1250000,
"records_written": 0,
"duration_seconds": 842,
"retry_count": 2,
"error_category": "schema_mismatch",
"owner": "commerce-data"
}
This is illustrative JSON. Standard fields let dashboards and alert rules work across many pipelines without parsing each job's custom log format.
Design principles at scale
- Use reusable instrumentation: Provide a shared library, orchestration callbacks, or platform wrapper so teams do not reinvent telemetry.
- Define tiers: Monitor critical revenue, finance, or customer-facing data more strictly than experimental or low-priority pipelines.
- Control metric cardinality: Use pipeline, environment, and team labels carefully. Avoid unbounded labels such as raw customer IDs or unique run IDs in metrics; keep those in logs or traces.
- Track dependencies: A single upstream incident can affect dozens of downstream assets. Group symptoms around the root cause.
- Design for self-service: Let owners see the last successful run, freshness, recent changes, quality failures, lineage, and runbook in one place.
Interview takeaway: Standardize collection and ownership, measure both pipeline health and data health, and prioritize incidents by downstream business impact.
2. What metrics would you monitor for a streaming pipeline?
Streaming systems need measurements that reveal whether events are arriving, being processed, and reaching consumers on time. A green worker dashboard is not enough if event-time lag is increasing or the sink is falling behind.
| Category | Metrics | What they tell you |
|---|---|---|
| Throughput | Events/records in and out per second, bytes/sec | Whether input rate changes or output falls behind |
| Lag and latency | Consumer lag, processing latency, end-to-end event-to-sink latency, watermark lag | Whether data is being delivered within its freshness target |
| Reliability | Failed records, retries, restarts, checkpoint failures, dead-letter volume | Whether the pipeline is unstable or silently skipping events |
| Capacity | CPU, memory, GC, backpressure, worker utilization, queue depth | Whether compute or downstream capacity is the bottleneck |
| Data quality | Duplicate rate, invalid events, schema violations, nulls, late-arriving events | Whether the delivered records are usable and conform to expectations |
| Sink health | Write latency, throttling, commit failures, rejected rows | Whether the destination is limiting end-to-end progress |
Understand the difference between lag and latency
- Consumer lag often measures how far a consumer is behind the latest available position in a broker or source system.
- Event-time delay measures how late an event is relative to its event timestamp or expected business time.
- End-to-end latency measures the elapsed time from event creation or ingestion to availability in the destination.
- Watermark lag helps show how far event-time progress trails the observed data, depending on the streaming framework.
For example, a Kafka consumer may have low offset lag but still deliver stale business data if the source itself stopped publishing events. Monitor source arrival and end-to-end freshness alongside consumer lag.
3. How would you distinguish a pipeline failure from a data quality failure?
A pipeline failure means the process did not execute or complete as expected—for example, a task crashed, credentials expired, or a destination rejected writes. A data quality failure means the process may have completed successfully, but its output violates correctness or completeness expectations.
| Signal | Pipeline failure | Data quality failure |
|---|---|---|
| Execution status | Failed, timed out, or did not start | May report success |
| Typical evidence | Exceptions, unavailable resources, failed commits, retries exhausted | Unexpected volume, duplicates, nulls, invalid ranges, reconciliation mismatch |
| Detection | Orchestrator status, logs, infrastructure metrics | Assertions, profiling, reconciliation, business rules, anomaly detection |
| Response | Repair dependency/configuration, retry safely, recover the run | Quarantine or block publication, investigate source or transformation, correct and replay |
Design for both types of failure
- Track execution status and data validation status as separate fields.
- Run quality checks at important boundaries: after ingestion, after major transformations, and before publishing critical data products.
- Use explicit thresholds and contracts where possible, such as uniqueness of a business key or a required freshness interval.
- Make publication behavior clear: either block a critical dataset from becoming available, or publish with an explicit degraded-quality status when the business has approved that policy.
- Record the failed rule, affected partition or time range, observed value, expected value, and owner.
Example: A scheduled job loads 2 million rows and exits successfully, but the expected volume was between 8 and 10 million. The orchestrator reports success; a volume anomaly should raise a data-quality incident. Conversely, a job that never writes because its credentials expired is an execution failure.
Interview takeaway: Job success is not equivalent to trustworthy data. Track execution health and output quality independently, and connect both to the dataset's publication policy.
4. How would you trace an incorrect metric back to its source?
Start with the incorrect metric and trace backward through its semantic definition, downstream model, transformations, intermediate tables, and source events. Use lineage, run metadata, version history, and reproducible checks to narrow the investigation.
A systematic debugging workflow
- Confirm the symptom: Clarify which metric is wrong, the affected time range, expected behavior, actual value, and who consumes it.
- Verify the definition: Check the metric's SQL, filters, join keys, time zone, aggregation grain, late-arriving data rules, and deduplication logic.
- Trace lineage: Identify upstream tables, source systems, transformation jobs, and the last successful run for each dependency.
- Find the first point of divergence: Compare row counts, key coverage, null rates, aggregates, and checksums at each layer. The earliest layer where values diverge is usually the most useful starting point.
- Check recent changes: Review code releases, schema changes, source changes, configuration updates, backfills, and table versions around the incident time.
- Reproduce and isolate: Re-run a bounded time range or sample with a known input snapshot. Compare old and new transformation logic.
- Correct and validate: Fix the root cause, replay the affected data idempotently, and reconcile the metric against an independent or trusted reference where available.
- Prevent recurrence: Add a targeted quality test, lineage metadata, regression case, or alert tied to the failure mode.
Business metric: daily net revenue
↓
Semantic model: revenue = SUM(order_total)
↓
Mart: orders joined to order_items
↓
Transformation: one order now matches multiple item rows
↓
Root cause: order_total is summed after a one-to-many join
↓
Fix: aggregate at the correct grain before joining,
or calculate revenue from item-level amounts
In this example, the pipeline may run without errors. The problem is a grain mismatch that duplicates an order-level amount after a one-to-many join. Observability needs semantic and data-level checks—not just job logs—to catch it.
Useful evidence to preserve
- Pipeline run IDs, code version, configuration, and input/output snapshot identifiers.
- Row counts and key-level reconciliation at each major transformation.
- Data lineage and the list of affected downstream assets.
- Quality-test results, schema history, and source change timestamps.
Interview takeaway: Trace from the business metric backward, find the earliest divergence, reproduce it with a known input, and fix the root cause rather than patching only the final dashboard.
5. How would you reduce alert fatigue?
Alert fatigue happens when teams receive too many low-value, duplicate, noisy, or non-actionable notifications. The solution is not simply to suppress alerts; it is to make alerts correspond to meaningful symptoms, clear ownership, and a specific action.
Practical techniques
- Alert on user impact: Prioritize missed freshness targets, failed critical data products, and imminent SLA breaches over every transient retry.
- Group correlated symptoms: If one upstream table fails and 40 downstream pipelines fail as a result, create a root-cause incident with affected assets listed rather than 40 separate pages.
- Use severity tiers: Page for urgent production impact; send lower-priority anomalies to a ticket, dashboard, or scheduled digest.
- Add persistence windows: Require a signal to remain abnormal for a suitable interval when a brief spike is not meaningful. Do not add delay to alerts where immediate action is required.
- Apply deduplication and cooldowns: Avoid repeatedly paging for the same unresolved incident, while still escalating if impact grows.
- Set ownership and runbooks: Every actionable alert should identify the affected asset, likely failure category, owner, impact, and first diagnostic steps.
- Review alert quality: Track alert-to-incident ratio, duplicate pages, false positives, time to acknowledge, and incidents discovered by users before monitoring.
- Use dynamic baselines carefully: Seasonal or volume-sensitive datasets may need time-aware thresholds instead of a single fixed row-count threshold.
Interview takeaway: Alert on actionable impact, group related failures, define ownership, and continuously measure alert usefulness. Suppression should reduce noise without hiding real incidents.
6. How would you measure data freshness and reliability?
Define freshness and reliability as explicit service-level objectives for each data product. Avoid relying on pipeline duration alone: a fast pipeline can publish stale data, and a pipeline can finish on time while producing incorrect records.
Freshness metrics
- End-to-end freshness lag: Current time minus the event or source timestamp of the latest expected data available in the target.
- Arrival delay: Time between when data was produced at the source and when it arrived in the destination.
- Schedule delay: Actual successful publication time compared with the expected publication time.
- Completeness window: Whether all expected partitions, source offsets, or time intervals have arrived.
Reliability metrics
- Successful publication rate: Percentage of expected data updates published successfully within the defined window.
- Freshness SLO attainment: Percentage of measurement windows in which freshness remained within its target.
- Data quality pass rate: Percentage of required validations that pass, reported with severity and affected scope.
- Completeness and reconciliation: Expected versus actual rows, keys, totals, or source offsets.
- Recovery time: Time to detect, diagnose, restore correct data, and make it available to consumers.
- Incident recurrence: Rate of repeat failures from the same root cause.
| Data product tier | Illustrative freshness target | Monitoring approach |
|---|---|---|
| Critical operational | Minutes | Near-real-time lag, completeness, quality, and sink health alerts |
| Daily finance or executive reporting | Published by an agreed morning deadline | Deadline monitoring, reconciliation, and explicit publication status |
| Exploratory analytics | Hours or daily | Dashboard trends, anomaly detection, and lower-severity alerts |
These targets are examples only. Set real targets with data consumers based on business decisions, source cadence, recovery expectations, and cost.
Example SLO calculation
If a critical data product must be fresh within 30 minutes, define a measurement window and calculate the proportion of eligible windows where the end-to-end freshness lag is at most 30 minutes.
Illustrative formula
Freshness SLO attainment = compliant measurement windows ÷ total eligible measurement windows × 100%
For a source that is expected to publish continuously, source inactivity must be distinguished from pipeline delay. For a daily batch, freshness should be measured against the expected publication deadline and the data's business timestamp—not merely the last file modification time.
Interview takeaway: Measure freshness end to end, define reliability at the data-product level, and publish SLOs that reflect consumer needs. Include correctness and completeness, not just job completion.
Quick Revision Cheat Sheet
- Observability at scale: Standard telemetry, centralized metadata, quality checks, lineage, ownership, and runbooks.
- Streaming: Track throughput, consumer lag, end-to-end latency, watermarks, backpressure, failures, and sink health.
- Pipeline vs quality failure: Track execution state separately from data validation state.
- Trace incorrect metrics: Start from the metric definition, follow lineage backward, and find the earliest divergence.
- Reduce alert fatigue: Prioritize impact, group root causes, route by ownership, and measure false positives.
- Freshness: Measure source-to-target delay and expected publication deadlines.
- Reliability: Track SLO attainment, completeness, quality, recovery time, and repeat incidents.
- Key principle: A successful job does not automatically mean the data is correct or fresh.
Scenario-Based Interview Practice
Try to answer each scenario aloud in two to three minutes before expanding the discussion points.
Scenario 1: Hundreds of pipelines fail at the same time
Discuss: Use lineage and timestamps to identify a shared upstream source, credential, network, or platform dependency. Group downstream symptoms into one root-cause incident, identify affected critical assets, and track recovery through freshness SLOs.
Scenario 2: A streaming job is healthy but the dashboard is stale
Discuss: Check source arrival, broker lag, processing latency, watermark progress, sink write latency, commit status, and dashboard cache or refresh behavior. Trace freshness end to end instead of trusting worker health alone.
Scenario 3: Revenue changes unexpectedly after a successful deployment
Discuss: Confirm the metric definition and grain, inspect recent code and schema changes, compare intermediate aggregates, identify the first divergence, validate with a reproducible input snapshot, and backfill affected dates after correcting the cause.
Scenario 4: Teams receive too many low-value alerts
Discuss: Categorize alerts by impact and actionability, consolidate correlated symptoms, use severity tiers, tune thresholds against historical behavior, add ownership and runbooks, and measure duplicate pages and false-positive rates.
Practice Challenge
Design observability for a platform with 300 batch pipelines, 20 streaming pipelines, and several executive dashboards. The business expects critical revenue data to be available by 7:00 AM, while streaming customer events should appear within five minutes.
- Which telemetry fields would every pipeline be required to emit?
- How would you define different SLOs for batch and streaming assets?
- How would you distinguish a source outage from a stuck consumer?
- How would lineage help identify the dashboards affected by a failure?
- Which incidents should page an on-call engineer versus create a ticket?
- How would you measure whether the observability platform is improving reliability?
A strong answer connects metrics to user impact, distinguishes execution health from data correctness, and explains how the platform detects, diagnoses, and recovers from failures.
Comments
Post a Comment