9 Apache Airflow Interview Questions and Answers

Apache Airflow is a workflow orchestration platform used to author, schedule, execute, and monitor data pipelines. Strong interview answers show that you understand both the Airflow control plane and the operational design choices that make workflows reliable.

A framework for answering Airflow questions

Explain the scheduler or task lifecycle, state the failure and retry behavior, then describe how you make the workflow observable, repeatable, and safe to operate. Mention that some details vary by Airflow version and executor.

1. How does Airflow schedule and execute a DAG?

Short answer: A DAG defines tasks and their dependencies. The scheduler evaluates DAG runs and task dependencies, creates runnable task instances, and sends eligible tasks to the configured executor, which runs them on workers or execution infrastructure.

1. DAG definition: Python code describes tasks, schedules, dependencies, retries, and other settings. A DAG represents workflow structure rather than the data itself.
2. DAG run creation: A scheduled DAG is evaluated according to its timetable and data interval. Manual or externally triggered runs can also be created.
3. Dependency evaluation: The scheduler checks upstream states, trigger rules, pools, concurrency limits, and other constraints.
4. Task scheduling: Eligible task instances are queued and handed to the executor.
5. Execution: A worker or execution environment runs the task, records logs and state, and reports completion or failure.
6. Downstream progression: As task states change, dependent tasks may become eligible to run.

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def extract():
    print("Extract source data")

def transform():
    print("Transform data")

with DAG(
    dag_id="daily_customer_pipeline",
    start_date=datetime(2026, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:
    extract_task = PythonOperator(
        task_id="extract",
        python_callable=extract,
    )
    transform_task = PythonOperator(
        task_id="transform",
        python_callable=transform,
    )

    extract_task >> transform_task

Key distinction: The scheduler decides what is eligible; the executor determines how tasks are launched. Airflow orchestrates work but is not intended to process large datasets directly inside the scheduler.

2. What happens when a task fails and is retried?

When a task execution fails, Airflow records the attempt and evaluates its retry configuration. If retries remain and the retry delay has elapsed, the task can be scheduled again. If retries are exhausted, the task generally ends in a failed state and downstream behavior depends on dependencies and trigger rules.

• retries: Number of additional attempts after the initial attempt.
• retry_delay: Delay between attempts; exponential backoff can be configured where supported.
• Logs: Inspect the task log for the failing attempt, exception, dependency error, or external system response.
• Downstream tasks: Tasks using the default all-success behavior do not run while required upstream tasks have failed, unless the workflow uses different trigger rules.
• Side effects: A task may have partially completed work before failing. A retry can repeat that work.

from datetime import timedelta
from airflow.operators.python import PythonOperator

load_task = PythonOperator(
    task_id="load_data",
    python_callable=load_data,
    retries=3,
    retry_delay=timedelta(minutes=5),
)

Interview tip: Retries improve resilience to transient errors, but they do not repair persistent failures such as invalid input, bad credentials, or a broken schema. Use clear logging, sensible retry limits, and alerts when retries are exhausted.

3. How would you design an idempotent DAG?

An idempotent task can be executed more than once with the same logical inputs without creating unintended additional effects. This is essential because retries, manual reruns, backfills, and worker failures can all repeat execution.

Design principles:

• Use deterministic data intervals: Process a specific interval or partition rather than “whatever data is new right now” without a stable boundary.
• Use stable keys: Upsert by business key or event identifier rather than blindly appending duplicate rows.
• Write atomically where possible: Stage output and commit it only after validation, or use transactional writes supported by the destination.
• Make file output repeatable: Use predictable partition paths and a deliberate overwrite/merge strategy; avoid creating a new duplicate file on every retry.
• Separate compute from side effects: Validate results before publishing them to downstream systems.
• Track completion explicitly: Use manifests, checkpoints, or run metadata where useful, while ensuring the completion marker cannot get ahead of the actual data write.

# Conceptual example: overwrite a deterministic daily partition
# instead of appending a duplicate result on every retry.
partition_date = logical_date.strftime("%Y-%m-%d")
target = f"gs://warehouse/daily_orders/date={partition_date}/"

write_partition(
    dataframe=transformed,
    target_path=target,
    mode="overwrite",
)

This is illustrative: an overwrite is safe only if the write is atomic enough for the storage system and the job owns that target partition. For databases, use transactions, merge/upsert logic, or unique constraints where appropriate.

Common mistake: A DAG is not idempotent merely because it has a fixed schedule. The tasks and their external side effects must also be safe to repeat.

4. What is the difference between catchup and backfill?

Both relate to running work for historical data intervals, but they describe different mechanisms.

Aspect	Catchup	Backfill
Meaning	Scheduler creates missing scheduled DAG runs for past intervals, when enabled	An operator intentionally runs a specified historical date range
Control	Primarily controlled by DAG configuration and scheduler behavior	Explicitly initiated for a chosen interval or range
Use case	Run missed daily intervals after a DAG becomes active	Reprocess a month after fixing a transformation bug
Risk	Unexpectedly many historical runs	Resource pressure or repeated side effects during reprocessing

Example: A daily DAG has a start date in January and is enabled in March. With catchup enabled, Airflow may create runs for eligible past intervals. A backfill explicitly targets a historical range to reprocess those intervals.

Important nuance: Exact backfill commands and behavior vary by Airflow version. Before historical runs, verify the DAG's data interval logic, task idempotency, concurrency limits, and downstream impact. Catchup is not the same as retrying a failed task.

5. How do you handle dependencies between pipelines?

Prefer explicit, observable dependencies over hidden timing assumptions. The best choice depends on whether the dependency is inside one workflow, between Airflow DAGs, or across external systems.

• Within one DAG: Use task dependencies such as task_a >> task_b.
• Between DAGs: Use supported mechanisms such as ExternalTaskSensor when the upstream DAG's schedule and data intervals align, or use dataset/asset-aware scheduling where supported by the Airflow version.
• External readiness: Use a sensor or event-driven trigger to verify that the source file, table partition, or upstream job is actually ready.
• Data contracts: Validate expected schema, freshness, row counts, or a completion marker before starting downstream work.
• Avoid fixed sleeps: Waiting a hard-coded number of minutes does not prove that upstream data is ready.
• Avoid circular dependencies: Document ownership and define a clear direction for data flow.

# Within a DAG:
extract >> validate >> publish

# Between DAGs, an ExternalTaskSensor can wait for
# a specific upstream task/run when intervals align.
# For newer Airflow versions, dataset/asset scheduling
# may be a better fit for data-aware dependencies.

Interview tip: Explain how you match the correct upstream data interval to the downstream run. A task being successful for yesterday's interval does not necessarily mean today's required data is ready.

6. How would you manage hundreds of DAGs without overwhelming the scheduler?

Scheduler scalability is a combination of efficient DAG authoring, controlled concurrency, fast parsing, and sensible operational limits.

• Keep DAG parsing lightweight: Avoid network calls, database queries, large file reads, or expensive computations at module import time.
• Generate DAGs efficiently: Use structured configuration and deterministic generation rather than hundreds of slow, duplicated definitions.
• Limit concurrency: Configure DAG/task concurrency, pools, parallelism, and active-run limits according to Airflow version and executor capacity.
• Use pools for scarce resources: Prevent many tasks from overwhelming a shared database, API, or warehouse.
• Reduce unnecessary task count: Avoid splitting trivial operations into thousands of tiny tasks when one well-sized task or a distributed compute job is more appropriate.
• Use deferrable operators or rescheduling sensors: Where supported, avoid tying up worker slots while waiting for external events.
• Monitor scheduler health: Track DAG parsing time, scheduling delay, queued tasks, executor capacity, metadata database performance, and scheduler logs.
• Separate orchestration from processing: Have Airflow launch Spark, Dataflow, dbt, or warehouse jobs instead of processing large datasets in the scheduler.

Production insight: Hundreds of DAGs are not automatically a problem. Hundreds of DAGs that perform slow top-level parsing, create excessive task instances, or flood a shared warehouse at once can be.

7. Why should large datasets not be passed through XCom?

XCom is intended for small pieces of task metadata or coordination values, such as a file path, partition date, job ID, or status—not for transferring a large dataset between tasks.

• Metadata database overhead: Traditional XCom values are stored in Airflow's metadata database, so large payloads can increase storage and query costs.
• Serialization and deserialization: Large values increase CPU and memory use and can create compatibility issues.
• Performance and reliability: Passing big objects between tasks can slow scheduling and task execution and create oversized metadata records.
• Wrong abstraction: Airflow coordinates work; object storage, data warehouses, and distributed processing systems should hold the data itself.

Better pattern: Write data to durable storage and pass a small reference through XCom.

def extract_to_storage():
    output_path = "gs://my-bucket/staging/run_date=2026-10-13/"
    write_source_data(output_path)
    return output_path  # Small string reference

def transform_from_storage(**context):
    output_path = context["ti"].xcom_pull(
        task_ids="extract"
    )
    transform_data_at_path(output_path)

Airflow can support alternative XCom backends, including object-storage-oriented approaches in some deployments, but XCom should still represent coordination rather than become a general-purpose data transport layer.

8. How would you troubleshoot a DAG that is queued but not running?

A queued task has been scheduled for execution but has not started. The root cause can be in Airflow's scheduling constraints, executor, worker infrastructure, or the runtime environment.

1. Check task state and timestamps: Confirm whether the task is queued, scheduled, deferred, or running and how long it has waited.
2. Inspect executor capacity: Check whether Celery workers, Kubernetes pods, LocalExecutor processes, or the configured execution backend are healthy.
3. Review concurrency limits: Inspect global parallelism, DAG/task limits, active DAG runs, pools, and pool slot availability.
4. Check scheduler logs and health: Look for scheduler restarts, parsing errors, metadata database latency, or scheduling delays.
5. Check infrastructure: For Kubernetes, inspect pod events, resource requests, quotas, image pulls, and service account permissions. For Celery, inspect worker health, broker connectivity, and queue routing.
6. Review task prerequisites: Verify dependencies, trigger rules, and any deferrable or sensor behavior.
7. Inspect logs and alerts: Determine whether the task ever started and whether logs point to an environment or dependency issue.

Do not immediately clear and rerun the task. First identify why it is queued. Repeatedly clearing tasks can hide the actual capacity or configuration issue and may repeat side effects if a prior attempt partially ran.

Useful troubleshooting order

Task state and dependencies → pool/concurrency limits → executor and worker health → scheduler and metadata database → runtime-specific events and logs.

9. How would you test and deploy DAG changes safely?

Treat DAGs as production code. Use automated checks, controlled promotion, and a rollback strategy so a syntax error or logic change does not disrupt scheduled pipelines.

Testing strategy:

• Static checks: Run formatting, linting, import checks, and DAG parsing tests in CI.
• Unit tests: Test Python callables, transformations, branching logic, and data validation independently of the scheduler where possible.
• DAG structure tests: Load the DAG in a test environment and verify task IDs, dependencies, schedules, pools, and important default arguments.
• Integration tests: Test connections and external operators in a controlled environment with representative inputs.
• Idempotency and retry tests: Simulate reruns and failures to verify that tasks do not create duplicate or partial outputs.
• Backfill/reprocessing checks: Validate data intervals and expected output before running historical intervals at scale.

Deployment strategy:

1. Version DAG code and dependencies in source control.
2. Run automated tests and import checks before merging.
3. Promote through development, staging, and production where practical.
4. Deploy in a way compatible with the environment's DAG distribution and scheduler parsing model.
5. Monitor import errors, scheduling delays, task failures, and data-quality checks after release.
6. Keep a rollback path and avoid changing shared interfaces without a compatibility plan.

# Example CI checks (commands depend on project setup)
python -m compileall dags/
pytest tests/
ruff check dags/

Interview tip: A DAG that imports successfully is not necessarily correct. Validate behavior, dependencies, retries, data intervals, permissions, and output correctness—not just syntax.

Quick Revision Cheat Sheet

Topic	Remember
DAG execution	Scheduler evaluates dependencies; executor launches eligible tasks
Retries	Retries help transient failures; tasks must be safe to repeat
Idempotency	Stable intervals, deterministic outputs, upserts, and atomic publication
Catchup vs backfill	Catchup creates eligible historical scheduled runs; backfill targets an explicit historical range
Pipeline dependencies	Use explicit task/DAG/data-aware dependencies and validate the correct interval
Scheduler scalability	Fast parsing, sensible concurrency, pools, and fewer unnecessary tasks
XCom	Pass small metadata or storage references, not large datasets
Queued tasks	Check limits, executor/worker health, scheduler, pools, and logs
Safe deployment	CI checks, unit/integration tests, staged promotion, monitoring, rollback

Scenario-Based Interview Practice

Try to answer each scenario out loud before expanding the suggested approach.

Scenario 1: A task fails after writing half its output, then retries. How do you prevent corruption?

Make the task idempotent. Write to a staging location or transaction, validate the result, and publish atomically where possible. Use deterministic partitions, overwrite/merge semantics, and a completion marker only after successful publication.

Scenario 2: Hundreds of tasks are queued, and the warehouse is overloaded. What would you do?

Use pools and concurrency limits to cap warehouse workload, review DAG/task parallelism, stagger heavy pipelines, and inspect queue age and warehouse saturation. Increase capacity only after confirming it is the right bottleneck to address.

Scenario 3: A downstream DAG starts before the correct upstream partition is ready. How do you fix it?

Replace timing assumptions with an explicit sensor, ExternalTaskSensor, or dataset/asset-aware dependency supported by your Airflow version. Match the upstream and downstream data intervals and validate a completion marker or data-quality contract.

Scenario 4: A DAG works locally but fails after deployment. What do you check?

Inspect DAG import errors, dependency versions, environment variables, connection IDs, secrets, permissions, worker images, and executor-specific configuration. Compare the deployed environment with CI/staging and inspect task logs before clearing tasks.

Chapter Summary

Reliable Airflow pipelines are built through clear orchestration, repeatable task logic, controlled concurrency, and disciplined deployment.

• Understand how DAG runs, task instances, schedulers, and executors fit together.
• Design tasks to be idempotent because retries and backfills repeat work.
• Use explicit dependencies and align data intervals across pipelines.
• Keep XCom small and move datasets through durable storage.
• Diagnose queued tasks by checking constraints and execution infrastructure.
• Test DAG behavior and deploy through a controlled release process.

Practice Challenge

Design an Airflow workflow that ingests daily files, validates their schema, launches a distributed transformation, and publishes results to a warehouse. Explain retries, idempotency, catchup/backfill, cross-pipeline dependencies, pool limits, XCom usage, failure alerts, and how you would test and deploy changes safely.