Airflow Best Practices

Analogy categoryAnalogies

Writing Idempotent and Atomic Tasks

An idempotent task produces the same end state no matter how many times it runs with the same inputs, which matters because Airflow retries failed tasks and backfills historical runs. Instead of appending rows with INSERT on every run, use a DELETE-then-INSERT or MERGE/UPSERT pattern keyed on the execution date so reruns don't duplicate data. Atomicity means a task either fully succeeds or leaves no partial trace, typically by wrapping the load in a database transaction or writing to a staging table before an atomic swap.

🏏

Cricket analogy: It's like DRS review protocol: whichever umpire reviews the delivery, the same replay footage and the same third-umpire decision must be reached every time, not a different verdict depending on who's reviewing.

DAG Design Principles

The Airflow scheduler parses every DAG file repeatedly (by default every 30 seconds) to detect changes, so any expensive top-level code, such as database queries, API calls, or large file reads placed outside of task callables, runs on every parse cycle and can severely slow the scheduler. Keep DAG files declarative: define operators and dependencies, but push actual data work inside task functions using the TaskFlow API's @dag and @task decorators, which also makes DAGs easier to read and unit test. Dynamic DAG generation from external sources (like a database of table names) should read from a cached local file rather than hitting a live API at parse time.

🏏

Cricket analogy: A ground curator inspecting the pitch every 30 seconds before a Test match would slow the entire ground down if the inspection itself involved re-digging the soil each time, instead the check should be a quick visual glance.

Managing Dependencies and XComs

XCom (cross-communication) is Airflow's built-in mechanism for passing small pieces of data between tasks, such as a file path, a row count, or a generated ID, but it is backed by the metadata database and is not meant for large payloads like full DataFrames or file contents; pushing megabytes through XCom bloats the metadata DB and slows the UI. Use task groups (TaskGroup or the @task_group decorator) to visually and logically cluster related tasks like extract_customers, extract_orders under an extract group, and prefer explicit bitshift dependencies (task_a >> task_b) over relying on implicit ordering from XCom references alone, since explicit dependencies are what the scheduler actually uses to build the execution graph.

🏏

Cricket analogy: A fielding captain relays quick hand signals to adjust the field, not the entire match strategy document, similarly XCom should carry small signals like a boundary count, not the full scorecard history.

Testing and CI/CD for DAGs

Before a DAG reaches production, it should pass automated checks: a basic DagBag import test (DagBag(dag_folder=...).import_errors should be empty) catches syntax errors and broken imports, while custom pytest assertions can verify things like every DAG has a retries default, no cycles exist, and tags follow naming conventions. For local iteration, airflow dags test <dag_id> <execution_date> runs a full DAG end-to-end without touching the scheduler or creating DagRun history clutter, which is much faster than deploying to a live environment to see if a change works. Set catchup=False on new DAGs unless historical backfilling is explicitly needed, since a DAG with a start_date months in the past and catchup=True will silently schedule a backlog of runs.

🏏

Cricket analogy: Before a bowler is allowed to bowl in a match, umpires check their action isn't illegal in the nets first, a cheap pre-check that catches problems before they cost the team in a real game.

python

from airflow.decorators import dag, task
from pendulum import datetime

@dag(
    dag_id="sales_summary_etl",
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args={"retries": 2, "retry_delay": 300},
    tags=["sales", "etl"],
)
def sales_summary_etl():

    @task
    def extract(ds=None) -> str:
        # ds is the logical date string; use it to make the read deterministic
        return f"/data/staging/sales_{ds}.parquet"

    @task
    def load(staging_path: str, ds=None) -> None:
        # Idempotent: delete existing partition for ds, then insert fresh
        # DELETE FROM sales_summary WHERE partition_date = '{ds}';
        # INSERT INTO sales_summary SELECT ... FROM read_parquet('{staging_path}');
        pass

    load(extract())

sales_summary_etl()

Use catchup=False for most operational DAGs. Reserve catchup=True (the default) specifically for pipelines where historical backfills matter, and always pair it with a bounded start_date to avoid an unexpected flood of backlog DagRuns.

Never put database connections, API calls, or heavy computation directly at the top level of a DAG file. The scheduler re-parses every .py file in the DAGs folder on every parsing loop, so top-level side effects run constantly and can bring the scheduler to a crawl across your entire Airflow deployment.

Design tasks to be idempotent (same result on rerun) and atomic (fully succeed or leave no partial state).
Keep DAG files declarative; move expensive logic inside task callables, not top-level code, to avoid slowing the scheduler's repeated file parsing.
Use the TaskFlow API (@dag, @task) for cleaner, more testable DAG definitions.
Pass only small metadata through XCom; never push large datasets or files through it.
Use TaskGroups to organize related tasks and explicit >> dependencies to define execution order.
Validate DAGs locally with DagBag import checks and airflow dags test before deploying.
Default new DAGs to catchup=False unless a deliberate historical backfill is required.