The catchup Parameter
When a DAG is first unpaused, Airflow's default behavior, catchup=True, is to create a DagRun for every data interval between the DAG's start_date and now, running them all as fast as the scheduler and executor can process them; setting catchup=False on the DAG (or the CATCHUP_BY_DEFAULT config globally) makes the scheduler only create a DagRun for the most recent completed interval going forward, which is almost always what you want for dashboards or alerting DAGs where historical backfill has no value.
Cricket analogy: Like a broadcaster who, upon regaining a lost satellite feed, chooses to replay every missed over from the start of the innings rather than just resuming live, catchup=True makes Airflow create a DagRun for every missed interval since start_date.
Manual Backfills with the CLI
The airflow dags backfill command lets you explicitly (re)run a DAG over a specified date range regardless of the catchup setting, which is the standard tool for reprocessing historical data after fixing a bug in a transformation, backfilling a brand-new DAG against months of historical source data, or regenerating a report after an upstream schema change; you can pass --reset-dagruns to clear and rerun existing DagRuns in that range, and -t to limit the backfill to specific tasks by regex rather than the whole DAG.
Cricket analogy: Like a broadcaster deliberately re-airing an entire historical Ashes series with corrected graphics after finding a scorecard error in the original footage, airflow dags backfill lets you deliberately rerun a DAG over a chosen historical date range to fix past output.
execution_date, Data Intervals, and Idempotency
Backfilling only works correctly if your tasks are idempotent and parameterized on the run's data interval, in Airflow 2.2+ accessed via {{ data_interval_start }} and {{ data_interval_end }} in Jinja templates, rather than on wall-clock time; a task that queries 'yesterday' using datetime.now() instead of the templated interval will produce identical, wrong output no matter which historical interval a backfill run is actually processing, since it ignores the execution context entirely.
Cricket analogy: Like a stats system that must calculate a player's figures for a specific historical match using that match's actual date rather than today's date, a backfilled task must use the templated data_interval_start rather than datetime.now(), or every replayed match returns identical, wrong figures.
# Backfill a DAG for a specific historical window, resetting any existing DagRuns
airflow dags backfill \
--start-date 2026-01-01 \
--end-date 2026-01-31 \
--reset-dagruns \
-t "transform_.*" \
orders_hourly_rollupIn Airflow 2.2+, prefer {{ data_interval_start }} and {{ data_interval_end }} over the deprecated {{ execution_date }} macro; for most schedules they represent the same window, but data_interval_* is explicit about representing the start and end of the processed interval rather than a single ambiguous timestamp.
Running airflow dags backfill against a DAG with max_active_runs set low can silently serialize what you expected to be a fast parallel replay; check max_active_runs and executor capacity before backfilling a large historical range, or the job can take far longer than anticipated.
- catchup=True (the default) creates a DagRun for every missed interval since start_date when a DAG is unpaused.
- catchup=False only schedules the most recent completed interval going forward, ideal for dashboards and alerting DAGs.
- airflow dags backfill explicitly (re)runs a DAG over a chosen historical date range regardless of catchup.
- --reset-dagruns clears and reruns existing DagRuns in the target range; -t limits the backfill to specific tasks by regex.
- Backfilling requires idempotent tasks parameterized on data_interval_start/data_interval_end, not wall-clock time.
- Tasks that hardcode datetime.now() instead of using the templated interval produce identical, wrong output for every backfilled run.
- max_active_runs and executor capacity determine how much a backfill can actually parallelize.
Practice what you learned
1. What does catchup=True cause Airflow to do when a DAG is first unpaused?
2. Which setting is typically recommended for a dashboard-refresh DAG where only the latest data matters?
3. What does the airflow dags backfill command do?
4. Why is using datetime.now() inside a task instead of {{ data_interval_start }} problematic for backfills?
5. What does the --reset-dagruns flag do when passed to airflow dags backfill?
Was this page helpful?
You May Also Like
The Scheduler
How Airflow's scheduler parses DAGs, evaluates dependencies, and triggers task instances to keep pipelines running on time.
Retries and Error Handling
Configuring per-task retries, retry delays, callbacks, and trigger rules so Airflow DAGs recover gracefully from transient failures.
SLAs and Alerting
Using Airflow's sla parameter, sla_miss_callback, and failure callbacks to detect and alert on late or broken pipelines.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics