Dynamic DAG Generation

Analogy categoryAnalogies

What Is Dynamic DAG Generation?

Because an Airflow DAG file is just Python that runs on every scheduler parse cycle, you are not limited to hand-writing one DAG per file. You can loop over a list of configurations and instantiate a DAG object for each one, registering it in the module's global namespace with globals()[dag_id] = dag so the DagBag picker discovers every generated DAG.

🏏

Cricket analogy: It's like a franchise drafting a squad list once and generating a fixture card for every team from that single roster file, instead of typing out each match schedule by hand.

Generating DAGs from a Config File

A common pattern reads a YAML or JSON file listing DAG definitions — schedule interval, source table, destination, owner — then calls a factory function once per entry. The factory builds the DAG and its tasks with the with DAG(...) as dag: context manager and returns it, and the loop assigns each result to globals() under a unique dag_id so dozens of near-identical ingestion pipelines can be added by editing config, not code.

🏏

Cricket analogy: It's like the IPL auction house entering each player's stats into one spreadsheet, and a template auto-builds every franchise's squad page from that shared data.

python

import yaml
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def build_dag(dag_id: str, schedule: str, table: str) -> DAG:
    with DAG(
        dag_id=dag_id,
        schedule=schedule,
        start_date=datetime(2026, 1, 1),
        catchup=False,
        tags=["ingestion", "generated"],
    ) as dag:
        PythonOperator(
            task_id="extract_and_load",
            python_callable=extract_and_load,
            op_kwargs={"table": table},
        )
    return dag

with open("/opt/airflow/dags/config/ingestion.yaml") as f:
    configs = yaml.safe_load(f)["pipelines"]

for cfg in configs:
    dag_id = f"ingest_{cfg['table']}"
    globals()[dag_id] = build_dag(dag_id, cfg["schedule"], cfg["table"])

Single-File Generation vs. Dynamic Task Mapping

Dynamic DAG generation (many DAG objects from one file, decided at parse time) is a different mechanism from dynamic task mapping (.expand(), introduced in Airflow 2.3), which creates a variable number of task instances of the same DAG at run time based on upstream data. Generation changes what DAGs exist in the DagBag; mapping changes how many task instances one DAG run has — they solve different scaling problems and are often combined.

🏏

Cricket analogy: It's the difference between a board deciding how many franchises exist before the season (generation) versus a captain deciding how many bowlers to rotate mid-innings based on the pitch (mapping).

Performance Pitfall: DAG File Parsing Cost

Every generated DAG still lives in a single .py file that the DAG File Processor re-parses on a fixed interval (min_file_process_interval, default 30 seconds). If the factory loop performs network calls, database queries, or reads a huge config on every parse — not just once at deploy time — parsing that file becomes slow and can delay the scheduler from picking up new task instances across the entire environment, since file processing is a shared, throttled resource.

🏏

Cricket analogy: It's like a scorer re-fetching every player's entire career stats from a slow archive every 30 seconds during a live match instead of caching the day's team sheet once, delaying score updates for every match on the ground.

Avoid API calls, database queries, or filesystem scans at the top level of a DAG file (outside of task callables). These run on every DAG File Processor parse cycle for every scheduler, not just once — a slow external call here can degrade scheduling latency across the whole Airflow instance. Cache configuration locally (e.g., a checked-in YAML/JSON file) or use Airflow Variables with caching instead.

Patterns for Scalable, Metadata-Driven Pipelines

In production, dynamic generation is most valuable for metadata-driven pipelines: a central registry (a database table, a YAML file in version control, or an Airflow Variable) drives dozens or hundreds of near-identical ETL DAGs, such as one-DAG-per-source-table ingestion jobs. Wrapping each generated DAG's tasks in a TaskGroup keeps the Grid view readable even as the number of tasks per DAG grows, and consistent dag_id naming makes the pattern auditable.

🏏

Cricket analogy: It's like a cricket board keeping one master player-registry database that auto-generates every domestic team's roster page, grouping each team's batting and bowling lineups into clearly labeled sections for easy scouting.

Wrap each generated DAG's tasks in a TaskGroup (with TaskGroup('extract_load') as tg:) so the Grid and Graph views stay collapsible and readable even when the factory scales to hundreds of DAGs.

DAG files are plain Python, so loops can create multiple DAG objects and register them via globals()[dag_id] = dag.
Config-driven factories (YAML/JSON → factory function → DAG) let you add pipelines by editing config instead of writing new code.
Dynamic DAG generation (many DAGs from one file) is distinct from dynamic task mapping (.expand()), which varies task instance count within one DAG run.
The DAG File Processor re-parses every file on min_file_process_interval; expensive calls at file top level slow scheduling for the whole environment.
Never make network or database calls outside task callables in a DAG file — cache config locally instead.
Metadata-driven pipelines (central registry driving many generated DAGs) are the primary real-world use case for this pattern.
Use TaskGroup inside generated DAGs to keep the UI readable as the factory scales to many tasks or DAGs.