What Is a DAG?
A DAG (Directed Acyclic Graph) in Apache Airflow is a plain Python file that describes a workflow as a collection of tasks and the dependencies between them. You instantiate a DAG object, usually inside a with DAG(...) as dag: context manager, give it a unique dag_id, and then define tasks that get automatically attached to that DAG instance. The word 'acyclic' matters: Airflow will refuse to run a graph that loops back on itself, because a workflow has to have a clear start and finish.
Cricket analogy: A DAG is like a full day-night ODI schedule card: openers bat, then middle order, then death overs bowlers, each slot fixed relative to the others, and the match can never loop back to over one once it has moved on.
Anatomy of a DAG File
A minimal DAG definition needs a dag_id (unique across the whole Airflow instance), a schedule (cron string, timedelta, preset like @daily, or None for manual triggering only), a start_date, and typically catchup=False unless you intentionally want historical backfills. default_args is a dictionary applied to every task in the DAG, commonly holding things like retries, retry_delay, and owner, so you don't repeat them on each task. Tags are optional but useful for filtering DAGs in the Airflow UI when you have dozens of pipelines.
Cricket analogy: dag_id is like a team's unique squad number list for a tournament, no two players share a jersey number, while default_args is like the standard fielding restrictions applied to every over of the innings.
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.operators.python import PythonOperator
import pendulum
def extract():
print("Extracting data from source system...")
with DAG(
dag_id="daily_sales_ingestion",
schedule="0 6 * * *",
start_date=pendulum.datetime(2026, 1, 1, tz="Asia/Kolkata"),
catchup=False,
default_args={
"owner": "data-eng",
"retries": 2,
"retry_delay": pendulum.duration(minutes=5),
},
tags=["sales", "daily"],
) as dag:
start = EmptyOperator(task_id="start")
extract_task = PythonOperator(task_id="extract", python_callable=extract)
start >> extract_taskHow Airflow Parses DAG Files
The Airflow scheduler continuously re-scans every .py file inside the dags_folder (by default every dag_dir_list_interval seconds, commonly 300, plus a min_file_process_interval for how often each file is re-parsed) and executes the top-level code in each file to rebuild the DagBag. This means any code sitting outside of a task's callable, such as a database query used to generate task lists dynamically, runs on every parse cycle, not just once. Keeping top-level code fast and side-effect-free is essential, because a slow-parsing DAG file directly slows down scheduler performance for the entire Airflow instance.
Cricket analogy: It's like the third umpire re-checking every over's footage on a fixed review cycle throughout the day rather than once at the end, so any slow replay process bottlenecks the whole match schedule.
Idempotency and Design Principles
Well-designed DAGs are idempotent: running the same DAG run twice for the same logical date should produce the same result and not create duplicate rows or double-send the same email. Airflow encourages this by giving each task run access to templated context variables like {{ ds }} or {{ data_interval_start }}, so a task can, for example, delete-then-insert partitioned data for that specific interval rather than blindly appending. Avoid dynamic side effects at DAG-parse time (like writing files or making network calls outside a task), since those run repeatedly on every scheduler parse cycle rather than once per actual execution.
Cricket analogy: It's like DRS reviews being re-checkable and consistent: replaying the same ball's footage twice must always give the same out or not-out verdict, never a different result on a second look.
Airflow uses DAG serialization: when core.dag_serialization is enabled (the default in modern Airflow), the scheduler parses DAG files and stores a serialized JSON representation in the metadata database. The webserver then reads from that serialized store instead of re-executing your DAG files, which is why the UI stays responsive even for DAG folders with slow-importing dependencies.
Never put expensive operations, such as a live database query to decide how many tasks to create, directly at the top level of a DAG file. That code runs on every single scheduler parse cycle, not once per DAG run, and a single slow DAG file can degrade scheduling latency for every other DAG in the instance.
- A DAG is a Python file that defines a workflow as tasks plus their dependencies, with no cycles allowed.
- Every DAG needs a unique dag_id, a schedule, a start_date, and typically catchup=False.
- default_args applies shared settings like retries and owner to every task in the DAG without repetition.
- The scheduler repeatedly re-parses DAG files, so top-level code outside task callables runs on every parse cycle.
- DAG serialization stores a JSON snapshot in the metadata database so the webserver doesn't need to re-import DAG files.
- Well-designed DAGs are idempotent: re-running the same logical date should not produce duplicate side effects.
- Templated context variables like {{ ds }} let tasks behave correctly for the specific interval they represent.
Practice what you learned
1. What does the 'acyclic' part of Directed Acyclic Graph guarantee in an Airflow DAG?
2. Why is it risky to put a slow database query directly at the top level of a DAG file (outside any task callable)?
3. What is the purpose of default_args in a DAG definition?
4. Why does Airflow use DAG serialization?
Was this page helpful?
You May Also Like
Operators
Understand what Airflow operators are, the major built-in categories, and how to configure or subclass them to build reliable tasks.
Task Dependencies
Learn how to wire tasks together with bitshift operators and the chain() helper, and how trigger rules control execution when upstream tasks fail or skip.
Scheduling and Cron Expressions
Learn how Airflow's schedule parameter works with cron expressions, presets, and timedeltas, and how data intervals, catchup, and backfill behave.
The TaskFlow API
Learn how Airflow's @dag and @task decorators let you write DAGs as plain Python functions, with automatic XCom passing and dynamic task mapping.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics