Your First DAG

Analogy categoryAnalogies

A DAG file is just a Python module that Airflow's scheduler periodically scans; as long as it contains a top-level DAG object (or uses the @dag decorator), Airflow will pick it up automatically from the DAGS_FOLDER. Writing your first real DAG means deciding three things up front: what tasks it needs, what order those tasks run in, and on what schedule (or trigger) the whole thing should fire — everything else in the file is really just configuring those three decisions in code.

🏏

Cricket analogy: Writing your first DAG is like planning your first net session as a young player: decide which drills (tasks) to do, in what order (batting before fielding warm-ups), and what time you show up (the schedule).

Anatomy of a DAG File

Every DAG needs a unique dag_id, a start_date marking the earliest logical date it's eligible to run, and a schedule (a cron expression, a preset like @daily, or a Dataset/None for event-driven or manually-triggered DAGs). catchup=False is worth setting explicitly on almost every DAG you write while learning, since Airflow's default behavior is to schedule a run for every interval between start_date and now — without it, a DAG with a start_date six months ago will immediately try to run hundreds of backfill runs the moment it's unpaused.

🏏

Cricket analogy: The start_date is like a player's official international debut date recorded by the ICC — everything before that date simply isn't part of their career stats.

python

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "data-team",
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
}

def fetch_weather(**context):
    city = context["dag_run"].conf.get("city", "London")
    print(f"Fetching weather for {city}")
    return {"city": city, "temp_c": 18}

def report(**context):
    ti = context["ti"]
    data = ti.xcom_pull(task_ids="fetch_weather")
    print(f"Weather report: {data['city']} is {data['temp_c']}C")

with DAG(
    dag_id="daily_weather_report",
    default_args=default_args,
    description="Fetches weather and emails a daily report",
    schedule="0 7 * * *",
    start_date=datetime(2026, 6, 1),
    catchup=False,
    tags=["weather", "tutorial"],
) as dag:

    check_api = BashOperator(
        task_id="check_api",
        bash_command="curl -sf https://api.example.com/health",
    )

    fetch = PythonOperator(
        task_id="fetch_weather",
        python_callable=fetch_weather,
    )

    send_report = PythonOperator(
        task_id="report",
        python_callable=report,
    )

    check_api >> fetch >> send_report

Task Dependencies

Beyond simple linear chains, Airflow supports fan-out (task_a >> [task_b, task_c]) and fan-in ([task_b, task_c] >> task_d) patterns to express parallel work that later converges, and BranchPythonOperator lets a task return the task_id of whichever downstream branch should actually run, skipping the rest. Data passed between tasks — like the small dictionary returned by fetch_weather in the example above — travels through XCom (cross-communication), Airflow's mechanism for sharing small values between tasks in the same DAG run; XCom is stored in the metadata database, so it is explicitly not meant for large payloads like entire dataframes.

🏏

Cricket analogy: Fan-out is like a coach sending three players to three different fitness stations simultaneously, and fan-in is like all three regrouping for a single team debrief afterward — XCom is like each player reporting their individual test results back to the coach.

XCom is convenient but easy to misuse. Passing a 2GB pandas DataFrame through XCom will either fail outright or silently degrade performance, since it round-trips through the metadata database. For large data, pass a reference instead — an S3 URI or a table name — and let the downstream task read the actual data from storage.

Testing and Triggering Your DAG

Before relying on the scheduler, you can validate a DAG file simply by running it with Python (python dags/daily_weather_report.py) to catch import errors, and use airflow dags list-import-errors to catch parsing failures across the whole DAGS_FOLDER. airflow tasks test <dag_id> <task_id> <date> runs a single task in isolation without touching the metadata database or scheduler — ideal for iterating on task logic — while airflow dags test <dag_id> <date> runs the full DAG end-to-end locally. Once you're satisfied, unpausing the DAG in the UI (or via airflow dags unpause) lets the scheduler pick it up for real, or you can trigger an immediate run with airflow dags trigger <dag_id>.

🏏

Cricket analogy: Running airflow tasks test on one task is like a bowler doing a solo run-up drill in the nets before the actual match, while airflow dags test is like a full intra-squad practice match simulating the whole game.

bash

# Validate the file parses without errors
python dags/daily_weather_report.py

# List any DAG parsing errors across the whole DAGS_FOLDER
airflow dags list-import-errors

# Test a single task in isolation (no DB writes, no scheduler involved)
airflow tasks test daily_weather_report fetch_weather 2026-07-10

# Run the whole DAG locally end-to-end for a given logical date
airflow dags test daily_weather_report 2026-07-10

# Unpause and trigger a real run once you're confident
airflow dags unpause daily_weather_report
airflow dags trigger daily_weather_report

A DAG file is a Python module containing a DAG object (or @dag-decorated function) that Airflow auto-discovers from DAGS_FOLDER.
Every DAG needs a unique dag_id, a start_date, and a schedule; set catchup=False to avoid an unwanted backfill storm.
Tasks are chained with >>, supporting linear chains, fan-out to parallel tasks, and fan-in back to a single task.
XCom passes small values between tasks via the metadata database; large data should be passed by reference instead.
BranchPythonOperator lets a DAG dynamically choose which downstream path to execute.
airflow tasks test runs a single task in isolation; airflow dags test runs the whole DAG locally end-to-end.
A DAG must be unpaused before the scheduler will create runs for it on its normal schedule.

Practice what you learned

1. What three things must every DAG definition specify at minimum?

2. What happens if you unpause a DAG with start_date six months ago and catchup left at its default value?

3. What is XCom used for in Airflow?

4. Which command runs a single task in isolation without writing to the metadata database?

5. What Airflow operator lets a DAG dynamically choose which downstream task path to execute?

Was this page helpful?

The Airflow UI

Related Study Notes in Programming

Browse all study notes

Programming

Apache Spark Study Notes

Programming · 30 topics

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Hadoop Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#ApacheAirflowStudyNotes#YourFirstDAG#DAG#Anatomy#File#Task#StudyNotes#SkillVeris#ExamPrep

Your First DAG

Your First DAG

Anatomy of a DAG File

Task Dependencies

Testing and Triggering Your DAG

Practice what you learned

You May Also Like

Related Reading

Related Study Notes in Programming

Topics covered