Airflow Cheat Sheet
A cheat sheet for Apache Airflow covering DAG authoring with the TaskFlow API, operators, task dependencies, and essential CLI commands.
2 PagesIntermediateMar 12, 2026
TaskFlow API DAG
Define a DAG with Python-native task decorators.
python
from airflow.decorators import dag, taskfrom datetime import datetime@dag(schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False)def etl_pipeline(): @task def extract(): return {'rows': 100} @task def transform(data): data['rows'] *= 2 return data @task def load(data): print(f"Loaded {data['rows']} rows") load(transform(extract()))etl_pipeline()
Classic Operators
Traditional operator-based DAG with explicit dependencies.
python
from airflow import DAGfrom airflow.operators.python import PythonOperatorfrom airflow.operators.bash import BashOperatorfrom datetime import datetimewith DAG('classic_dag', start_date=datetime(2024, 1, 1), schedule='0 6 * * *') as dag: t1 = BashOperator(task_id='print_date', bash_command='date') t2 = PythonOperator(task_id='say_hi', python_callable=lambda: print('hi')) t1 >> t2 # t1 must run before t2
CLI Commands
Manage the webserver, scheduler, and DAG runs.
bash
airflow webserver -p 8080 # Start the UIairflow scheduler # Start the schedulerairflow dags list # List all DAGsairflow dags trigger etl_pipeline # Manually trigger a DAG runairflow tasks test etl_pipeline extract 2024-01-01 # Test a single task
Core Concepts
Key Airflow terminology.
- DAG- Directed Acyclic Graph describing task dependencies and a schedule
- Operator- Template for a single task, e.g. BashOperator, PythonOperator, KubernetesPodOperator
- Task Instance- A specific run of a task for a given execution/logical date
- XCom- Mechanism for passing small pieces of data between tasks
- Sensor- Special operator that waits for a condition, like a file arriving, before continuing
- Executor- Determines how tasks run: LocalExecutor, CeleryExecutor, or KubernetesExecutor
Dynamic Task Mapping
Fan out a task over a runtime-determined list without writing a Python for-loop at parse time.
python
from airflow.decorators import dag, taskfrom datetime import datetime@dag(schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False)def mapped_pipeline(): @task def list_files(): return ['a.csv', 'b.csv', 'c.csv'] @task def process_file(filename: str, chunk_size: int = 1000): print(f'Processing {filename} with chunk size {chunk_size}') # .expand fans out one mapped task instance per list item process_file.partial(chunk_size=500).expand(filename=list_files())mapped_pipeline()
TaskGroups & Branching
Visually cluster related tasks and route execution conditionally based on upstream results.
python
from airflow.decorators import dag, task, task_groupfrom datetime import datetime@dag(schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False)def branching_pipeline(): @task.branch def choose_path(value: int): return 'high_path' if value > 50 else 'low_path' @task_group def high_path(): @task def alert(): print('value exceeded threshold') alert() @task_group def low_path(): @task def log_normal(): print('value within range') log_normal() choose_path(75) >> [high_path(), low_path()]branching_pipeline()
Dataset-Aware Scheduling
Trigger a downstream DAG automatically when an upstream DAG produces (updates) a Dataset, instead of relying on cron.
python
from airflow import Datasetfrom airflow.decorators import dag, taskfrom datetime import datetimeraw_orders = Dataset('s3://bucket/raw/orders')@dag(schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False)def producer(): @task(outlets=[raw_orders]) def extract(): print('wrote new orders file') extract()producer()@dag(schedule=[raw_orders], start_date=datetime(2024, 1, 1), catchup=False)def consumer(): @task def transform(): print('orders dataset updated, running transform') transform()consumer()
Trigger Rules & Failure Callbacks
Control when a task runs relative to upstream outcomes and hook into failures for alerting.
python
from airflow.operators.python import PythonOperatorfrom airflow.utils.trigger_rule import TriggerRuledef notify_slack(context): ti = context['task_instance'] print(f'Task {ti.task_id} failed in dag {ti.dag_id}')cleanup = PythonOperator( task_id='cleanup', python_callable=lambda: print('cleanup'), trigger_rule=TriggerRule.ALL_DONE, # runs even if upstream tasks failed on_failure_callback=notify_slack, retries=2, sla=None,)
Production Operational Concepts
Terms that matter once a DAG moves from a laptop to a shared, multi-tenant scheduler.
- Deferrable operator- Releases its worker slot while waiting (e.g. for a sensor condition) via the triggerer process, avoiding wasted resources on long polls
- Pool- Named resource limiting concurrent task instances (e.g. cap DB-heavy tasks at 5 concurrent slots)
- mode='reschedule'- Sensor option that frees the worker slot between poke intervals instead of blocking it (vs default mode='poke')
- Backfill (airflow dags backfill)- Runs a DAG for a historical date range to reprocess or fill gaps in past logical dates
- Secrets backend- Pluggable store (Vault, AWS Secrets Manager, GCP Secret Manager) for Connections/Variables instead of the metadata DB
- dag.test()- Runs a full DAG synchronously in-process for local debugging without a scheduler or database backend
Pro Tip
Keep DAG files lightweight and free of heavy top-level computation or database calls — the scheduler re-parses every DAG file on a short interval, so slow imports or expensive logic at import time will bottleneck the entire scheduler, not just one DAG.
Was this cheat sheet helpful?
Explore Topics
#Airflow#AirflowCheatSheet#DataScience#Intermediate#TaskFlowAPIDAG#ClassicOperators#CLICommands#CoreConcepts#MachineLearning#APIs#CommandLine#CheatSheet#SkillVeris
