Airflow Quick Reference

Analogy categoryAnalogies

CLI Commands Cheat Sheet

The airflow CLI is the fastest way to inspect and control a deployment without opening the UI: airflow dags list shows all registered DAGs, airflow dags trigger <dag_id> manually kicks off a run outside the schedule, airflow tasks test <dag_id> <task_id> <date> runs a single task locally with full logs, airflow dags backfill -s <start> -e <end> <dag_id> replays historical runs, and airflow dags list-import-errors surfaces any DAG files that failed to parse. Knowing these commands cold means you can debug a production issue from a terminal in seconds instead of clicking through multiple UI screens under pressure during an incident.

🏏

Cricket analogy: Knowing the Airflow CLI commands cold is like a captain knowing DRS review signals instantly without hesitation, split-second recall under match pressure beats fumbling for the right gesture.

bash

# List all DAGs
airflow dags list

# Trigger a DAG run manually right now
airflow dags trigger orders_etl

# Test a single task for a specific logical date (prints logs, doesn't persist state)
airflow tasks test orders_etl extract 2026-07-10

# Run the full DAG end-to-end for a date, locally
airflow dags test orders_etl 2026-07-10

# Backfill a range of missed daily runs
airflow dags backfill -s 2026-07-07 -e 2026-07-10 orders_etl

# Check for DAG files that failed to import (syntax errors, bad imports)
airflow dags list-import-errors

# Pause / unpause a DAG from the CLI
airflow dags pause orders_etl
airflow dags unpause orders_etl

Common Operators and Sensors

The most frequently used operators are PythonOperator (or the @task decorator equivalent) for arbitrary Python logic, BashOperator for shell commands, and provider-specific operators like PostgresOperator, S3ToRedshiftOperator, or KubernetesPodOperator for talking to specific systems. Common sensors include FileSensor (waits for a file to appear on disk), ExternalTaskSensor (waits for a task in a different DAG to reach a given state), and HttpSensor (polls an HTTP endpoint until a condition is met); nearly all built-in sensors accept mode='reschedule' to avoid holding a worker slot for the whole wait, plus a timeout parameter that fails the sensor cleanly instead of waiting forever.

🏏

Cricket analogy: PythonOperator is like a versatile all-rounder who can bowl, bat, or field depending on what's needed, while a specific tool like S3ToRedshiftOperator is like a specialist wicketkeeper built for exactly one role.

Scheduling Syntax Reference

The schedule parameter accepts standard cron expressions ("0 6 * * *" for 6 AM daily), cron presets (@daily, @hourly, @weekly, @monthly, @once, @yearly), a datetime.timedelta for simple fixed intervals (timedelta(hours=4)), or None for a DAG that only runs when triggered manually or by an external event; newer Airflow versions also support custom Timetable classes for schedules that don't fit cron, such as 'the last business day of each month.' Remember that a run only triggers once its full interval has elapsed and start_date has passed, so a DAG with start_date=datetime(2026,7,10) and @daily schedule won't produce its first run until July 11th, covering the July 10th interval.

🏏

Cricket analogy: A cron preset like @daily is like a fixed county championship round-robin schedule, predictable and standard, while a custom Timetable for 'last business day of month' is like a bespoke tournament format built for one specific league's needs.

Configuration Quick Reference

Key airflow.cfg (or environment variable) settings worth knowing: parallelism caps the total number of task instances that can run concurrently across the entire Airflow instance, dag_concurrency/max_active_tasks_per_dag caps concurrent tasks within one DAG, and max_active_runs_per_dag caps how many DagRuns of the same DAG can be active simultaneously (important for backfills). Connections (stored via airflow connections add or the UI) and Variables (airflow variables set) are the standard way to externalize credentials and config from DAG code, and should be backed by a secrets backend (like AWS Secrets Manager or HashiCorp Vault) in production rather than the default metadata database storage for anything sensitive.

🏏

Cricket analogy: parallelism capping total concurrent tasks is like a cricket board capping how many matches can be played across all grounds nationwide simultaneously, a hard ceiling on total concurrent activity.

For anything sensitive (API keys, database passwords), configure a secrets backend such as AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault rather than storing raw values in Airflow's default metadata-database-backed Connections and Variables.

Remember that execution_date/logical_date always represents the start of the scheduled interval, not the time the DAG actually ran. A @daily DAG's run that triggers on July 11th at 00:05 will have its logical_date stamped July 10th, which surprises many newcomers writing date-dependent queries.

Master core CLI commands: dags list, dags trigger, tasks test, dags test, dags backfill, dags list-import-errors.
PythonOperator/@task and BashOperator cover most general logic; provider-specific operators handle system integrations.
Sensors (FileSensor, ExternalTaskSensor, HttpSensor) poll for conditions; prefer mode='reschedule' with a timeout set.
schedule accepts cron strings, cron presets (@daily, @hourly), timedelta objects, None, or custom Timetable classes.
A scheduled run only triggers after its full interval elapses; logical_date marks the interval start, not the trigger time.
Use parallelism, max_active_tasks_per_dag, and max_active_runs_per_dag to control concurrency at different scopes.
Externalize credentials via Connections/Variables backed by a real secrets manager in production, never hardcode them in DAG code.