The Airflow Architecture
A running Airflow deployment is made up of four core components: the Scheduler, which parses DAG files and decides which task instances are ready to run; the Webserver, which serves the UI you interact with; the Executor, which is the mechanism responsible for actually running tasks (in-process, on local worker processes, or on remote workers); and the Metadata Database, which stores the state of everything — DAG runs, task instances, connections, and variables. In distributed setups, a Triggerer process also runs for deferrable operators, and a message broker like Redis or RabbitMQ sits between the scheduler and workers when using CeleryExecutor.
Cricket analogy: The four components map to a franchise's operations: the Scheduler is like the team manager deciding the day's training slots, the Webserver is the fan-facing app showing live scores, the Executor is the actual players executing drills, and the Metadata DB is the official scorebook recording everything.
The Scheduler and Executor
The Scheduler continuously parses every DAG file in your DAGS_FOLDER, evaluates each DAG's schedule to determine if a new DAG run is due, and for each DAG run, checks which tasks have their upstream dependencies satisfied and queues them. It doesn't run the task code itself — it hands that off to the Executor, which is configured in airflow.cfg (or the AIRFLOW__CORE__EXECUTOR environment variable) and determines the actual concurrency model: whether tasks run sequentially, in local subprocesses, or distributed across a worker fleet.
Cricket analogy: The Scheduler is like a tour selector reviewing fitness reports and picking who's available for the next Test, while the Executor is the physio and support staff actually getting the chosen players match-ready and onto the field.
# airflow.cfg
[core]
executor = CeleryExecutor
dags_folder = /opt/airflow/dags
parallelism = 32
max_active_tasks_per_dag = 16
[celery]
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:airflow@postgres/airflow
[database]
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres/airflowExecutors: Sequential, Local, Celery, Kubernetes
SequentialExecutor runs one task at a time in the scheduler's own process and is only usable with SQLite — it exists purely as a zero-dependency default for a first airflow standalone run. LocalExecutor runs tasks as parallel subprocesses on the same machine as the scheduler, which is fine for a single powerful server but doesn't scale horizontally. CeleryExecutor distributes tasks across a fleet of worker machines via a message broker (Redis or RabbitMQ), giving you horizontal scalability with persistent, always-on workers. KubernetesExecutor instead launches a fresh Kubernetes pod per task, giving perfect per-task resource isolation and autoscaling at the cost of pod startup latency for every task.
Cricket analogy: SequentialExecutor is like one net bowler bowling to one batter with no one else practicing; LocalExecutor is like several nets running at once at the same ground; CeleryExecutor is like a full national academy with permanent coaching staff at multiple centers; KubernetesExecutor is like hiring a fresh specialist coach flown in for each individual session, then released.
For most teams beyond a single-developer prototype, CeleryExecutor or KubernetesExecutor is the right choice. Managed services abstract this decision for you: Google Cloud Composer and Amazon MWAA both run on KubernetesExecutor-style isolation under the hood, so you rarely configure executors directly on those platforms.
The Metadata Database
Every piece of Airflow's runtime state lives in the metadata database: DAG run records, individual task instance states (queued, running, success, failed, up_for_retry), XCom values passed between tasks, Connections and Variables, and user/role data for the UI. Because the Scheduler, Webserver, and every Worker all read and write to this single database, its performance directly bounds Airflow's overall scalability — a common production issue at scale is metadata DB connection exhaustion, which is why connection pooling (e.g., via PgBouncer) is standard practice for large Airflow deployments.
Cricket analogy: The metadata DB is like the official ICC scorecard system that every broadcaster, commentator, and stats analyst reads from simultaneously — if that central feed slows down, everyone downstream is affected.
- Airflow's four core components are the Scheduler, Webserver, Executor, and Metadata Database.
- The Scheduler parses DAGs and decides which tasks are ready to run, but does not execute task code itself.
- The Executor determines the concurrency model: Sequential, Local, Celery, or Kubernetes.
- SequentialExecutor is single-task, SQLite-only, and meant for trivial local trials.
- LocalExecutor parallelizes tasks on one machine; CeleryExecutor and KubernetesExecutor scale horizontally.
- The Triggerer process handles deferrable operators, and a broker like Redis sits between scheduler and Celery workers.
- The metadata database is a shared bottleneck read/written by every component, making connection pooling important at scale.
Practice what you learned
1. Which Airflow component is responsible for parsing DAG files and deciding which tasks are ready to run?
2. Which executor launches a brand-new Kubernetes pod for every single task?
3. Which executor is restricted to using SQLite and runs tasks one at a time?
4. What component sits between the Scheduler and workers when using CeleryExecutor?
5. Why is the metadata database considered a common scaling bottleneck in large Airflow deployments?
Was this page helpful?
You May Also Like
What Is Apache Airflow?
An introduction to Apache Airflow as a platform for programmatically authoring, scheduling, and monitoring workflows defined as code.
Installing Airflow
A practical walkthrough of the main ways to install Apache Airflow, from a local pip install to Docker Compose, and how to verify the install worked.
The Airflow UI
A guided tour of the Airflow web interface, from the DAGs list and Grid view to task logs, Connections, and Variables.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics