How the Scheduler Works
The Airflow scheduler is the persistent process that continuously scans your DAGs folder, parses each DAG file into a DAG object, and decides when task instances should run. On every loop it evaluates each DAG's schedule (either a legacy schedule_interval cron string or a Timetable object in Airflow 2.4+), checks whether the current data interval has completed, and if so creates a DagRun and queues the first eligible TaskInstances. The scheduler does not execute your tasks itself; it hands them off to an executor, which is why a healthy pipeline needs both a running scheduler and a running executor.
Cricket analogy: Like the third umpire who doesn't bowl or bat but continuously reviews replays to decide exactly when a run-out call should be triggered, the scheduler doesn't execute tasks, it only decides the precise moment each one is due and signals the executor to act.
Scheduler Loop and DAG Parsing
Each scheduler run is split between two responsibilities: the DAG file processor, which parses .py files in the DAGS_FOLDER into serialized DAG representations stored in the metadata database, and the main scheduling loop, which reads those serialized DAGs rather than re-executing your Python files on every tick. This separation, introduced with DAG Serialization in Airflow 2.0, means the scheduler no longer needs direct filesystem access to your DAG code at scheduling time, and the parsing_processes and min_file_process_interval settings control how many parallel processes scan the folder and how often each file is re-parsed.
Cricket analogy: Like a broadcaster's stats team that transcribes every ball into a structured scorecard once, so commentators reference that scorecard instead of rewatching raw footage, Airflow parses DAG files into a serialized form once so the scheduling loop reads structured data instead of re-executing Python repeatedly.
High Availability and Scaling the Scheduler
Since Airflow 2.0, you can run multiple scheduler replicas concurrently for high availability; they coordinate through row-level locking (SELECT ... FOR UPDATE SKIP LOCKED) on the metadata database so two schedulers never queue the same task instance twice. This lets you scale scheduler throughput horizontally by adding replicas rather than vertically scaling a single process, and the scheduler_heartbeat_sec setting controls how often each replica updates its liveness timestamp so a stalled instance can be detected and its DAGs reassigned.
Cricket analogy: Like two third umpires at different grounds simultaneously reviewing DRS calls from a shared central feed, coordinated so no two umpires re-review the same dismissal, multiple Airflow schedulers use row-level locks so no two replicas queue the same task instance twice.
from airflow import DAG
from airflow.timetables.trigger import CronTriggerTimetable
from airflow.operators.python import PythonOperator
import pendulum
with DAG(
dag_id="orders_hourly_rollup",
start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
timetable=CronTriggerTimetable("0 * * * *", timezone="UTC"),
catchup=False,
max_active_runs=1,
) as dag:
def rollup(**context):
interval_start = context["data_interval_start"]
print(f"Rolling up orders for interval starting {interval_start}")
PythonOperator(task_id="rollup_orders", python_callable=rollup)Tune parsing_processes (default 2) in airflow.cfg to add parallel DAG file parsers if your DAGS_FOLDER holds hundreds of files; the scheduler's ability to detect new DagRuns is directly limited by how fast it can re-parse and serialize your DAGs.
A single DAG file that does expensive work at import time, such as querying a database to build tasks dynamically, slows down every parsing cycle for every DAG, not just its own, because the file processor re-imports it on every min_file_process_interval tick. Keep top-level DAG file code cheap.
- The scheduler parses DAG files, evaluates schedules/timetables, and creates DagRuns and TaskInstances but never executes tasks itself.
- DAG Serialization (Airflow 2.0+) stores parsed DAGs in the metadata database so the scheduling loop doesn't re-parse Python files on every tick.
- parsing_processes and min_file_process_interval control how many parallel file processors run and how often each DAG file is re-parsed.
- Multiple scheduler replicas can run concurrently for HA, coordinated via row-level locking (SELECT FOR UPDATE SKIP LOCKED).
- scheduler_heartbeat_sec governs how liveness is tracked so a stalled scheduler replica can be detected.
- Expensive top-level code in a DAG file slows down parsing for that file on every processing cycle, so keep DAG-file-level logic cheap.
Practice what you learned
1. What is the primary responsibility of the Airflow scheduler?
2. What did DAG Serialization (introduced in Airflow 2.0) change about scheduling?
3. How do multiple scheduler replicas avoid queuing the same task instance twice?
4. Which setting controls how often each DAG file is re-parsed by the file processor?
5. Why is expensive top-level code in a DAG file discouraged?
Was this page helpful?
You May Also Like
Executors: Local, Celery, Kubernetes
Comparing Airflow's LocalExecutor, CeleryExecutor, and KubernetesExecutor for running task instances at different scales.
Backfilling and Catchup
Understanding Airflow's catchup behavior and the airflow dags backfill command for (re)processing historical data intervals.
Retries and Error Handling
Configuring per-task retries, retry delays, callbacks, and trigger rules so Airflow DAGs recover gracefully from transient failures.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics