Airflow Architecture — Scheduler, Executor, Metadata DB

Apache Airflow is a platform for programmatically authoring, scheduling, and monitoring data workflows. Understanding its internal architecture is essential before writing production DAGs: knowing what the scheduler does, how executors run tasks, and what the metadata database stores explains why certain Airflow anti-patterns cause mysterious failures and why certain configuration choices have outsized performance impact. Airflow's architecture is more complex than it first appears — four separate components must work together correctly for a DAG to execute reliably at scale.

The four core Airflow components are: the Scheduler (parses DAG files and schedules task instances), the Executor (runs task instances — locally or on a distributed worker pool), the Metadata Database (stores all DAG definitions, task states, connection configs, and XCom values), and the Webserver (serves the UI and REST API). In a production deployment these run as separate processes. The Scheduler is the most critical single point of failure — if it stops, no new task instances are scheduled, but running tasks continue to completion.

Analogy🏏Cricket

🏏 Think of it like cricket: Migrating from Airflow to Prefect is like the same bowling coach shifting from traditional Test cricket notation to a modern T20 analytics dashboard — the underlying ball-by-ball data (the business logic) is exactly the same. What changes is how the data is recorded, displayed, and acted upon. The yorker that Bumrah bowls in over 20 is identical whether it is recorded in the old scorebook (Airflow DAG file) or the new analytics platform (Prefect flow). The migration is a transcription exercise, not a strategy change — and a wise coach verifies that the runs, wickets, and economies match exactly between the old and new system before decommissioning the scorebook. That verification step is the whole heart of the migration: because the yorker is unchanged, the only honest test is to run the same over through both systems and confirm the recorded runs, wickets and economies match to the last digit before the old scorebook is thrown away. Rushing to burn the scorebook the moment the shiny dashboard lights up is how teams lose a season of records to a silent transcription slip. The coach keeps both systems running in parallel for a while, reconciles their outputs ball by ball, and only when every figure agrees does he trust the new dashboard alone — a transcription is only complete when you have proven nothing was lost in the copying.