Airflow vs Prefect vs Dagster

Airflow is fundamentally DAG-and-task-centric: you define a static graph of tasks and their dependencies, and Airflow's scheduler executes them according to a schedule, with the DAG structure treated as largely fixed at parse time. Prefect takes a flow-based approach where Python functions decorated with @flow and @task can branch, loop, and make dynamic runtime decisions much like ordinary code, with less emphasis on a rigid pre-declared graph. Dagster is asset-centric: instead of thinking in terms of tasks that run, you declare software-defined assets (tables, files, ML models) and their dependencies, and Dagster reasons about data lineage and freshness across those assets, which shifts the mental model from 'what runs when' to 'what data exists and how is it derived.'

Airflow's architecture separates the scheduler (which decides what should run), the executor (CeleryExecutor, KubernetesExecutor, etc., which decides how tasks are physically executed), and workers, all coordinated through a metadata database; this is battle-tested but has more operational surface area to run yourself or pay for via managed services like MWAA or Astronomer. Prefect uses 'deployments' tied to 'work pools' and 'workers,' with Prefect Cloud offering a hosted control plane while execution can happen on ephemeral infrastructure like ECS or Kubernetes jobs spun up per flow run, generally with a lighter local setup for development. Dagster runs via a webserver plus a daemon for scheduling, with 'runs' executed by configurable 'run launchers,' and it places strong emphasis on partitions (e.g., daily partitions of an asset) as a first-class concept for backfills and freshness checks, which Airflow supports but less natively.

Airflow historically required deploying a full local environment (webserver, scheduler, metadata DB) via Docker Compose or the Astro CLI just to iterate, though airflow dags test has improved this; its DAG-as-code model is verbose but explicit and predictable. Prefect flows are closer to plain Python functions and can be run directly with python my_flow.py without any server running at all, which gives a notably fast local feedback loop, at the cost of some behaviors (like caching and orchestration rules) only fully activating once connected to a Prefect server or Cloud. Dagster provides dagster dev to spin up a local webserver with hot-reloading of code changes and strong built-in support for typed inputs/outputs and asset checks, giving strong IDE-friendly type safety at the cost of a steeper initial learning curve around its asset/op/job vocabulary.

Choose Airflow when you need a mature, widely adopted orchestrator with the largest ecosystem of provider packages (AWS, GCP, Snowflake, dbt, and hundreds more), strong community support, and your workflows are primarily scheduled, relatively static batch pipelines; it's the safest default for most data engineering teams and has managed offerings like MWAA and Astronomer. Choose Prefect when your workflows need heavy runtime dynamism (conditional branching, retries with complex logic, or ad-hoc scripts triggered by events) and you want the fastest possible local development loop without deploying infrastructure first. Choose Dagster when data lineage, asset freshness, and strong typing across a data platform are top priorities, such as when multiple teams need to understand 'what feeds what' across dbt models, ML features, and reporting tables in one unified asset graph.