Prefect's server is the centralised coordination layer that stores flow run metadata, deployment configurations, schedules, and work queue state. Unlike Airflow's architecture — where server and workers share a metadata database and must be tightly coupled — Prefect separates the server (metadata only) from workers (execution only). Workers poll the server for scheduled runs and execute them in their own environment. This agent-based model means the server never needs to reach into the execution environment, enabling workers behind firewalls or in external cloud accounts without inbound network access.
Prefect Cloud is the managed version: Prefect hosts the server, and the team only runs workers. Prefect offers a generous free tier for individual and small-team use cases. Self-hosted Prefect Server runs using Docker Compose, requiring PostgreSQL and Redis. The operational overhead is significantly lower than self-hosted Airflow because the server is a single Docker image rather than a multi-component stack. For most data engineering teams, Prefect Cloud is the right choice unless data residency requirements prohibit sending run metadata to a managed service.
Analogy🏏Cricket
🏏 Think of it like cricket: Migrating from Airflow to Prefect is like the same bowling coach shifting from traditional Test cricket notation to a modern T20 analytics dashboard — the underlying ball-by-ball data (the business logic) is exactly the same. What changes is how the data is recorded, displayed, and acted upon. The yorker that Bumrah bowls in over 20 is identical whether it is recorded in the old scorebook (Airflow DAG file) or the new analytics platform (Prefect flow). The migration is a transcription exercise, not a strategy change — and a wise coach verifies that the runs, wickets, and economies match exactly between the old and new system before decommissioning the scorebook. That verification step is the whole heart of the migration: because the yorker is unchanged, the only honest test is to run the same over through both systems and confirm the recorded runs, wickets and economies match to the last digit before the old scorebook is thrown away. Rushing to burn the scorebook the moment the shiny dashboard lights up is how teams lose a season of records to a silent transcription slip. The coach keeps both systems running in parallel for a while, reconciles their outputs ball by ball, and only when every figure agrees does he trust the new dashboard alone — a transcription is only complete when you have proven nothing was lost in the copying.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.