Running Airflow on Docker Compose
The official docker-compose.yaml distributed by the Airflow project runs the webserver, scheduler, triggerer, and (with CeleryExecutor) worker and Redis/Flower containers alongside a Postgres metadata database, all sharing a mounted ./dags, ./logs, and ./plugins volume so code changes on the host are picked up without rebuilding images. This setup is well suited to local development and small deployments, but every component sharing one Docker network means resource limits (CPU/memory) apply per-container, not per-task, so a single memory-heavy task can starve other containers on the same host.
Cricket analogy: It's like a club team sharing one training ground with separate nets for batting, bowling, and fielding drills, all pulled from the same equipment shed — convenient for practice, but if one drill hogs all the bowling machines, the others wait.
Running Airflow on Kubernetes: The KubernetesExecutor
The KubernetesExecutor launches a brand-new Kubernetes Pod for every single task instance, using a base Pod template (image, resource requests/limits, node selectors) that can be overridden per-task via executor_config. This gives each task fully isolated CPU, memory, and dependencies — a task needing a GPU node or 16GB of RAM gets its own Pod spec, without affecting any other task — and Pods are automatically cleaned up (or left for debugging, if delete_worker_pods is False) once the task finishes, so idle capacity isn't reserved between runs.
Cricket analogy: It's like a franchise flying in a specialist bowler with their own kit and training regimen just for one specific match situation, rather than keeping every specialist on the payroll and training ground year-round.
# Per-task pod override via executor_config (KubernetesExecutor)
from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import (
KubernetesExecutor,
)
from kubernetes.client import models as k8s
heavy_task = PythonOperator(
task_id="train_model",
python_callable=train_model,
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
requests={"memory": "16Gi", "cpu": "4"},
limits={"memory": "16Gi", "cpu": "4", "nvidia.com/gpu": "1"},
),
)
]
)
)
},
)
KubernetesExecutor vs. CeleryExecutor vs. KubernetesPodOperator
It's worth distinguishing three related-but-different concepts: the KubernetesExecutor changes how Airflow schedules and isolates every task in the whole deployment (one Pod per task instance); the CeleryExecutor instead runs tasks on a fixed pool of long-lived worker processes (often deployed as Kubernetes Pods themselves via the Celery+Kubernetes hybrid pattern) that pick up work from a Redis/RabbitMQ queue; and the KubernetesPodOperator is a single operator, usable under any executor, that runs one specific task's logic inside an arbitrary container image, independent of what image the rest of Airflow uses.
Cricket analogy: It's like choosing between fielding a fresh substitute for every single ball bowled (KubernetesExecutor), keeping a fixed squad of twelve rotating through overs (CeleryExecutor), or bringing in one specialist guest player just for a single delivery regardless of the squad system (KubernetesPodOperator).
Common Pitfall: Image and DAG Sync Drift
A frequent operational headache in containerized Airflow is DAG sync drift: if the scheduler, workers, and webserver each pull DAG files from a different source (a Git-sync sidecar, a baked-in image layer, a shared volume with inconsistent mount timing) they can briefly disagree on what a DAG looks like, causing a task to run with a different code version than the UI displays. The standard fix is a git-sync sidecar container attached to every Pod that needs DAG code, pulling from the same Git commit on a shared interval, so all components stay consistent.
Cricket analogy: It's like the scoreboard operator, the third umpire, and the on-field umpire all working from slightly different versions of the rulebook due to a late rule change — leading to a decision dispute mid-match until everyone syncs to the same edition.
Ensure every Pod that needs DAG code (scheduler, workers, webserver, triggerer) pulls from the identical source and commit — typically via a shared git-sync sidecar on a consistent interval, not independently baked images with different build times. Mismatched DAG versions across components are a common source of confusing 'it ran differently than the UI showed' incidents.
- Docker Compose's official setup shares one Docker network and host resources across all Airflow components — good for local dev, limited for production isolation.
KubernetesExecutorlaunches one Kubernetes Pod per task instance, giving full per-task resource and dependency isolation viaexecutor_config/pod_override.CeleryExecutoruses a fixed pool of long-lived workers pulling from a queue, trading per-task isolation for lower Pod-creation overhead.KubernetesPodOperatorruns a single task in an arbitrary container image and works under any executor — it solves a different problem than the executor choice itself.- DAG sync drift across scheduler/worker/webserver Pods is a common Kubernetes-deployment pitfall, typically fixed with a shared
git-syncsidecar. - Resource requests/limits should be set per-task via
executor_configon KubernetesExecutor so heavy tasks don't starve lightweight ones. - Choose KubernetesExecutor for heterogeneous task resource needs and strong isolation; choose CeleryExecutor for lower latency and simpler ops at moderate scale.
Practice what you learned
1. What is the key behavioral difference of the `KubernetesExecutor` compared to `CeleryExecutor`?
2. How do you give one specific task extra memory and a GPU under the KubernetesExecutor?
3. What problem does the `KubernetesPodOperator` solve that is independent of executor choice?
4. What commonly causes DAG sync drift between scheduler, worker, and webserver in a Kubernetes deployment?
Was this page helpful?
You May Also Like
Monitoring and Logging
How Airflow's per-task logging works, how to configure remote logging for distributed deployments, and how to build alerting that avoids fatigue while catching real incidents.
DAG Versioning and CI/CD
How Airflow tracks DAG structure changes over time via DAG Versioning, and how to build a GitOps-style CI/CD pipeline that tests and deploys DAGs safely.
Testing DAGs
How to validate Airflow DAGs at three layers — import integrity, structural correctness, and task logic — using pytest, DagBag, and dag.test().
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics