Airflow with Docker and Kubernetes

Analogy categoryAnalogies

Running Airflow on Docker Compose

The official docker-compose.yaml distributed by the Airflow project runs the webserver, scheduler, triggerer, and (with CeleryExecutor) worker and Redis/Flower containers alongside a Postgres metadata database, all sharing a mounted ./dags, ./logs, and ./plugins volume so code changes on the host are picked up without rebuilding images. This setup is well suited to local development and small deployments, but every component sharing one Docker network means resource limits (CPU/memory) apply per-container, not per-task, so a single memory-heavy task can starve other containers on the same host.

🏏

Cricket analogy: It's like a club team sharing one training ground with separate nets for batting, bowling, and fielding drills, all pulled from the same equipment shed — convenient for practice, but if one drill hogs all the bowling machines, the others wait.

Running Airflow on Kubernetes: The KubernetesExecutor

The KubernetesExecutor launches a brand-new Kubernetes Pod for every single task instance, using a base Pod template (image, resource requests/limits, node selectors) that can be overridden per-task via executor_config. This gives each task fully isolated CPU, memory, and dependencies — a task needing a GPU node or 16GB of RAM gets its own Pod spec, without affecting any other task — and Pods are automatically cleaned up (or left for debugging, if delete_worker_pods is False) once the task finishes, so idle capacity isn't reserved between runs.

🏏

Cricket analogy: It's like a franchise flying in a specialist bowler with their own kit and training regimen just for one specific match situation, rather than keeping every specialist on the payroll and training ground year-round.

yaml

# Per-task pod override via executor_config (KubernetesExecutor)
from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import (
    KubernetesExecutor,
)
from kubernetes.client import models as k8s

heavy_task = PythonOperator(
    task_id="train_model",
    python_callable=train_model,
    executor_config={
        "pod_override": k8s.V1Pod(
            spec=k8s.V1PodSpec(
                containers=[
                    k8s.V1Container(
                        name="base",
                        resources=k8s.V1ResourceRequirements(
                            requests={"memory": "16Gi", "cpu": "4"},
                            limits={"memory": "16Gi", "cpu": "4", "nvidia.com/gpu": "1"},
                        ),
                    )
                ]
            )
        )
    },
)

KubernetesExecutor vs. CeleryExecutor vs. KubernetesPodOperator

It's worth distinguishing three related-but-different concepts: the KubernetesExecutor changes how Airflow schedules and isolates every task in the whole deployment (one Pod per task instance); the CeleryExecutor instead runs tasks on a fixed pool of long-lived worker processes (often deployed as Kubernetes Pods themselves via the Celery+Kubernetes hybrid pattern) that pick up work from a Redis/RabbitMQ queue; and the KubernetesPodOperator is a single operator, usable under any executor, that runs one specific task's logic inside an arbitrary container image, independent of what image the rest of Airflow uses.

🏏

Cricket analogy: It's like choosing between fielding a fresh substitute for every single ball bowled (KubernetesExecutor), keeping a fixed squad of twelve rotating through overs (CeleryExecutor), or bringing in one specialist guest player just for a single delivery regardless of the squad system (KubernetesPodOperator).

Common Pitfall: Image and DAG Sync Drift

A frequent operational headache in containerized Airflow is DAG sync drift: if the scheduler, workers, and webserver each pull DAG files from a different source (a Git-sync sidecar, a baked-in image layer, a shared volume with inconsistent mount timing) they can briefly disagree on what a DAG looks like, causing a task to run with a different code version than the UI displays. The standard fix is a git-sync sidecar container attached to every Pod that needs DAG code, pulling from the same Git commit on a shared interval, so all components stay consistent.

🏏

Cricket analogy: It's like the scoreboard operator, the third umpire, and the on-field umpire all working from slightly different versions of the rulebook due to a late rule change — leading to a decision dispute mid-match until everyone syncs to the same edition.

Ensure every Pod that needs DAG code (scheduler, workers, webserver, triggerer) pulls from the identical source and commit — typically via a shared git-sync sidecar on a consistent interval, not independently baked images with different build times. Mismatched DAG versions across components are a common source of confusing 'it ran differently than the UI showed' incidents.

Docker Compose's official setup shares one Docker network and host resources across all Airflow components — good for local dev, limited for production isolation.
KubernetesExecutor launches one Kubernetes Pod per task instance, giving full per-task resource and dependency isolation via executor_config/pod_override.
CeleryExecutor uses a fixed pool of long-lived workers pulling from a queue, trading per-task isolation for lower Pod-creation overhead.
KubernetesPodOperator runs a single task in an arbitrary container image and works under any executor — it solves a different problem than the executor choice itself.
DAG sync drift across scheduler/worker/webserver Pods is a common Kubernetes-deployment pitfall, typically fixed with a shared git-sync sidecar.
Resource requests/limits should be set per-task via executor_config on KubernetesExecutor so heavy tasks don't starve lightweight ones.
Choose KubernetesExecutor for heterogeneous task resource needs and strong isolation; choose CeleryExecutor for lower latency and simpler ops at moderate scale.