Native Kubernetes Deployment Modes
Flink supports two native Kubernetes deployment modes: Application Mode, where the job's main class and dependencies are baked into a custom Docker image and the JobManager is launched per-application (recommended for production, since each job gets full isolation and its own JobManager lifecycle), and Session Mode, where a long-lived Flink cluster runs independently and multiple jobs are submitted to it afterward, sharing JobManager and TaskManager resources across jobs. Application Mode has become the default recommendation because a crashing job can only ever take down its own dedicated cluster, whereas a Session Mode cluster's shared JobManager is a single point of failure for every job submitted to it.
Cricket analogy: It's like the difference between a franchise owning its own dedicated training facility for one team (Application Mode: isolated, if it burns down only that team is affected) versus a shared national academy used by multiple franchises (Session Mode: efficient, but a facility-wide issue disrupts everyone).
The Flink Kubernetes Operator
The Flink Kubernetes Operator is a separate community project that introduces FlinkDeployment and FlinkSessionJob custom resources, letting you manage Flink jobs declaratively with kubectl apply the same way you'd manage a Deployment or StatefulSet, including automated savepoint-based upgrades when you change the job's image or configuration, built-in support for the adaptive scheduler's autoscaling, and automatic restart-from-checkpoint on TaskManager pod failures without manual intervention. Before the operator existed, teams had to script savepoint-stop-then-restart sequences themselves in CI/CD pipelines; the operator formalizes this into a reconciliation loop that watches the desired state in the CRD and drives the cluster toward it.
Cricket analogy: It's like a professional franchise having a dedicated team manager who automatically handles substitutions and rotation policy according to a written team charter, instead of the captain manually renegotiating lineup changes with the board before every match.
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: order-enrichment-job
spec:
image: registry.example.com/flink-jobs/order-enrichment:1.4.0
flinkVersion: v1_18
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
state.checkpoints.dir: s3://flink-checkpoints/order-enrichment
state.savepoints.dir: s3://flink-savepoints/order-enrichment
execution.checkpointing.interval: "30s"
serviceAccount: flink
jobManager:
resource:
memory: "2048m"
cpu: 1
taskManager:
resource:
memory: "4096m"
cpu: 2
job:
jarURI: local:///opt/flink/usrlib/order-enrichment.jar
parallelism: 8
upgradeMode: savepointState Backends, Checkpointing, and Pod Lifecycle
On Kubernetes, TaskManager pods are ephemeral — a pod eviction, node drain, or spot-instance reclaim can kill a TaskManager at any time — so production Flink deployments almost always use the RocksDB state backend with checkpoints written to a remote, durable object store like S3 or GCS rather than local disk, ensuring that a rescheduled pod on a completely different node can still recover full state from the last checkpoint. The high-availability configuration (typically backed by Kubernetes ConfigMaps in native HA mode, kubernetes.cluster-id and high-availability.type: kubernetes) is equally important, since it lets a new JobManager pod, if the original is evicted, discover the last checkpoint and resume the job without operator intervention.
Cricket analogy: It's like knowing any player could be substituted mid-match due to injury at any time, so the team keeps a shared, centrally accessible scorebook (remote checkpoint store) rather than relying on any one player's personal notes, which would be lost if that player left the field.
The Kubernetes Operator's autoscaler (built on the same mechanism as Reactive Scaling Mode) can automatically adjust job parallelism based on observed backpressure and utilization metrics, triggering a savepoint-based rescale without human involvement — configure it via the autoscaler.enabled and related properties in the FlinkDeployment spec.
Never rely on local TaskManager disk for checkpoint or savepoint storage in a Kubernetes deployment — pods are scheduled onto arbitrary nodes and local storage does not survive pod rescheduling, which will make recovery impossible after any node failure.
- Application Mode packages the job into its own image with a dedicated JobManager, providing isolation; Session Mode shares a long-lived cluster across jobs.
- The Flink Kubernetes Operator manages jobs declaratively via FlinkDeployment/FlinkSessionJob custom resources with kubectl apply.
- The operator automates savepoint-based upgrades, checkpoint-based recovery, and integrates with Flink's autoscaler.
- TaskManager pods are ephemeral on Kubernetes, so RocksDB state backend with remote object storage (S3/GCS) is standard for checkpoints.
- Native Kubernetes HA mode uses ConfigMaps so a replacement JobManager pod can discover and resume from the last checkpoint.
- Local disk should never be relied on for checkpoint/savepoint storage since pods can be rescheduled onto different nodes.
- The operator's autoscaler can trigger savepoint-based rescaling automatically based on backpressure and utilization signals.
Practice what you learned
1. Why is Application Mode generally recommended over Session Mode for production Flink jobs on Kubernetes?
2. What custom resources does the Flink Kubernetes Operator introduce?
3. Why is RocksDB with remote object storage typically preferred over local disk for checkpoints on Kubernetes?
4. What does native Kubernetes HA mode use to let a replacement JobManager resume a job?
5. What does the Flink Kubernetes Operator's autoscaler use to trigger a rescale?
Was this page helpful?
You May Also Like
Scaling and Parallelism
Learn how Flink parallelizes work across task slots and how to scale jobs up or down without losing correctness.
Flink with Kafka
Understand how to build reliable, exactly-once streaming pipelines by connecting Apache Flink to Apache Kafka as both source and sink.
Monitoring with the Flink Dashboard
Learn to read Flink's web dashboard and metrics system to diagnose backpressure, checkpoint failures, and skewed workloads.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics