The JobManager and TaskManager Split
A Flink cluster is built around two kinds of processes: the JobManager and one or more TaskManagers. The JobManager is the master process responsible for coordinating the distributed execution of a job: it receives the JobGraph submitted by the client, turns it into an ExecutionGraph, schedules tasks onto available TaskManagers, and coordinates checkpoints and recovery on failure. TaskManagers are the worker processes that actually execute the operators, each one running one or more tasks in parallel and exchanging data streams with other TaskManagers over the network.
Cricket analogy: The JobManager is like a team's captain such as Rohit Sharma setting the field and deciding bowling changes, while the TaskManagers are the fielders and bowlers actually executing those instructions on the ground.
Each TaskManager offers a fixed number of task slots, which represent a slice of its resources (primarily managed memory). A slot can host one parallel instance of a pipeline of chained operators, and the number of slots across all TaskManagers determines the maximum parallelism a cluster can support. Slot sharing, Flink's default behavior, allows different tasks from the same job (e.g., a source, a map, and a sink) to share a single slot, which improves resource utilization and simplifies capacity planning.
Cricket analogy: Task slots are like the eleven playing positions in a squad; a franchise like Mumbai Indians can only field as many specialists as it has slots, and an all-rounder occupying one slot can bowl and bat, similar to slot sharing.
Deployment Modes
Flink supports several deployment modes that determine how the JobManager and TaskManager processes are allocated. In Session Mode, a long-running cluster is started ahead of time and accepts multiple jobs, sharing resources across them, which suits many short-lived jobs but risks resource contention. In Per-Job Mode (deprecated in favor of Application Mode as of Flink 1.15), a dedicated cluster is spun up for a single job and torn down when it completes. Application Mode runs the job's main() method inside the cluster itself rather than on the client, reducing client-side resource usage and network transfer of the JobGraph, and is now the recommended approach for production deployments on Kubernetes or YARN.
Cricket analogy: Session Mode is like a stadium that hosts an entire IPL season with the same ground staff and floodlights shared across many matches, while Application Mode is like building a temporary pop-up ground dedicated to a single exhibition match.
Checkpoint Coordination and High Availability
The JobManager includes a Checkpoint Coordinator that periodically triggers distributed checkpoints by injecting checkpoint barriers into the source operators, which flow downstream alongside regular data. When all operators acknowledge a barrier, the checkpoint is considered complete and the resulting state snapshot is durably stored, typically in a distributed filesystem like S3 or HDFS. For high availability, Flink can run multiple JobManager instances with one active leader elected via Zookeeper or the Kubernetes leader election API, so that a JobManager failure triggers fast failover without losing job state.
Cricket analogy: Checkpoint barriers are like the drinks break markers in a Test match; once all fielders and the umpire acknowledge the break, the match state (score, overs) is safely recorded before play resumes.
# flink-conf.yaml (excerpt) — high availability configuration
high-availability.type: zookeeper
high-availability.zookeeper.quorum: zk1:2181,zk2:2181,zk3:2181
high-availability.storageDir: s3://flink-ha/recovery
high-availability.cluster-id: flink-prod-cluster
taskmanager.numberOfTaskSlots: 4
parallelism.default: 8
execution.checkpointing.interval: 60s
execution.checkpointing.mode: EXACTLY_ONCE
state.checkpoints.dir: s3://flink-ha/checkpointsPer-Job Mode was deprecated starting in Flink 1.15 in favor of Application Mode. New production deployments, especially on Kubernetes, should use Application Mode to avoid relying on a soon-to-be-removed deployment path.
- A Flink cluster consists of a JobManager (coordinator) and TaskManagers (workers).
- TaskManagers expose task slots, which bound cluster parallelism and support slot sharing by default.
- Session Mode runs a shared long-lived cluster; Application Mode runs the job's main() inside the cluster.
- Per-Job Mode is deprecated since Flink 1.15 in favor of Application Mode.
- The Checkpoint Coordinator injects barriers to create consistent distributed snapshots.
- High availability uses leader election (Zookeeper or Kubernetes) across multiple JobManagers.
- Checkpoints are durably stored in distributed filesystems like S3 or HDFS.
Practice what you learned
1. What is the primary responsibility of the JobManager in a Flink cluster?
2. What does a TaskManager's task slot represent?
3. Which deployment mode is recommended for production as of Flink 1.15, replacing Per-Job Mode?
4. What mechanism does Flink use to create consistent distributed checkpoints?
5. How does Flink achieve JobManager high availability?
Was this page helpful?
You May Also Like
What Is Apache Flink?
An introduction to Apache Flink, the open-source distributed engine for stateful computations over unbounded and bounded data streams.
The Flink Execution Model
How Flink transforms a program into a JobGraph and ExecutionGraph, schedules parallel tasks, and moves data between operators via streams and partitions.
Installing and Running Flink
A practical guide to installing Apache Flink locally, starting a standalone cluster, and submitting your first job through the CLI and Web UI.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics