Monitoring with the Flink Dashboard

Analogy categoryAnalogies

Navigating the Web UI

The Flink Dashboard, served by the JobManager on port 8081 by default, shows the running job graph, per-task subtask metrics, checkpoint history, and TaskManager resource usage in one place. The Job Graph view lets you click into any operator and see per-subtask records-in, records-out, and bytes, which is usually the first place to look when throughput drops — a healthy pipeline shows records-out roughly matching records-in across each stage, while a stalled stage shows one operator's records-out flatlining while upstream records-in keeps climbing.

🏏

Cricket analogy: It's like a stadium's giant scoreboard showing runs-per-over for each batting partnership — a healthy chase shows steadily climbing totals, while a stalled partnership shows the run rate flatlining even as balls keep being bowled, immediately flagging where the innings is stuck.

Diagnosing Backpressure

Backpressure occurs when a downstream operator can't keep up and its input buffers fill, causing Flink's credit-based flow control to slow upstream operators to match — the dashboard's Backpressure tab (or the busy/backpressured ratio metrics in newer Flink versions) samples each subtask's thread stack to estimate the percentage of time spent blocked waiting to emit records, with sustained values above roughly 50% flagged as high backpressure. The key diagnostic skill is tracing backpressure to its source: if operator C is slow and blocking operator B which blocks operator A, all three subtasks will show as backpressured, but only C is the actual bottleneck — you fix C, not A or B.

🏏

Cricket analogy: It's like a slow over-rate caused by one fielder constantly having to chase the ball to the boundary — the bowler, keeper, and everyone else look 'stalled' waiting on the throw back, but the actual bottleneck is that one fielder's fitness, not the bowler's pace.

bash

# Query key metrics directly from the REST API instead of clicking through the UI
curl http://jobmanager:8081/jobs/<job-id>/vertices/<vertex-id>/backpressure

curl 'http://jobmanager:8081/jobs/<job-id>/checkpoints' | jq '.latest.completed'

# Per-subtask throughput metrics
curl 'http://jobmanager:8081/jobs/<job-id>/vertices/<vertex-id>/subtasks/metrics?get=numRecordsInPerSecond,numRecordsOutPerSecond'

Checkpoint Health and Metrics Systems

The Checkpoints tab shows history, duration, and size of each checkpoint, along with per-task alignment time — a rising checkpoint duration trend usually signals either backpressure (barriers can't flow through congested buffers) or state size growth outpacing your storage throughput; the 'end to end duration' versus 'sync/async duration' breakdown tells you whether the bottleneck is barrier alignment (network/backpressure) or the actual state snapshot write (storage). For production alerting, the dashboard alone isn't enough — teams export Flink's metrics (via the built-in reporters for Prometheus, Datadog, or JMX) to a proper time-series system and alert on numRecordsOutPerSecond dropping to zero, checkpoint duration exceeding a threshold, or restartCount incrementing unexpectedly.

🏏

Cricket analogy: It's like distinguishing a slow over-rate caused by field changes (alignment/network, like players walking to new positions) from one caused by ball-tampering checks needing a physical inspection (storage write) — same symptom, different root cause requiring different fixes.

Enable unaligned checkpoints (execution.checkpointing.unaligned.enabled: true) when backpressure is causing checkpoint alignment to dominate checkpoint duration — barriers overtake buffered records instead of waiting for them, trading slightly larger checkpoint size for much faster completion under load.

A steadily increasing 'restartCount' metric with no corresponding alert usually means a job is stuck in a crash-loop that recovers just long enough to look healthy in a quick dashboard glance — always alert on restart count, not just on job status.

The Flink Dashboard (default port 8081) shows the job graph, per-subtask I/O metrics, checkpoint history, and TaskManager resource usage.
Backpressure is measured as the percentage of time a subtask spends blocked waiting to emit records; sustained values above ~50% indicate a problem.
Backpressure propagates upstream, so multiple operators may appear backpressured when only the most downstream one is the actual bottleneck.
Checkpoint duration trends reveal whether the bottleneck is barrier alignment (network/backpressure) or the actual state snapshot write (storage).
Unaligned checkpoints help checkpoint completion under sustained backpressure by letting barriers overtake buffered records.
Production monitoring should export metrics to Prometheus/Datadog/JMX rather than relying solely on manual dashboard checks.
Alert on restartCount incrementing, not just job status, since crash-loops can briefly appear healthy between restarts.