Why Kafka Monitoring Is Non-Negotiable
Kafka clusters fail silently far more often than they fail loudly: a broker can stay technically 'up' while under-replicated partitions pile up, consumer lag balloons, or disk usage creeps toward capacity, none of which triggers an obvious crash but all of which degrade correctness and latency for every application depending on the cluster. Effective monitoring means tracking broker-level, topic-level, and client-level metrics continuously, not just checking whether the process is alive.
Cricket analogy: A bowler can look fine standing at the top of his mark over after over while his workload quietly creeps past a safe threshold, and only a physio tracking exact overs bowled and speed drop-off, not just whether he's still on the field, catches the risk before a stress fracture.
The single most important health metric for a Kafka cluster is UnderReplicatedPartitions, exposed as a broker-level JMX metric; any non-zero value means at least one partition's follower replicas have fallen out of sync with the leader, risking data loss if the leader fails before followers catch up. Close behind it is consumer lag, the difference between the latest offset produced and the latest offset committed by a consumer group, which directly measures how far behind real-time a downstream application is running.
Cricket analogy: UnderReplicatedPartitions is like a fielding drill where the backup fielder covering a boundary has fallen a step behind the primary fielder's positioning; nothing has gone wrong yet, but if the primary fielder pulls up injured mid-play, there's now a real gap uncovered.
# Check under-replicated partitions across the cluster
kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions
# Query consumer lag for a specific group
kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
--describe --group order-processing-group
# Example Prometheus alert rule (via JMX exporter)
- alert: KafkaUnderReplicatedPartitions
expr: kafka_server_replicamanager_underreplicatedpartitions > 0
for: 2m
labels:
severity: critical
annotations:
summary: "Broker {{ $labels.instance }} has under-replicated partitions"Most production teams export Kafka's JMX metrics via the Prometheus JMX exporter as a Java agent, scrape them with Prometheus, and visualize broker, topic, and consumer group health in Grafana dashboards alongside standard host metrics like disk and network I/O.
Broker Resource Metrics and Alerting Strategy
Beyond replication and lag, disk usage per broker deserves dedicated alerting because Kafka retains data on disk for the configured retention period and a broker that fills its disk will refuse writes and can crash; RequestQueueSize and network processor idle percentage reveal whether brokers are becoming a bottleneck under load, since a consistently low network processor idle ratio signals the broker's I/O threads are saturated and requests are queuing.
Cricket analogy: Disk usage alerting is like a groundskeeper tracking exactly how much of the pitch's protective covering material remains before a rain-heavy season, because running out mid-tournament means matches simply can't be protected and get abandoned.
Never rely solely on log.retention.bytes or log.retention.hours to prevent disk exhaustion; a sudden producer throughput spike or a stuck consumer preventing log compaction can fill disks faster than retention policies clean them up, so active disk usage alerting at 75-80% capacity is essential.
- UnderReplicatedPartitions is the single most critical broker health metric; any non-zero value demands immediate investigation.
- Consumer lag measures how far a consumer group trails behind the latest produced offset and directly reflects processing delay.
- JMX metrics exported via the Prometheus JMX exporter and visualized in Grafana form the standard production monitoring stack.
- Disk usage per broker must be alerted on proactively since Kafka relies on disk for retention and full disks halt writes.
- Network processor idle percentage reveals I/O thread saturation and impending request queuing bottlenecks.
- kafka-consumer-groups.sh --describe is the fastest CLI way to check lag for a specific consumer group manually.
- Monitoring must cover broker, topic, and client-level metrics together, not just process-level 'is it running' checks.
Practice what you learned
1. Which metric is considered the single most critical broker-level health indicator in Kafka?
2. What does consumer lag measure?
3. What is the standard open-source stack for exporting and visualizing Kafka JMX metrics?
4. Why is relying solely on log.retention settings insufficient to prevent disk exhaustion?
5. A consistently low network processor idle percentage on a broker suggests what?
Was this page helpful?
You May Also Like
Common Kafka Failure Modes
The most frequent ways Kafka clusters and clients fail in production, and how to recognize and recover from each.
Scaling Kafka Clusters
Strategies for growing Kafka capacity: adding brokers, reassigning partitions, and choosing the right partition count upfront.
Kafka Security Basics
Core mechanisms for securing a Kafka cluster: encryption in transit, authentication, and fine-grained authorization.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics