Kafka Quick Reference

Analogy categoryAnalogies

Why a Quick Reference Matters

Kafka's operational surface area is large enough that even experienced engineers keep a cheat sheet handy for topic management, consumer group inspection, and the handful of configuration properties that matter most day to day. This reference consolidates the CLI commands and settings you'll reach for repeatedly — creating and describing topics, checking consumer lag, and the core producer/consumer/broker properties — so you don't have to re-derive flag names under pressure during an incident.

🏏

Cricket analogy: It's like a fielding captain keeping a laminated card of field placements for different bowlers in their pocket — not because they don't know the game, but because recalling the exact setup instantly during a tense over saves precious seconds, just as a Kafka cheat sheet does during an incident.

Essential CLI Commands

The core operational commands revolve around kafka-topics.sh for topic management, kafka-console-producer.sh/kafka-console-consumer.sh for quick manual testing, and kafka-consumer-groups.sh for inspecting and resetting consumer lag — the single most useful command during an on-call incident, since it shows exactly how far behind each partition's consumer is. In newer Kafka versions running in KRaft mode (no ZooKeeper), the same tools work but you point --bootstrap-server at a broker rather than --zookeeper at a ZooKeeper ensemble, since ZooKeeper is no longer part of the architecture.

🏏

Cricket analogy: It's like a team analyst pulling up a player's real-time strike rate and run rate on a tablet mid-innings — kafka-consumer-groups.sh gives that same instant 'how far behind are we' snapshot for consumer lag during an incident.

bash

# Topic management
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic events --partitions 6 --replication-factor 3
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic events
kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic events --partitions 12
kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic events

# Manual produce/consume for testing
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic events
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic events --from-beginning

# Consumer group inspection (the most-used incident command)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-service
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group order-service \
  --topic events --reset-offsets --to-earliest --execute

Key Configuration Properties and Formulas

A handful of properties account for most of Kafka's day-to-day tuning surface: acks, enable.idempotence, and min.insync.replicas for producer durability; auto.offset.reset, enable.auto.commit, and max.poll.records for consumer behavior; and retention.ms, replication.factor, and min.insync.replicas at the topic level. Two formulas are worth memorizing: consumer lag is simply (latest offset in the partition) minus (last committed offset for that consumer group), and the maximum number of broker failures a topic can survive without becoming unavailable for writes is replication.factor minus min.insync.replicas.

🏏

Cricket analogy: It's like calculating a required run rate — runs needed divided by balls remaining — a simple formula every commentator recalls instantly; consumer lag is just as simple: latest offset minus committed offset.

min.insync.replicas is a topic-level (or broker default) setting, not a producer setting — it works together with the producer's acks=all to define durability. A common production baseline is replication.factor=3 with min.insync.replicas=2, tolerating one broker failure while still accepting writes.

kafka-consumer-groups.sh --reset-offsets is destructive and immediately changes what a live consumer group will read next; always run it with --dry-run first (the default without --execute) to preview the new offsets before committing to --execute.

kafka-topics.sh handles topic creation, description, partition increases, and deletion.
kafka-consumer-groups.sh --describe is the fastest way to check per-partition consumer lag during an incident.
In KRaft mode, use --bootstrap-server instead of the legacy --zookeeper flag across all CLI tools.
Consumer lag = latest partition offset minus the consumer group's last committed offset.
A topic tolerates (replication.factor − min.insync.replicas) broker failures while still accepting writes.
A common production baseline is replication.factor=3 with min.insync.replicas=2.
Always dry-run kafka-consumer-groups.sh --reset-offsets before adding --execute, since it's a destructive operation.