What Is a Kafka Producer?
A Kafka producer is a client application that publishes (writes) records to one or more Kafka topics. Each record consists of an optional key, a value, and an optional list of headers, and the producer client library handles connecting to the cluster, discovering which broker leads each partition, and routing the record accordingly. Producers do not push directly to consumers; they only write to the topic's partition log, decoupling the sender from whoever eventually reads the data.
Cricket analogy: Like a batsman deliberately placing a shot through a specific fielding zone rather than hitting randomly, e.g., Virat Kohli threading a cover drive through a gap, a producer deliberately routes each record to a specific partition.
Batching and Serialization
Producers rarely send one record per network call; instead they accumulate records destined for the same partition into a batch, controlled by batch.size (max bytes per batch) and linger.ms (how long to wait for more records before sending). Larger batches, optionally combined with compression (compression.type=snappy, lz4, or zstd), dramatically reduce per-message overhead and improve throughput at the cost of a small added latency. Before batching, the key and value objects are converted to bytes using a key.serializer and value.serializer (for example StringSerializer or a schema-registry-aware AvroSerializer).
Cricket analogy: Like a team manager waiting to fill a whole bus with players before departing for the ground instead of sending a separate car per player, similar to how linger.ms holds records briefly to fill a batch.
Acknowledgements, Retries, and Idempotence
The acks setting controls durability: acks=0 fires and forgets, acks=1 waits for the partition leader to write the record, and acks=all (the safest, and recommended for production) waits for the leader plus all in-sync replicas defined by min.insync.replicas. Combined with retries and delivery.timeout.ms, a producer can survive transient broker failures without losing data. Because retries can create duplicates, enabling the idempotent producer (enable.idempotence=true, the default when acks=all in modern Kafka) assigns each producer a producer ID and per-partition sequence numbers so the broker can detect and drop duplicate retries.
Cricket analogy: Like the third umpire requiring confirmation from every replay angle (acks=all) before signaling a run-out, versus trusting the on-field umpire's single call (acks=1).
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
props.put("enable.idempotence", "true");
props.put("compression.type", "lz4");
props.put("linger.ms", 20);
props.put("batch.size", 32768);
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
ProducerRecord<String, String> record =
new ProducerRecord<>("orders", "order-4821", "{\"status\":\"CREATED\"}");
producer.send(record, (metadata, exception) -> {
if (exception != null) {
exception.printStackTrace();
} else {
System.out.printf("Sent to partition %d at offset %d%n",
metadata.partition(), metadata.offset());
}
});
}For production topics, pair acks=all with min.insync.replicas=2 on a topic with replication.factor=3. This means a write is only acknowledged once it exists on the leader and at least one follower, so a single broker failure never loses acknowledged data.
- A producer serializes a record's key and value into bytes before sending it to a topic.
- Records are grouped into batches controlled by batch.size and linger.ms to improve throughput.
- Compression (snappy, lz4, zstd) reduces network and storage cost for batched records.
- acks controls durability: 0 (fire-and-forget), 1 (leader only), all (leader + in-sync replicas).
- enable.idempotence=true prevents duplicate writes caused by producer-side retries.
- min.insync.replicas works together with acks=all to guarantee no acknowledged write is lost.
- The producer only writes to partitions; it never talks directly to consumers.
Practice what you learned
1. Which producer setting provides the strongest durability guarantee?
2. What does linger.ms control?
3. What problem does enable.idempotence=true solve?
4. Which component converts a Java object into bytes before a producer sends it?
5. With acks=0, what happens if the broker crashes right after receiving a record?
Was this page helpful?
You May Also Like
Kafka Consumers Explained
How Kafka consumer clients poll for records, track their read position with offsets, and deserialize messages back into usable objects.
Message Keys and Partitioning
How Kafka uses a record's key to choose a partition, why that determines per-key ordering, and the risks of custom partitioners and changing partition counts.
Delivery Guarantees in Kafka
The at-most-once, at-least-once, and exactly-once delivery semantics Kafka supports, and the producer/consumer configuration choices that determine which one you get.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics