The producer and consumer are the two sides of every Kafka pipeline. A producer publishes messages to Kafka topics; a consumer reads messages from those topics. The `kafka-python` library provides the standard Python implementation of both, wrapping the Kafka protocol in a Python-friendly API that handles connection management, serialisation, retries, and offset tracking transparently. Understanding the key configuration options for both — particularly the producer's durability settings and the consumer's offset commit strategy — determines whether the pipeline achieves at-least-once, at-most-once, or effectively-exactly-once message delivery.
The most important conceptual distinction in Kafka pipeline design is between the producer's responsibility (getting a message into Kafka durably) and the consumer's responsibility (processing that message exactly once and committing the offset at the right moment). These are independent contracts — a durable producer combined with a consumer that commits before processing gives at-most-once delivery, while the same producer combined with a consumer that processes before committing gives at-least-once delivery. Neither combination alone gives exactly-once; that requires additional coordination at the destination.