Apache Kafka is the dominant distributed event streaming platform, connecting producers that generate events to consumers that process them through a durable, fault-tolerant, high-throughput log. Originally built at LinkedIn to handle billions of events per day, Kafka decouples the systems that produce data from the systems that consume them — a producer does not need to know who consumes its events, and a consumer does not need to be running when the event is produced. This decoupling enables asynchronous, independently scalable architectures impossible with direct service-to-service calls.
Kafka's architecture is built around three fundamental abstractions: topics, which are named, ordered, durable logs of events; partitions, which are the unit of parallelism and ordering within a topic; and brokers, which are the server processes that store and serve partition data. Understanding how these three components interact — how topics are split across partitions, how partitions are distributed across brokers, how replication provides fault tolerance — is the foundation for designing Kafka pipelines that are both high-throughput and reliable.