What is Kafka Streams and how does it differ from a plain consumer?
Learn what Kafka Streams is and how it differs from a plain consumer, covering stateful processing, windowing, joins, and exactly-once semantics.
Expected Interview Answer
Kafka Streams is a client library for building real-time stream-processing applications that transform, aggregate, join, and enrich data in Kafka topics using a high-level DSL, whereas a plain consumer only reads raw records and leaves all processing logic to you.
A plain consumer gives you records and manual control over offsets; you must write your own state handling, windowing, joins, and fault recovery. Kafka Streams builds on top of consumers and producers but adds stateful operations, local state stores backed by changelog topics, event-time windowing, exactly-once processing, and automatic partition-based scaling. It runs as an ordinary library inside your application with no separate cluster, so you get a full processing engine without the overhead of a dedicated framework.
- High-level DSL for map, filter, join, and aggregate
- Built-in stateful processing with fault-tolerant local state stores
- Event-time windowing and out-of-order handling
- Exactly-once processing semantics
- Scales automatically by partition without a separate cluster
AI Mentor Explanation
A plain consumer is a scorer who just writes down each ball as it happens, doing no maths. Kafka Streams is the full statistics engine that, from that same ball feed, computes running strike rates, partnership totals, and required run rates over rolling windows. It keeps its own scorebook as a state store so if the scorer faints and is replaced, the new one rebuilds every stat exactly from the recorded log.
Step-by-Step Explanation
Step 1
Read the source topics
Define a KStream or KTable over input topics using the Streams DSL — no manual poll loop required.
Step 2
Apply transformations
Chain stateless operators (map, filter, flatMap) and stateful ones (groupBy, aggregate, join, windowedBy).
Step 3
Keep local state
Stateful operators store data in local state stores backed by compacted changelog topics for durability.
Step 4
Handle time
Use event-time windows and grace periods so out-of-order and late records are processed correctly.
Step 5
Write results
Emit processed records to output topics with a to() or through() operation, optionally with exactly-once semantics.
Step 6
Scale by running more instances
Launch more app instances sharing a group id; Streams rebalances partitions and their state stores across them.
What Interviewer Expects
- Clear line between a consumer (read) and Streams (process)
- Understanding of stateful operations and local state stores
- Awareness of event-time windowing and out-of-order handling
- Knowledge that Streams runs as a library, not a cluster
- Mention of exactly-once processing and changelog-backed fault tolerance
Common Mistakes
- Thinking Kafka Streams needs a separate processing cluster
- Confusing a KStream (event log) with a KTable (changelog snapshot)
- Reimplementing windowing and joins by hand on a plain consumer
- Ignoring state store sizing and changelog topics
- Assuming a plain consumer gives exactly-once by default
Best Answer (HR Friendly)
“A plain Kafka consumer just reads messages and hands them to you to process however you like. Kafka Streams is a library that adds a full toolkit for transforming, combining, and aggregating that data in real time, including memory of past events, so you do not have to build all that logic yourself.”
Code Example
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> lines = builder.stream("text-input");
lines.flatMapValues(v -> Arrays.asList(v.toLowerCase().split("\\W+")))
.groupBy((key, word) -> word)
.count()
.toStream()
.to("word-counts", Produced.with(Serdes.String(), Serdes.Long()));
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();Follow-up Questions
- What is the difference between a KStream and a KTable?
- How does Kafka Streams achieve exactly-once processing?
- How are state stores made fault tolerant?
- What is the difference between tumbling, hopping, and session windows?
- When would you use the Processor API instead of the DSL?
MCQ Practice
1. What does a plain Kafka consumer provide that Kafka Streams builds upon?
A plain consumer only reads records and manages offsets; Streams adds processing, state, and windowing on top of that foundation.
2. How is Kafka Streams state made fault tolerant?
Local state stores are backed by compacted changelog topics, so state can be rebuilt on another instance after failure.
3. How does Kafka Streams scale processing?
Additional instances with the same application id trigger a rebalance that redistributes partitions and their state stores.
Flash Cards
What is Kafka Streams? — A client library for real-time stateful stream processing over Kafka topics using a high-level DSL.
Streams vs plain consumer? — A consumer only reads records; Streams adds transforms, joins, aggregations, windowing, and fault-tolerant state.
Where does Kafka Streams run? — As a library inside your application, scaling by partitions — no separate processing cluster is required.
How is Streams state durable? — Local state stores are backed by compacted changelog topics that can rebuild state after a failure.