Why Stream Processing Is Different
Traditional batch processing operates on a bounded dataset that has already fully arrived, computing an answer once and moving on. Stream processing instead operates on an unbounded, continuously arriving sequence of events, producing and refining results incrementally as new data shows up, which means the notion of a computation ever being 'finished' has to be redefined in terms of time windows and watermarks rather than a dataset's end.
Cricket analogy: Batch processing is like calculating a player's final career average after retirement from the complete scorecard archive, while stream processing is like a live strike-rate ticker that updates after every single ball bowled in an ongoing IPL match.
Event Time vs. Processing Time
Event time is the timestamp of when something actually happened in the real world, typically embedded in the record itself, while processing time is when the stream processing system happens to observe and handle that record; these two can diverge significantly due to network delays, retries, or a mobile client buffering events offline. Correct stream processing systems, including Kafka Streams, let you choose which timestamp drives windowing decisions, because grouping events by processing time alone can badly misrepresent when things actually occurred.
Cricket analogy: Event time is like the exact moment a boundary was struck on the field, while processing time is like when that highlight finally appears on a fan's phone after a satellite broadcast delay, and a good scoring system must key off the former, not the latter.
Windowing and Watermarks
Because a stream never ends, aggregations like 'count events per user' need a boundary to become meaningful, which is what windows provide: tumbling windows divide time into fixed, non-overlapping buckets, hopping windows allow overlapping buckets that advance by a smaller step, and session windows group events by periods of activity separated by a gap of inactivity. Watermarks (or, in Kafka Streams, grace periods) address the reality that late-arriving events will still show up after a window's nominal end time, defining how long a window stays open to accept stragglers before being finalized and emitted.
Cricket analogy: A tumbling window is like scoring stats per fixed over in a T20 innings, non-overlapping six-ball buckets, while a session window is like grouping a batting partnership's runs by continuous time at the crease until a wicket falls creates a gap.
Joins Between Streams
Stream processing systems support joining two streams much like relational joins, but time-bounded: a stream-stream join only matches records that fall within a defined time window of each other, since neither side is guaranteed to 'wait' for a match arriving an hour later. A stream-table join, by contrast, looks up the current value in a KTable for each incoming stream record, which is useful for enriching events, such as attaching current customer profile data to every order event, without needing the profile update to be temporally close to the order.
Cricket analogy: A stream-stream join is like matching a bowler's delivery event with the batsman's shot event only if they happened within the same ball, while a stream-table join is like enriching every ball event with the batsman's current season average looked up from a standings table.
// A tumbling window aggregation over a 1-minute window with a 10-second grace period
KTable<Windowed<String>, Long> pageViewsPerMinute = pageViews
.groupByKey()
.windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(1), Duration.ofSeconds(10)))
.count(Materialized.as("page-views-per-minute"));
// A stream-table join enriching order events with current customer tier
KStream<String, EnrichedOrder> enrichedOrders = orders.join(
customerTiers,
(order, tier) -> new EnrichedOrder(order, tier)
);
Choosing processing time instead of event time for windowing can silently produce misleading results, records from a mobile app that buffered offline for an hour and then reconnected will all land in the 'current' window rather than the window they actually occurred in.
- Stream processing operates on unbounded, continuously arriving data instead of a fixed, already-complete dataset.
- Event time reflects when something actually happened; processing time reflects when the system observed it, and they can diverge.
- Tumbling windows are fixed and non-overlapping; hopping windows overlap and advance by a smaller step; session windows are defined by gaps in activity.
- Grace periods (watermarks) determine how long a window stays open to accept late-arriving events before finalizing.
- Stream-stream joins are time-bounded; stream-table joins look up current state without a time constraint.
- Using the wrong timestamp semantics for windowing can silently produce incorrect aggregations.
Practice what you learned
1. What distinguishes a session window from a tumbling window?
2. Why might event time and processing time differ for the same record?
3. What is the purpose of a grace period (watermark) on a window?
4. What is a key difference between a stream-stream join and a stream-table join?
Was this page helpful?
You May Also Like
Kafka Streams Basics
An introduction to Kafka Streams, the client library for building stateful stream processing applications directly on top of Kafka topics.
Kafka Connect Explained
How Kafka Connect standardizes moving data into and out of Kafka using configurable connectors instead of custom integration code.
Schema Registry and Avro
How the Schema Registry and Avro serialization give Kafka's opaque byte messages an enforced, evolvable structure across producers and consumers.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics