Batch vs Stream Processing: How Do You Choose?
Compare batch and stream processing — bounded vs unbounded data, windowing, watermarks, and the Lambda architecture.
Expected Interview Answer
Batch processing runs computations over a bounded, already-collected chunk of data on a schedule, trading latency for simplicity and easy correctness, while stream processing computes continuously over an unbounded flow of events as they arrive, trading added complexity for near-real-time results.
Batch systems (like a nightly Spark job over a data warehouse) read a fixed dataset, process it start to finish, and produce a complete, easily-reproducible result; because the input is bounded and static, retries and reprocessing are simple, and tools can use exact aggregations without worrying about late-arriving data. Stream systems (like Kafka Streams or Flink) process each event as it arrives, so results update continuously, but they must explicitly handle out-of-order and late-arriving events using watermarks, define what a “window” of time means (tumbling, sliding, session windows), and choose between at-least-once and exactly-once processing semantics, all of which batch systems get for free by simply waiting until all data exists. In practice most large systems adopt a Lambda or Kappa architecture: a stream layer serves fast, approximate real-time results, while a batch layer (or a stream layer replayed from a durable log) periodically recomputes exact, corrected results, giving both low latency and eventual correctness.
- Batch processing gives simple, reproducible, exactly-correct results over bounded data
- Stream processing delivers near-real-time results as events happen, not after a delay
- Windowing and watermarks let stream systems approximate correctness despite out-of-order events
- A Lambda/Kappa architecture combines both to get low latency and eventual correctness
AI Mentor Explanation
Batch processing is like compiling the full scorecard analysis after the match has completely ended, using every ball bowled to produce one final, unambiguous statistics report. Stream processing is like the live run-rate ticker that updates after every single ball while the match is still in progress, giving an immediate but provisional picture that can shift as more balls are bowled. The ticker has to handle oddities like a delayed replay overturning a decision, the same way stream systems handle late-arriving events. Combining both — a live ticker during play plus the definitive scorecard afterward — mirrors exactly how batch and stream processing are used together.
Step-by-Step Explanation
Step 1
Characterize latency needs
Decide whether results are needed within seconds (favor streaming) or a scheduled delay is acceptable (favor batch).
Step 2
Assess data boundedness
Bounded, already-collected datasets suit batch jobs; unbounded, continuously arriving events suit stream processing.
Step 3
Design for correctness
For streaming, define windowing (tumbling/sliding/session) and watermarks to handle late and out-of-order events.
Step 4
Combine when needed
Use a Lambda/Kappa architecture: fast approximate stream results plus periodic exact batch recomputation.
What Interviewer Expects
- Contrasts bounded vs unbounded data as the core distinction
- Names concrete tools: batch (Spark, Hadoop/MapReduce) vs stream (Kafka Streams, Flink)
- Explains windowing and watermarks for handling late/out-of-order events
- Mentions Lambda or Kappa architecture as the real-world hybrid answer
Common Mistakes
- Treating streaming as strictly “better” without acknowledging its added complexity
- Not mentioning how late or out-of-order events are handled in streaming
- Confusing stream processing with simple request/response APIs
- Forgetting that batch gives easy reproducibility that streaming has to work harder for
Best Answer (HR Friendly)
“Batch processing crunches a full chunk of data at once, usually on a schedule, which is simple and gives one clean final answer. Stream processing handles data continuously as it arrives, so results update in near real time, but it has to deal with things arriving late or out of order. Many systems use both: a fast, live stream for immediate insight and a batch job later to produce the exact, corrected numbers.”
Code Example
# Batch: runs once over a bounded, already-collected dataset
def nightly_revenue_report(date):
orders = warehouse.read_orders(date=date) # fixed, complete dataset
total = sum(o.amount for o in orders)
warehouse.write_report(date, total)
# Stream: continuously aggregates over an unbounded event flow
def revenue_stream_job(event_stream):
window = TumblingWindow(size_minutes=5, watermark_minutes=2)
for event in event_stream: # events keep arriving indefinitely
window.add(event.timestamp, event.amount)
if window.is_closed(event.timestamp):
emit(window.key, window.sum()) # near real-time, approximate
window.advance()Follow-up Questions
- What is a watermark in stream processing and why is it necessary?
- What is the difference between tumbling, sliding, and session windows?
- How does a Lambda architecture differ from a Kappa architecture?
- How do exactly-once processing semantics differ from at-least-once in a streaming system?
MCQ Practice
1. What is the fundamental difference between batch and stream processing?
Batch operates over a fixed, already-collected dataset, while streaming continuously processes events as they arrive with no defined end.
2. What problem do watermarks solve in stream processing?
Watermarks estimate how long to wait for late events before closing a window and emitting a result, balancing latency against completeness.
3. What does a Lambda architecture combine?
Lambda architecture pairs a low-latency streaming layer for fast approximate results with a batch layer that later produces exact, corrected results.
Flash Cards
Core difference: batch vs stream? — Batch processes a bounded, complete dataset; stream processes an unbounded, continuous flow of events.
Why do streams need watermarks? — To decide when a time window can be closed despite events arriving late or out of order.
Name a batch and a stream tool. — Batch: Spark/Hadoop. Stream: Kafka Streams or Flink.
What is a Lambda architecture? — A hybrid combining a fast approximate stream layer with a periodic exact batch layer for corrected results.