What is Lambda Architecture in System Design?
Learn Lambda architecture: batch layer, speed layer, serving layer merge, trade-offs, and how it compares to Kappa architecture.
Expected Interview Answer
Lambda architecture is a data-processing pattern that runs a slow, accurate batch layer over the complete historical dataset alongside a fast speed layer over recent events, merging both at query time through a serving layer so users get results that are both comprehensive and up to date.
The batch layer periodically recomputes precomputed views from the full immutable dataset, which is slow but produces correct, complete results because it can reprocess everything from scratch if logic changes. The speed layer processes only the most recent events with a stream processor to compensate for the batch layer’s latency, offering low-latency approximate results that get corrected once the batch layer catches up. The serving layer merges batch views and speed views at query time, so a dashboard or API can answer with recent data blended on top of the durable historical picture. The main criticism is that you must write and maintain the same business logic twice, once in a batch framework like Spark and once in a stream framework like Flink, which is why many teams now prefer a simplified, stream-only Kappa architecture where that duplication makes sense.
- Combines the accuracy of full batch recomputation with the low latency of stream processing
- Tolerant of bugs — reprocessing the immutable batch log corrects any speed-layer errors
- Serving layer merges both views so clients see one coherent, near-real-time answer
- Batch layer remains the durable source of truth even if streaming infrastructure fails
AI Mentor Explanation
Lambda architecture is like a scoring team keeping both an official end-of-day scorebook, painstakingly recalculated from every ball bowled, and a live scoreboard operator who updates runs after each delivery for spectators. The live operator’s numbers are fast but can be slightly off if a decision is later overturned, while the official scorebook is always eventually correct because it is recomputed from the full ball-by-ball record. Fans watching the match see the live board blended with the last confirmed totals, giving them a near-real-time yet trustworthy picture. That two-track approach, fast-but-provisional plus slow-but-authoritative, merged for the viewer, is exactly what Lambda architecture does with streaming and batch layers.
Step-by-Step Explanation
Step 1
Ingest into an immutable log
All raw events land in an append-only store (e.g., Kafka or HDFS) that both layers read from independently.
Step 2
Batch layer recomputes full views
A batch job (e.g., Spark) periodically reprocesses the entire dataset into precomputed batch views, slow but always correct.
Step 3
Speed layer processes recent events
A stream processor (e.g., Flink) computes low-latency incremental views only for data not yet covered by the latest batch run.
Step 4
Serving layer merges both views
Queries hit a layer that combines the batch view with the speed view’s delta, giving a near-real-time yet eventually accurate answer.
What Interviewer Expects
- Clearly separates batch layer, speed layer, and serving layer and their roles
- Explains why the speed layer exists (compensating for batch latency) and is provisional
- Names the core trade-off: duplicated logic across batch and stream frameworks
- Can compare it to Kappa architecture and articulate when each is appropriate
Common Mistakes
- Describing Lambda architecture as just “using Kafka and Spark” without the three-layer structure
- Forgetting that the batch layer is the durable source of truth and the speed layer is a temporary correction
- Not mentioning the operational cost of maintaining two codebases for the same logic
- Confusing Lambda architecture with the unrelated AWS Lambda serverless compute service
Best Answer (HR Friendly)
“Lambda architecture is a way of processing data where you run two pipelines side by side: a slow, thorough one that reprocesses all the historical data to get fully accurate results, and a fast one that only looks at the newest data to give quick answers. A final layer combines both, so users get results that are timely and still end up completely accurate.”
Code Example
def get_metric(key, batch_views, speed_views, batch_cutoff_ts):
"""Merge the durable batch view with the speed layer delta
for events that arrived after the last batch run completed."""
batch_value = batch_views.get(key, 0)
# speed layer only covers events newer than the last batch cutoff
speed_delta = sum(
event.value
for event in speed_views.get(key, [])
if event.timestamp > batch_cutoff_ts
)
return batch_value + speed_delta
def run_batch_layer(all_events, cutoff_ts):
"""Recompute the full, accurate view from the immutable event log."""
view = {}
for event in all_events:
if event.timestamp <= cutoff_ts:
view[event.key] = view.get(event.key, 0) + event.value
return viewFollow-up Questions
- What problem does the speed layer solve that the batch layer alone cannot?
- How does Kappa architecture simplify Lambda architecture, and what does it give up?
- What happens to correctness if the speed layer and batch layer disagree at the serving layer?
- Why is the raw event log usually append-only and immutable in this design?
MCQ Practice
1. What is the primary role of the batch layer in Lambda architecture?
The batch layer reprocesses the entire immutable dataset to produce comprehensive, always-correct precomputed views.
2. Why does Lambda architecture include a speed layer at all?
The speed layer covers the gap between “now” and the last completed batch run, trading some accuracy for low latency.
3. What is the most commonly cited drawback of Lambda architecture?
Implementing and keeping equivalent logic in sync across separate batch and stream codebases is Lambda architecture’s main operational cost.
Flash Cards
What are Lambda architecture’s three layers? — Batch layer, speed layer, and serving layer, which merges the other two.
Why is the batch layer considered the source of truth? — Because it reprocesses the complete immutable dataset, so it is always eventually correct.
What is the speed layer’s trade-off? — Low latency on recent data in exchange for provisional, sometimes approximate results.
Main criticism of Lambda architecture? — Duplicated business logic maintained across separate batch and stream processing frameworks.