What is Kappa Architecture and How Does It Differ from Lambda?
Understand Kappa architecture, its single stream-pipeline design, replay-based reprocessing, and how it compares to Lambda architecture.
Expected Interview Answer
Kappa architecture simplifies Lambda architecture by processing all data — historical and real-time — through a single stream-processing pipeline, treating the append-only log as the only source of truth and reprocessing by replaying that log through the same code rather than maintaining separate batch and speed layers.
In Kappa architecture, every event lands in a durable, replayable log (typically Kafka with a long or infinite retention), and one stream-processing job (e.g., Flink or Kafka Streams) reads it to produce output views. There is no separate batch layer with its own codebase; instead, if the business logic changes or a bug is fixed, you spin up a new instance of the stream job, replay the log from the beginning, and once it catches up, atomically switch consumers over to the new output and retire the old one. This eliminates the “write every rule twice” problem of Lambda architecture at the cost of requiring your stream processor to comfortably handle both real-time and full historical replay volumes. Kappa fits well when a single, well-tested streaming engine can express all needed logic and the operational simplicity outweighs Lambda’s batch-layer accuracy guarantees.
- Removes the need to write and maintain the same logic twice in batch and stream frameworks
- Single unified codebase and pipeline simplifies operations, testing, and debugging
- Reprocessing is just replaying the log through a new version of the same job, not a separate system
- Well suited to modern streaming engines that can handle both real-time and bulk replay workloads
AI Mentor Explanation
Kappa architecture is like a franchise that keeps a single, complete ball-by-ball commentary feed as the only record of every match, rather than maintaining a separate live commentator and a separate end-of-day statistician with different note-taking systems. If the scoring rules change mid-season, the franchise simply replays the entire commentary feed from ball one through the new rules to regenerate every statistic, instead of updating two disconnected systems. This keeps only one pipeline to maintain, at the cost of that single feed needing to handle both live commentary speed and full-season replay volume. That one-log, one-pipeline, replay-to-fix approach is exactly what Kappa architecture does with streaming data.
Step-by-Step Explanation
Step 1
Persist everything to one replayable log
All events, historical and new, flow into a single durable, long-retention log such as Kafka.
Step 2
One stream job produces every view
A single stream-processing pipeline reads the log and produces all output views, with no separate batch codebase.
Step 3
Deploy a new job version to fix logic
When logic changes, a new instance of the stream job is deployed alongside the old one, not patched in place.
Step 4
Replay, catch up, then cut over
The new job replays the log from the start; once it catches up to real time, consumers switch to its output and the old job is retired.
What Interviewer Expects
- Explains Kappa as “stream-only” processing with a single replayable log as the source of truth
- Articulates the core trade versus Lambda: no logic duplication, but the stream engine must handle full replay volume
- Describes the reprocessing mechanism as deploying a new job and replaying, not patching live state
- Can state when Kappa is a better fit than Lambda (single strong stream engine, simpler ops) versus not (very heavy batch-only analytics)
Common Mistakes
- Saying Kappa architecture has no batch processing at all, rather than “no separate batch codebase”
- Not explaining how corrections/reprocessing actually happen (replay + cutover)
- Confusing Kappa architecture with the Greek-letter-named AWS Lambda service
- Ignoring that the log must have long enough retention to support full historical replay
Best Answer (HR Friendly)
“Kappa architecture is a simpler alternative to Lambda architecture where everything, live and historical, flows through one streaming pipeline instead of two separate systems. If you need to fix or improve how the data is processed, you just replay all the past events through an updated version of the same pipeline, rather than maintaining two different codebases that both need to agree.”
Code Example
async function deployNewPipelineVersion(newJob, eventLog) {
// 1. Start the new job reading from the very beginning of the log
await newJob.start({ fromOffset: "earliest" })
// 2. Wait until it has caught up to near real time
while (!(await newJob.isCaughtUp({ lagThresholdMs: 2000 }))) {
await sleep(5000)
}
// 3. Atomically point consumers at the new job's output topic
await router.switchOutput({
from: "views-v1",
to: "views-v2",
})
// 4. Retire the old job now that nothing reads its output
await oldJob.stop()
}Follow-up Questions
- What retention requirements does the underlying log need to support Kappa architecture?
- How do you handle a schema change in the events when replaying the entire log?
- When would you still prefer Lambda architecture over Kappa architecture?
- How does Kappa architecture handle the resource cost of a full historical replay during a cutover?
MCQ Practice
1. What is the defining characteristic of Kappa architecture compared to Lambda architecture?
Kappa architecture treats the replayable log as the sole source of truth and uses one stream pipeline for everything, unlike Lambda’s dual batch/speed layers.
2. How does Kappa architecture handle a change in business logic?
Corrections are made by running a new pipeline version against a replay of the full log, then switching consumers to its output.
3. What capability must the underlying log support for Kappa architecture to work well?
Since reprocessing means replaying from the start, the log needs retention long enough to hold the full history needed for recomputation.
Flash Cards
What is Kappa architecture in one line? — A stream-only data architecture that treats a single replayable log as the source of truth for all processing.
How does Kappa avoid Lambda’s logic-duplication problem? — By using one stream pipeline for both real-time and historical processing instead of separate batch and speed codebases.
How do you fix a bug in Kappa architecture? — Deploy a new job version, replay the log from the start, then cut consumers over once it catches up.
Main requirement for Kappa to work? — A durable, long-retention log capable of a full historical replay through the stream engine.