The capstone project builds a production-patterned streaming analytics pipeline for IPL stadium IoT sensor events — wearable performance monitors on players, pitch sensors measuring ball impact and spin, and crowd engagement sensors tracking sound levels and movement. These devices generate thousands of events per second during live play, requiring a pipeline that ingests from Kafka, processes with Spark Structured Streaming, stores analytical results in Delta Lake, and serves aggregates to a query layer. Every concept from Modules 1 through 5 is exercised in one coherent end-to-end system.
The pipeline follows the Kappa architecture: a single streaming layer handles both real-time analytics (live match dashboards) and historical batch analytics (season trend reports) by reading from the same Kafka topic with different consumer groups at different latencies. There is no separate batch layer — historical analysis is performed by replaying the Kafka topic from the beginning with `startingOffsets=earliest`. This simplifies the operational footprint to one pipeline codebase and one data format (Delta Lake) that serves both use cases.
Analogy🏏Cricket
🏏 Think of it like cricket: Imagine the DRS review system deployed across three independent video-review centres in Mumbai, Chennai, and London, each holding a copy of the ball-tracking data. A CAP partition is a network outage that cuts communication between them. A CP system says: if the centres cannot synchronise, no review decision is issued — no player walks until the system is restored. Consistency is guaranteed; availability is sacrificed. An AP system says: each centre issues its own decision based on its local data, even if that means Mumbai says out and London says not out — reviews continue but different centres may give different verdicts. Partition tolerance is non-negotiable because the network always has the possibility of failing; the choice is whether umpires wait for consensus or proceed with local data.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.
Project Specification
Architecture
The pipeline has four stages. Stage 1 — Ingest: a Python Kafka producer publishes IoT sensor events with Avro serialisation, partition-key routing by sensor type, and `acks=all` durability. Stage 2 — Stream Processing: Spark Structured Streaming applies watermarks, computes 1-minute tumbling window aggregates, and detects anomalies using a rolling baseline. Stage 3 — Delta Lake: results are written via idempotent MERGE. Stage 4 — Query: a batch job reads the Delta tables and produces a session-level player performance report.
Analogy🏏Cricket
🏏 Think of it like cricket: The four-stage pipeline — ingest, stream-process, store, query — is the chain that carries a delivery from the bowler's hand all the way to the published scorecard, each stage with one clear job. Ingest is the on-field scorer capturing every ball into the durable match log the instant it happens (the Kafka producer writing events). Stream processing is the analyst who continuously reads that log and computes rolling figures — run rates, partnership totals — as play unfolds (Structured Streaming). Storage is filing those computed aggregates into the permanent, correction-friendly record book (Delta Lake). Query is anyone later opening that book to answer questions (SQL over the tables). Just as a delivery must be captured before it can be analysed, analysed before its summary is filed, and filed before anyone can look it up, the stages form a strict pipeline where each depends on the clean output of the one before. And just as separating capture from analysis from filing lets you fix or rerun any one role without disturbing the others, the stage boundaries keep the system debuggable and independently scalable. One event, four hand-offs, a published result.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.
Deliverables and Evaluation
The complete submission must include: a Kafka producer with Avro schema and Schema Registry integration; a Structured Streaming pipeline with watermarks, tumbling windows, and `foreachBatch` Delta MERGE; a Delta table for raw events (append mode) and one for aggregated stats (MERGE mode); a `checkpointLocation` for every streaming query; an idempotency test that replays the pipeline and asserts zero Delta row count change; and a batch query job that reads both Delta tables and produces a session comparison report. The pipeline must pass all assertions in the provided integration test.
Analogy🏏Cricket
🏏 Think of it like cricket: The capstone deliverables are like the full checklist a match-data operation must hand in before its scoring system is certified for a live final. Just as the on-field capture crew must log every ball into a durable, agreed format, your submission needs a Kafka producer using an Avro schema with Schema Registry integration and acks=all durability. Just as the live analyst must compute rolling figures while tolerating a late-arriving corrected delivery and closing each over's window cleanly, you need a Structured Streaming pipeline with watermarks, tumbling windows, and a foreachBatch Delta MERGE. Just as the record book keeps both the raw ball log and the tidy summary tables, you supply a Delta table for raw events in append mode and one for aggregated stats via MERGE, each streaming query with its own checkpointLocation. And just as re-tallying a match from the tape must never double-count a run, the idempotency test replays the pipeline and asserts zero Delta row-count change, plus a batch query producing a session comparison report. The payoff is a pipeline passing every integration assertion — provably correct and replay-safe.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.
Scaffolding Provided: The three exercise lessons that follow build the pipeline incrementally — Kafka ingest in Lesson 32, Structured Streaming processing in Lesson 33, and Delta write plus batch query in Lesson 34. Each lesson builds on the previous one and includes end-of-step assertions. The final capstone submission lesson asks you to run the complete integration test and submit your project repository. Work through each lesson in order and verify all assertions before proceeding.