Project Brief — Automated Multi-Stage Data Pipeline

The capstone project builds a production-patterned, automated multi-stage data pipeline for the IPL analytics platform that integrates all five modules of Course 4. The pipeline is orchestrated by an Airflow DAG with sensors, branching, and dynamic task mapping; validated by Great Expectations quality gates at source and after transformation; transformed by dbt models across three layers with incremental materialisation; monitored with tiered alerting and OpenLineage lineage emission; and accompanied by full dbt documentation. Every orchestration concept from the course appears in one coherent, end-to-end system.

The pipeline follows the ELT pattern: raw data is extracted from two sources — the IPL API for match metadata and a PostgreSQL operational database for delivery records — and loaded into a staging area with minimal transformation. The GE quality gate validates the staged data before dbt runs the transformation layer. dbt builds staging, intermediate, and mart models, runs all tests, and generates updated documentation. The pipeline completes with an idempotency test and a lineage event summary verifying that all four stages have emitted correctly structured lineage metadata.

Analogy🏏Cricket

🏏 Think of it like cricket: Migrating from Airflow to Prefect is like the same bowling coach shifting from traditional Test cricket notation to a modern T20 analytics dashboard — the underlying ball-by-ball data (the business logic) is exactly the same. What changes is how the data is recorded, displayed, and acted upon. The yorker that Bumrah bowls in over 20 is identical whether it is recorded in the old scorebook (Airflow DAG file) or the new analytics platform (Prefect flow). The migration is a transcription exercise, not a strategy change — and a wise coach verifies that the runs, wickets, and economies match exactly between the old and new system before decommissioning the scorebook. That verification step is the whole heart of the migration: because the yorker is unchanged, the only honest test is to run the same over through both systems and confirm the recorded runs, wickets and economies match to the last digit before the old scorebook is thrown away. Rushing to burn the scorebook the moment the shiny dashboard lights up is how teams lose a season of records to a silent transcription slip. The coach keeps both systems running in parallel for a while, reconciles their outputs ball by ball, and only when every figure agrees does he trust the new dashboard alone — a transcription is only complete when you have proven nothing was lost in the copying.

Project Specification

Architecture

The pipeline has five stages: (1) S3 sensor waits for the scorecard file, then extract tasks run in parallel via dynamic task mapping per match; (2) a GE checkpoint validates completeness, validity, and freshness — failures route to DLQ; (3) dbt runs source freshness, builds three model layers, and runs all tests; (4) quality results and SLA metrics publish to tiered alert channels; (5) OpenLineage START and COMPLETE events are emitted for every task with structured input and output dataset references.

Analogy🏏Cricket

🏏 Think of it like cricket: The capstone's five-stage architecture is a full match-day operation where each phase hands cleanly to the next, and none may begin before its predecessor is truly done. The S3 sensor is the third umpire waiting for the scorecard file to cross the line before play starts — nothing extracts until the ball is confirmed in. Then extraction fans out via dynamic task mapping, one task per match like a separate scorer assigned to each simultaneous fixture, all working in parallel. The GE checkpoint is the match referee inspecting every scorecard at the boundary gate, refusing entry to any that fail the playing conditions and branching the bad ones aside. The dbt transformation is the statistics department refining the cleared raw scores into polished season tables, and the lineage-and-alerting stage is the official archive recording exactly how every figure came to be, with the physio on the boundary ready to react if anything goes wrong. Five stages, one disciplined chain — each a gate the innings must pass before the next over can be bowled.

Deliverables

The complete submission must include: an Airflow-style DAG with sensor, dynamic task mapping, branching quality gate, and `ALL_DONE` cleanup task; a GE suite with at least six expectations covering all quality dimensions; three dbt models with schema.yml tests; a tiered alert router with two severity levels; OpenLineage events for at least two pipeline stages; an idempotency assertion confirming identical output when run twice; and a lineage summary asserting all emitted events have non-null job names with at least one input and one output dataset.

Analogy🏏Cricket

🏏 Think of it like cricket: The deliverables checklist is the match-day kit inspection where a team is only allowed to take the field once every mandatory item is present and correct. A DAG with a working sensor, dynamic task mapping, a branching quality gate and an `ALL_DONE` cleanup task is the equivalent of pads, gloves, helmet, and the twelfth man's drinks tray all accounted for — miss one and you simply aren't match-ready, however good the rest. The GE suite with its minimum count of expectations is the agreed set of playing conditions signed before the toss; the dbt models and tests are the certified statistics workings; the lineage and alerting are the official record-keeping and the physio on standby. The point of a fixed submission spec is exactly the point of a kit inspection: it removes judgement in the heat of the moment, so nobody walks out having forgotten a compulsory piece. Every required item is a gate, and the whole submission clears only when each gate is satisfied — a complete, inspectable, match-ready package rather than a talented but under-equipped side.

Scaffolding Provided: The three exercise lessons that follow build each major stage of the capstone incrementally. Lesson 32 builds the Airflow DAG with sensors, dynamic mapping, and branching. Lesson 33 adds the dbt transformation and test layer. Lesson 34 integrates monitoring, alerting, and lineage emission. The final project submission lesson asks you to run the complete integration test and submit your project. Complete each lesson in order and verify all assertions before proceeding to the next.