Exercise 3 and final project — polyglot architecture and integration

What You'll Build

This final exercise has two parts. Exercise 3 asks you to design the polyglot persistence architecture for CricketVerse — deciding which database technology serves each component and documenting the decision with explicit trade-offs. The Final Project then integrates everything: the schema from Exercise 1, the queries from Exercise 2, and the architecture from Exercise 3 into a single, documented platform deliverable with a data flow design and a complete architecture decision record. This is the capstone of the capstone — the artifact that demonstrates your end-to-end data engineering competence.

The polyglot architecture must satisfy all four non-functional requirements: sub-50ms leaderboard reads at peak, 100+ deliveries-per-second ingestion, sub-1s career statistics, and archiving of data older than two years. Each non-functional requirement maps to a specific database technology and pattern from the course. Your task is to assemble these into a coherent architecture where each component plays to its database's strength, and where the data flows between components are clearly defined.

Analogy🏏Cricket

🏏 Think of it like cricket: This exercise runs like a proper tournament bureau's production week, and the step order is the point. Step 1 is the pitch inspection before play: you verify the ground truth — no orphaned scorecard lines, no impossible totals — because an analysis built on a corrupt book is a match played on a dangerous pitch: everything after it is invalidated. Step 2 is the specialist coaches' reports: batting summaries with rankings and form lines, each an independent, checkable piece of work using window functions over the validated data. Step 3 is the selectors' composite: batting and bowling folded into one all-rounder view via conditional aggregation — the wide wall chart built from the long book. Step 4 is the match referee's reconciliation: the chart's totals must re-add to the book's totals exactly, or something was dropped or double-counted on the way. Validate, analyse, combine, reconcile — every production pipeline plays in that order.

Exercise 3 — Polyglot Architecture Design

Design the database technology assignment for each CricketVerse component. For each non-functional requirement, identify which database technology and pattern best satisfies it, then document why. Attempt this mapping yourself before reading the reference — for each NFR, ask: what is the dominant characteristic (read latency, write throughput, query complexity, data lifecycle), and which database's primary strength matches it?

The reference architecture makes a deliberate and important choice: it uses primarily two databases (PostgreSQL and Redis) rather than the full four-database polyglot stack. This reflects the Module 5 principle that polyglot persistence should be driven by measured requirements, not anticipated scale. At CricketVerse's stated scale (100 deliveries/second), PostgreSQL handles ingestion comfortably with partitioning, and its JSONB capability handles variable player attributes without needing MongoDB. Cassandra and MongoDB are documented as migration paths for when scale grows — not adopted preemptively. This restraint is itself a senior-engineering decision worth recognising.

Analogy🏏Cricket

🏏 Think of it like cricket: Choosing two databases instead of four is a selection lesson every young captain learns: you don't pick a specialist for every conceivable condition — you pick the smallest squad that wins the actual series. Four databases is four specialists who each need their own coach, physio and travel arrangements: every additional store brings operational load — backups, monitoring, failure modes, expertise — whether or not it earns its place. The reference architecture asks each NFR the selector's hard question: does this requirement genuinely demand a specialist, or can a player already in the XI cover it? PostgreSQL with the right partitioning covers ingestion and archive scans; Redis covers the sub-50ms leaderboard; the exotic picks stay home. Polyglot persistence is a tool, not a badge — the mature design is the one carrying the fewest players who all demonstrably earn their spot, with the reasoning written down like a selection rationale.

Final Project — Integration and Data Flow

Assemble the complete platform: integrate the schema (Exercise 1), the analytical queries (Exercise 2), and the architecture (Exercise 3) into a single deliverable with a documented data flow. The data flow describes how data moves through the platform from live match ingestion through to dashboard serving — the operational story that ties all the components together.

The integrated data flow tells the operational story that ties all the components together. During a live match, each delivery follows a write path that fans out to three destinations: PostgreSQL for durable storage, Redis sorted sets for real-time leaderboard rank, and Redis streams for the broadcaster feed. The read path during the match serves the leaderboard from Redis (meeting the sub-50ms requirement) while the detailed scorecard comes from PostgreSQL. Post-match batch jobs refresh the materialised views and re-seed Redis to correct any drift. Annually, old partitions are detached for archiving. Every component plays to its database's strength, and the data flow makes the integration explicit.

Analogy🏏Cricket

🏏 Think of it like cricket: The integrated write path is what happens in the seconds after a wicket falls, traced end to end. The ball is bowled (the delivery event arrives): first it must enter the official book — the durable PostgreSQL write, the record of truth that survives any dispute. Then the consequences ripple outward on different clocks: the stadium scoreboard and the app leaderboard update within moments (the Redis sorted-set bump), while the deep archive that analysts will query at season's end absorbs the row into the right partition without any spectator noticing. One event, several destinations, each serving a different reader at a different speed — and the acceptance-criteria run at the end is the umpires' post-match reconciliation: every functional requirement demonstrated against its query, every NFR measured against its number, so 'the platform works' is a verified scorecard rather than a captain's optimistic claim.

The acceptance criteria verification section closes the loop on the entire capstone: it tests each functional requirement query for correctness and each non-functional requirement against its target. This is the discipline emphasised throughout the capstone — verifying work against explicit criteria rather than assuming correctness. The final deliverable checklist confirms that every piece of the platform, from the 3NF schema through the documented data flow, has been delivered and verified.

Completing this capstone means you have exercised the full data engineering lifecycle for a relational and polyglot data platform: requirements analysis, logical schema design, physical optimisation, analytical query development, and architecture decision-making with explicit trade-off documentation. These are the core competencies of a working data engineer — and the CricketVerse platform you have built is a portfolio artifact that demonstrates them concretely. Course 1 of the Data Engineer Learning Path is now complete.