Capstone — Submit Cloud DE Portfolio Project

This is the final submission for Course 5: Cloud Data Engineering. You have built a production-patterned cloud-native lakehouse pipeline integrating every module: cloud infrastructure patterns with Terraform, IAM, and VPC networking (Module 1); AWS data engineering with S3 lifecycle, Glue ETL, Redshift, EMR, and Kinesis (Module 2); GCP data engineering with BigQuery, Dataflow, Pub/Sub, Dataproc, and BQML (Module 3); Azure data engineering with ADLS Gen2, ADF, Synapse, Databricks, and Event Hubs (Module 4); and cloud-native patterns with Iceberg, medallion architecture, serverless Lambda orchestration, multi-cloud strategies, and FinOps discipline (Module 5).

The submission requires the full integration test from Lesson 34 to pass with all nine assertions verified. Before submitting, ensure you have completed all three exercise lessons in order, run the full integration test and confirmed all nine assertions pass, reviewed the 22-item submission checklist and verified each item, and optionally extended the pipeline with the bonus challenge items. A submission where all required assertions pass represents mastery of production cloud data engineering at the level expected of a Senior Data Engineer.

Analogy🏏Cricket

🏏 Think of it like cricket: This exercise is the IPL official statistics team building their daily automated processing pipeline — the complete workflow that takes raw ball-by-ball records from every ground and produces the certified statistics published on the official website by midnight. Stage 1 is the data catalogue check: verify that the incoming scorecards match the expected format before any processing begins. Stage 2 is the statistics calculation: joins with match metadata, derivation of over-level stats, economy rate computation. Stage 3 is the official record update: load the new statistics into the production database using the certified upsert protocol — delete the old version of today's record and insert the freshly computed one — so no match ever has two records in the official database.

Submission Checklist

python

# Course 5 Capstone Submission Checklist
# Verify each item before submitting

# [REQUIRED] Module 1 — Cloud Infrastructure
# 1.  Terraform CIDR non-overlap validation passes for all subnets
# 2.  All resources have the six required FinOps tags
# 3.  S3 bucket has versioning, AES256 encryption, block_public, prevent_destroy
# 4.  No security group allows database ports (5432, 5439, 3306) from 0.0.0.0/0

# [REQUIRED] Module 2 — Bronze Ingestion
# 5.  Lambda trigger activates for scorecard CSV files and skips other file types
# 6.  Lambda idempotency: two triggers for the same file result in one pipeline run
# 7.  Bronze has 252 rows (240 good + 2 duplicates + 10 nulls) — no filtering
# 8.  Bronze includes _ingested_at and _source_file metadata columns
# 9.  Bronze immutability: second write to same path raises error
# 10. Iceberg Bronze snapshot: operation=append, added_records=252

# [REQUIRED] Module 3 — Silver Transformation
# 11. Silver < Bronze (deduplication and null rejection applied)
# 12. Silver quality gate: no null bowlers, all runs in [0,6]
# 13. Silver includes phase and is_boundary derived columns
# 14. Iceberg Silver snapshot: operation=overwrite
# 15. Silver idempotency: row count and runs sum identical on re-run
# 16. Schema evolution: wicket_phase column added, non-wicket rows default to None

# [REQUIRED] Module 4 — Gold Analytics
# 17. Gold delivery conservation: gold.deliveries.sum() == silver.row_count
# 18. All economy rates are non-negative and non-null
# 19. Three Athena SQL analytics queries execute successfully
# 20. Iceberg Gold snapshot: operation=overwrite

# [REQUIRED] Module 5 — FinOps and Integration
# 21. All five pipeline resources have all six required FinOps tags
# 22. Spot cost saving >= 60% vs On-Demand
# 23. Total Iceberg snapshots: 4 (1 append + 3 overwrites)
# 24. Full integration test: all 9 assertions pass simultaneously

# [RECOMMENDED] Code Quality
# 25. All pipeline functions have docstrings
# 26. Lambda idempotency uses a persistent store (not in-memory dict)
# 27. Deduplication uses deterministic ORDER BY in ROW_NUMBER window
# 28. All assertions include descriptive error messages

print("Checklist reviewed. Submit pipeline files to the SkillVeris capstone portal.")

What You Have Built: Over the six modules of Course 5, you have built a comprehensive cloud data engineering skill set covering the full modern stack: cloud service models, networking, IAM, and IaC with Terraform (Module 1); the complete AWS data engineering toolkit — S3, Glue, Redshift, EMR, and Kinesis (Module 2); the complete GCP toolkit — BigQuery, Dataflow, Pub/Sub, Dataproc, and BQML (Module 3); the complete Azure toolkit — ADLS Gen2, ADF, Synapse, Databricks, and Event Hubs (Module 4); and the cloud-native patterns that make modern lakehouses production-ready — Iceberg, medallion architecture, serverless orchestration, multi-cloud strategy, and FinOps (Module 5). You are equipped to design and build cloud data platforms on any major cloud provider.