100% Free Forever
AI-Powered Learning
Industry Expert Content
Certificates & Badges
Learn At Your Own Pace
Cloud Data Engineering
35 minintermediate

Amazon Redshift — Architecture and COPY Command

Amazon Redshift is AWS's fully managed cloud data warehouse — a massively parallel processing (MPP) columnar database designed for analytical workloads across terabytes to petabytes of data. Unlike transactional databases optimised for row-level operations, Redshift is optimised for high-throughput aggregation queries that read large fractions of many columns across billions of rows. A query that takes minutes in PostgreSQL on a 500GB table often completes in seconds on Redshift because MPP distributes the work across many nodes and columnar storage reads only the columns needed.

Redshift's architecture separates the leader node from compute nodes. The leader node parses SQL queries, builds execution plans, and coordinates query distribution. Compute nodes execute the plan in parallel, each processing a slice of the data. When a query scans the deliveries fact table, the leader distributes the scan to all compute nodes simultaneously — each node scans its local slice and returns partial results, which the leader aggregates into the final result. The number of compute nodes and instance type determine the cluster's compute capacity and total storage.

Analogy🏏Cricket
🏏 Think of it like cricket: This exercise is the IPL official statistics team building their daily automated processing pipeline — the complete workflow that takes raw ball-by-ball records from every ground and produces the certified statistics published on the official website by midnight. Stage 1 is the data catalogue check: verify that the incoming scorecards match the expected format before any processing begins. Stage 2 is the statistics calculation: joins with match metadata, derivation of over-level stats, economy rate computation. Stage 3 is the official record update: load the new statistics into the production database using the certified upsert protocol — delete the old version of today's record and insert the freshly computed one — so no match ever has two records in the official database.
Lesson 9 of 35
0% complete