AWS Glue — Crawlers, ETL Jobs and Data Catalog

AWS Glue is a fully managed serverless ETL service that eliminates the need to provision and manage Spark infrastructure. It consists of three integrated components: the Data Catalog (a metadata store for schemas and table definitions), Crawlers (schema discovery agents that scan data sources and populate the Catalog), and ETL Jobs (Apache Spark or Python Shell scripts executed in a managed serverless environment). Together these provide the metadata-driven ETL pipeline that forms the transformation backbone for most AWS-native data lake architectures.

The AWS Glue Data Catalog is a centralised metadata repository compatible with the Apache Hive Metastore. Registering a table in the Catalog makes it immediately queryable from Athena, EMR, Redshift Spectrum, and Glue ETL jobs without additional configuration. A delivery records Parquet table registered in the Catalog is queryable from Athena for ad-hoc analysis, from Redshift Spectrum for SQL joins with warehouse tables, and from Spark on EMR for programmatic transformations — all reading the same S3 data through the same schema.

Analogy🏏Cricket

🏏 Think of it like cricket: This exercise is the IPL official statistics team building their daily automated processing pipeline — the complete workflow that takes raw ball-by-ball records from every ground and produces the certified statistics published on the official website by midnight. Stage 1 is the data catalogue check: verify that the incoming scorecards match the expected format before any processing begins. Stage 2 is the statistics calculation: joins with match metadata, derivation of over-level stats, economy rate computation. Stage 3 is the official record update: load the new statistics into the production database using the certified upsert protocol — delete the old version of today's record and insert the freshly computed one — so no match ever has two records in the official database.