Amazon S3 is the foundational object storage layer for virtually every AWS data engineering architecture. Its role extends far beyond simple file storage: S3 is the data lake substrate that separates compute from storage, the landing zone for raw ingested data, the staging area between pipeline stages, the destination for processed analytical data, and the source for direct query engines like Athena and Redshift Spectrum. Understanding S3's storage classes, lifecycle policies, and event notification system transforms it from a passive file store into an active pipeline component.
S3 provides eleven storage classes optimised for different access frequency, retrieval latency, and cost profiles. The key insight is that storage cost and retrieval cost are inversely related — cheaper storage classes have higher per-GB retrieval fees and minimum storage duration charges. The correct storage class for any S3 object depends on how frequently it will be accessed and what retrieval latency is acceptable. S3 Intelligent-Tiering automatically monitors access patterns and moves objects between tiers when access frequency changes, making it the lowest-management choice for data with unpredictable access patterns.