Data Lake
A data lake is a centralized storage repository that holds raw data in its native format — structured, semi-structured, or unstructured — without requiring a predefined schema before the data is written.
Definition
A data lake is a centralized storage repository that holds raw data in its native format — structured, semi-structured, or unstructured — without requiring a predefined schema before the data is written.
Overview
Where a data warehouse enforces a defined schema before data is loaded ("schema-on-write"), a data lake takes the opposite approach: data is ingested as-is — CSV files, JSON logs, images, video, sensor data, database exports — and structure is applied later, at query time, an approach known as "schema-on-read." This makes data lakes well suited to storing large volumes of diverse data cheaply, especially data whose eventual analytical use isn't fully known at ingestion time, such as raw clickstream logs or machine-generated sensor data. Data lakes are typically built on low-cost, highly scalable object storage — Amazon S3, Azure Data Lake Storage, or Google Cloud Storage — and queried using engines like Apache Spark, Presto, or DuckDB rather than a traditional SQL database engine. Because there's no schema enforcement on write, data lakes are prone to becoming a "data swamp" over time without strong metadata management, cataloging, and governance discipline to track what data exists, its format, and its lineage. Data lakes commonly serve as a landing zone in ELT pipelines, where raw data is loaded first and transformed later, sometimes directly within the lake using Spark jobs, and sometimes by subsequently loading curated subsets into a downstream data warehouse for structured analytics. This two-tier pattern — cheap raw storage in a lake feeding a more expensive, structured warehouse — was a common architecture before the rise of the data lakehouse, which tries to collapse the two into a single system using formats like Delta Lake and Apache Iceberg that add transactional and schema guarantees directly on top of object storage. Data engineering teams building this kind of infrastructure typically pair a data lake with orchestration tools like Apache Airflow to manage ingestion and transformation jobs reliably at scale.
Key Concepts
- Stores raw data in native format — structured, semi-structured, or unstructured
- Schema-on-read: structure is applied at query time, not at ingestion
- Built on low-cost, scalable object storage (S3, ADLS, GCS)
- Queried with engines like Apache Spark, Presto, or DuckDB
- Well suited to data whose future analytical use isn't fully known upfront
- Requires strong metadata and governance to avoid becoming a disorganized "data swamp"
- Commonly serves as the landing zone in ELT-based data pipelines