Delta Lake — ACID Transactions on Data Lake — Big Data & Distributed Computing

Data lakes built on raw Parquet or ORC files are eventually consistent — a reader may see a mix of new and old data if a writer is mid-operation when the read begins. This lack of transactional guarantees causes the data swamp problem: accumulated stale files, failed partial writes, and undocumented schema changes that make the data unreliable for analytical queries. Delta Lake solves this by adding a transaction log to Parquet-based storage, providing ACID guarantees while retaining the scale and openness of a data lake.

Delta Lake is now the default table format in the Databricks Lakehouse Platform and is increasingly adopted in open-source Spark deployments as the standard format for any table that receives incremental writes. The open-source Delta Lake project is maintained by the Linux Foundation, ensuring vendor neutrality, and the Delta format is natively readable by Spark, Trino, Flink, and Hive, making it a practical universal standard for data lake tables that require reliability without sacrificing ecosystem compatibility.

Analogy🏏Cricket

🏏 Think of it like cricket: Imagine the DRS review system deployed across three independent video-review centres in Mumbai, Chennai, and London, each holding a copy of the ball-tracking data. A CAP partition is a network outage that cuts communication between them. A CP system says: if the centres cannot synchronise, no review decision is issued — no player walks until the system is restored. Consistency is guaranteed; availability is sacrificed. An AP system says: each centre issues its own decision based on its local data, even if that means Mumbai says out and London says not out — reviews continue but different centres may give different verdicts. Partition tolerance is non-negotiable because the network always has the possibility of failing; the choice is whether umpires wait for consensus or proceed with local data.