Reading and Writing Parquet, ORC and Delta — Big Data & Distributed Computing

The choice of file format is one of the highest-leverage decisions in a distributed data pipeline. Parquet, ORC, and Delta Lake are the three dominant columnar formats for Spark workloads, and each optimises a different set of trade-offs. Parquet is the universal, framework-agnostic standard for analytical storage. ORC is the Hive-ecosystem incumbent with slightly better performance for Hive-managed tables. Delta Lake extends Parquet with ACID transactions, schema enforcement, and time travel, making it the format of choice for data lake tables that require write reliability and audit trails.

All three formats are columnar — data stored column-by-column — enabling Spark to read only the columns needed by a query without scanning the full row. Parquet and ORC also support row-group-level statistics (min, max, null count per column) that allow Spark to skip entire row groups whose statistics prove they contain no matching rows for a filter. This row-group skipping is the mechanism behind massive performance gains when data is sorted or partitioned to co-locate relevant rows in the same row groups.

Analogy🏏Cricket

🏏 Think of it like cricket: Imagine the DRS review system deployed across three independent video-review centres in Mumbai, Chennai, and London, each holding a copy of the ball-tracking data. A CAP partition is a network outage that cuts communication between them. A CP system says: if the centres cannot synchronise, no review decision is issued — no player walks until the system is restored. Consistency is guaranteed; availability is sacrificed. An AP system says: each centre issues its own decision based on its local data, even if that means Mumbai says out and London says not out — reviews continue but different centres may give different verdicts. Partition tolerance is non-negotiable because the network always has the possibility of failing; the choice is whether umpires wait for consensus or proceed with local data.