Apache Parquet

Definition

Apache Parquet is an open-source columnar storage file format for big data, designed to store large datasets efficiently on disk and enable fast analytical queries by reading only the columns a query actually needs. It is widely used across the Hadoop and Spark ecosystems as the standard on-disk format for analytical tables, and it supports schema evolution, compression, and encoding optimizations tuned for column-oriented access.

Overview

Analytical queries typically touch only a handful of columns out of a wide table — computing an average or filtering on one field — but row-oriented file formats force a query engine to read entire rows just to get at those few columns, wasting I/O at scale. Apache Parquet was designed to solve this by physically storing data column by column rather than row by row, so a query only needs to read the specific columns it references, dramatically cutting the data scanned for typical analytical workloads. Mechanically, a Parquet file organizes data into row groups, each of which is further split into column chunks, with each column chunk stored contiguously and compressed using an encoding suited to that column's data type and value distribution, such as dictionary encoding for low-cardinality strings or run-length encoding for repeated values. Metadata describing the schema, row group boundaries, and column statistics (like min/max values) is embedded in the file's footer, which query engines read first so they can skip entire row groups that can't possibly match a query's filters. Within the big-data storage landscape, Parquet's main peer is Apache ORC, another columnar format with a similar goal, originally developed for the Hive ecosystem; both offer comparable columnar benefits, with differences mainly in default compression choices, metadata layout, and which engines historically optimized for which format. Parquet has broader adoption across Spark, Presto/Trino, and cloud data warehouses, while ORC has stronger roots in the Hive/Hadoop world, though most modern engines support both. In practice, Parquet is the default storage format for data lakes built on cloud object storage, used to persist datasets that will be queried repeatedly by engines like Apache Spark, Trino, or cloud data warehouses such as BigQuery and Redshift Spectrum. Its column-pruning and predicate-pushdown-friendly design make it well suited to write-once, read-many analytical workloads rather than transactional systems with frequent row-level updates. Parquet's trade-offs mirror those of any columnar format: it is a poor fit for workloads that need frequent single-row updates or deletes, since modifying a value typically means rewriting the affected row group or file rather than an in-place edit, and it is less efficient than row-oriented formats for queries that need to read entire rows repeatedly. Table formats like Apache Iceberg, Delta Lake, and Apache Hudi were built on top of Parquet specifically to add update, delete, and transactional semantics that the raw file format itself does not provide.

Key Concepts

• Columnar layout that lets queries read only the columns they need
• Row groups split into per-column chunks with type-specific encoding
• Embedded footer metadata including schema and column statistics
• Support for predicate pushdown by skipping non-matching row groups
• Schema evolution for adding or reordering columns over time
• Broad support across Spark, Trino, Presto, and cloud data warehouses

Use Cases

Frequently Asked Questions

Related in Database