Parquet

Definition

Apache Parquet is an open-source columnar storage file format designed for efficient storage and retrieval of large analytical datasets. Instead of storing rows contiguously, it groups data by column, which lets analytical engines read only the columns a query needs, apply targeted compression per column type, and skip irrelevant data blocks entirely, making it a common storage layer for big-data processing frameworks.

Overview

Parquet was created to address a mismatch between how data was commonly generated and how it was actually queried in large-scale analytics. Traditional row-oriented formats like CSV or Avro store every field of a record together, which is efficient for writing full records but wasteful for analytical queries that typically scan a handful of columns across millions of rows. Parquet inverts this layout so that each column's values are stored contiguously, letting an engine skip unused columns entirely. Internally, a Parquet file is organized into row groups, each of which is further split into column chunks, and each column chunk is broken into pages that carry encoded and optionally compressed values along with statistics like minimum and maximum values. Because values in a single column tend to share a data type and often similar values, Parquet applies encoding schemes such as dictionary encoding and run-length encoding before general-purpose compression, which typically yields much smaller files than row-based alternatives. The embedded per-column statistics also let query engines perform predicate pushdown, skipping entire row groups that cannot match a filter without reading their data. Parquet is frequently compared to other big-data serialization formats: Avro remains row-oriented and is preferred for write-heavy streaming pipelines and schema evolution, while ORC offers a similar columnar layout with different internal indexing choices, mostly favored in the Hive ecosystem. Parquet has become the de facto standard for data lakes because of its broad support across query engines rather than because it strictly outperforms every alternative in every scenario. In practice, Parquet is the storage format underlying most modern data lakes and lakehouse architectures, used by engines like Apache Spark, Presto/Trino, Apache Hive, and cloud data warehouses such as BigQuery and Snowflake for external tables. Data engineers write intermediate and final outputs of ETL pipelines as Parquet files precisely because analytical queries downstream run faster and cheaper against them than against row-oriented equivalents. The format's columnar layout is a poor fit for workloads that need to read or write entire records frequently, such as transactional systems, since reconstructing a full row means touching every column chunk. Parquet files are also immutable once written, so updating a single record requires rewriting the affected files, which is why it is paired with table formats like Apache Iceberg or Delta Lake when mutability and versioning are required on top of it. Small files also erode Parquet's efficiency advantage, since its columnar layout and metadata overhead pay off best when row groups are large enough to amortize that overhead across many records.

Key Features

• Stores data by column rather than by row for analytical efficiency
• Applies per-column encoding and compression tailored to each data type
• Embeds min/max statistics per row group to enable predicate pushdown
• Supports nested and complex data structures via a shredding scheme
• Works natively with Spark, Hive, Presto, Trino, and major cloud warehouses
• Reduces storage footprint significantly compared to row-oriented formats
• Enables reading only the columns required by a given query
• Is a self-describing format that embeds its own schema in file metadata

Use Cases

Alternatives

Frequently Asked Questions

Related in Programming