ORC file format
Apache ORC — an Apache Software Foundation project
ORC (Optimized Row Columnar) is a columnar file format for storing structured data efficiently on distributed storage, originally developed to speed up Apache Hive queries over large datasets in the Hadoop ecosystem. It stores data by…
Definition
ORC (Optimized Row Columnar) is a columnar file format for storing structured data efficiently on distributed storage, originally developed to speed up Apache Hive queries over large datasets in the Hadoop ecosystem. It stores data by column rather than by row, which lets analytical queries read only the columns they need and enables aggressive compression and indexing that reduce both storage size and query time.
Overview
Traditional row-oriented storage formats write each record's fields together, which works well for transactional workloads that read or write whole rows but is wasteful for analytical queries that typically scan a handful of columns across billions of rows, since the query still has to read every field of every row even when only a few columns matter. ORC was created within the Hadoop ecosystem specifically to address this mismatch for Hive workloads, replacing earlier, less efficient formats with one designed from the ground up around columnar layout and built-in statistics. Mechanically, an ORC file organizes data into stripes, each holding a batch of rows, and within each stripe the values for each column are stored contiguously and independently compressed, since values within a single column tend to be far more similar to each other than values across a whole row, which compresses more effectively. Each stripe also carries lightweight index data, including minimum and maximum values, counts, and bloom filters for each column chunk, that lets a query engine skip entire stripes without decompressing them if it can determine from the statistics alone that no matching rows are present. A file-level footer records the schema and stripe locations, letting readers seek directly to the relevant data rather than scanning the whole file. ORC's closest peer is Apache Parquet, which solves the same columnar-storage problem and emerged from the same era but grew out of the broader Hadoop and Spark ecosystem rather than Hive specifically; the two formats are conceptually similar and mostly interchangeable for many analytical use cases, with the practical choice often coming down to which processing engine a team already standardizes on, since ORC has historically had deeper, more native integration with Hive, while Parquet has broader adoption across Spark, Presto and Trino, and cloud data warehouses. Both are meaningfully different from Avro, a row-oriented format optimized for schema evolution and streaming rather than analytical scan performance. In practice, ORC is used as the storage format underlying Hive tables in data lakes, particularly in organizations with a long-standing Hadoop investment, and it is also supported as a read and write format by Spark, Presto, Trino, and various cloud data-warehouse and lakehouse products, even where Hive itself isn't the primary query engine. Choosing ORC typically comes down to compatibility with an existing Hive-centric pipeline or a specific need for its built-in predicate pushdown and indexing behavior. The trade-off with any columnar format, including ORC, is that it performs poorly for workloads that need to read or write full rows frequently, such as transactional updates, since writing even a single row touches every column's separate stream. ORC files are also less human-inspectable than text formats like CSV or JSON, requiring specific tooling to read, and choosing between ORC and Parquet mostly reduces to ecosystem fit rather than any large inherent performance gap between the two in typical analytical workloads.
Key Concepts
- Columnar storage layout organized into row-batch stripes
- Per-column compression exploiting similarity within columns
- Built-in min/max and bloom-filter indexes for stripe skipping
- Native integration with Apache Hive query planning
- Supported by Spark, Presto, Trino, and cloud data warehouses
- File-level footer enabling efficient schema and location lookups
- Predicate pushdown reducing unnecessary data scanning
- Open-source, Apache Software Foundation-governed project
Use Cases
Frequently Asked Questions
From the Blog
Python File I/O: Reading and Writing Files
Almost every real Python program reads or writes files — logs, configs, CSVs, JSON, reports. This guide covers text files, CSV, JSON, binary files, and the modern pathlib approach, with best practices for safe file handling.
Read More ProgrammingPython File Handling: Read and Write Files
Learn to read and write files in Python using open() and the with statement. Covers text and binary modes, reading line by line, appending, and safe file handling.
Read More ProgrammingUnderstanding Python String Formatting (f-strings)
F-strings are the fastest, most readable way to format strings in Python. Learn how to embed variables, format numbers, align text, and debug with f-string syntax.
Read More ProgrammingUnderstanding Python Modules and Packages
A Python module is a single .py file and a package is a folder of modules. Learn how imports, __init__.py, and namespaces organize larger Python projects.
Read More