Apache Hudi
Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data lake table format and framework that enables efficient upserts, deletes, and incremental data processing on top of large analytical datasets.
Definition
Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data lake table format and framework that enables efficient upserts, deletes, and incremental data processing on top of large analytical datasets.
Overview
Hudi was originally developed at Uber to address a problem that plain append-only data lakes handle poorly: efficiently applying updates and deletes to massive datasets, and processing only the records that changed since the last pipeline run rather than reprocessing everything. It introduces a table format with indexing and a timeline of commits, allowing engines to perform record-level upserts and deletes on data stored in formats like Parquet. A key differentiator for Hudi is its strong focus on incremental processing: consumers can query only the data that changed since a given point in time, which is valuable for building efficient streaming and near-real-time ETL pipelines rather than repeatedly scanning entire tables. Hudi supports two main table types, Copy-on-Write and Merge-on-Read, that trade off write amplification against query latency depending on the workload. Like its counterparts Apache Iceberg and Delta Lake, Hudi integrates with engines such as Apache Spark, Presto, and Trino, and it is one of the three dominant open table formats used to build a transactional data lakehouse on cloud object storage.
Key Features
- Record-level upserts and deletes on data lake storage
- Incremental queries that read only changed records
- Copy-on-Write and Merge-on-Read table storage types
- Timeline-based metadata for transactional consistency
- Built-in indexing for efficient record lookups
- Integration with Spark, Presto, Trino, and other engines
- Designed for near-real-time streaming ETL pipelines