Hoodie (Apache Hudi)
Apache table format for incremental data lake updates
Apache Hudi (Hadoop Upserts Deletes and Incrementals), sometimes referred to informally by its early codename Hoodie, is an open-source data lake table format that adds transactional upsert, delete, and incremental processing capabilities…
Definition
Apache Hudi (Hadoop Upserts Deletes and Incrementals), sometimes referred to informally by its early codename Hoodie, is an open-source data lake table format that adds transactional upsert, delete, and incremental processing capabilities to large datasets stored on distributed file systems or object storage. It was created to solve a gap in traditional data lakes, which historically supported efficient append-only writes but struggled with the update-heavy, mutable workloads common in real-world data pipelines.
Overview
Before formats like Hudi existed, data lakes built on systems like Hadoop's HDFS or cloud object storage were optimized for writing new, immutable files and reading them back, but updating or deleting individual records within already-written files required rewriting entire partitions, which was slow and operationally painful at scale. Hudi addresses this by managing datasets as a structured table with an internal timeline of commits, tracking file-level changes so that upserts (insert-or-update operations) and deletes can be applied efficiently without rewriting the entire dataset each time. Mechanically, Hudi organizes data into base files and, in its Merge-on-Read table type, accompanying delta log files that record incremental changes; reads can either merge these logs with base files at query time for fresher data, or Hudi can periodically compact logs into new base files for read-optimized access, trading off between write latency and read performance depending on which table type and configuration is chosen. Hudi also maintains a commit timeline, giving each table version-like snapshot and incremental query capabilities, so downstream jobs can pull only the records that changed since a previous point rather than reprocessing an entire dataset. Hudi belongs to a category of table formats — alongside Apache Iceberg and Delta Lake — that emerged to bring transactional, mutable-table semantics to data lakes, an area traditional Hadoop-ecosystem formats like plain Parquet or ORC files did not address on their own. The three formats overlap significantly in goals but differ in implementation details: Hudi has historically emphasized incremental processing and fast upserts as its primary design focus, while Iceberg emphasizes format-level abstraction and broad engine compatibility, and Delta Lake grew out of and remains closely tied to the Databricks/Spark ecosystem, though all three have converged toward supporting similar core capabilities over time. In practice, Hudi is used in data engineering pipelines that need to continuously ingest and update large volumes of changing data — for example, syncing a database's change stream into a data lake, maintaining slowly changing dimension tables, or supporting near-real-time analytics on data that is frequently updated rather than purely append-only. It integrates with big data processing engines like Apache Spark, Apache Flink, and Presto/Trino, letting existing query and processing infrastructure work against Hudi tables largely as it would against other file-based data lake tables. Hudi's flexibility around table types and compaction strategies is also a source of operational complexity: choosing between Copy-on-Write and Merge-on-Read table types, and tuning compaction and clustering settings, requires understanding the trade-offs involved, and teams new to Hudi can find its configuration surface larger than simpler append-only formats. As with its peer formats, Hudi also depends on the broader data lake and processing engine ecosystem being correctly integrated, and mismatched engine versions or configurations can lead to compatibility issues that require careful operational attention.
Key Features
- Adds transactional upsert and delete support to data lake tables
- Maintains a commit timeline enabling incremental and time-travel queries
- Offers Copy-on-Write and Merge-on-Read table type options
- Uses delta log files to record incremental changes efficiently
- Integrates with Apache Spark, Flink, and Presto/Trino processing engines
- Competes with Apache Iceberg and Delta Lake as a lakehouse table format
- Emphasizes fast incremental ingestion and update-heavy workloads
Use Cases
Alternatives
Frequently Asked Questions
From the Blog
Introduction to Apache Spark for Beginners
Apache Spark is a fast, distributed engine for processing huge datasets across many machines. Learn what it is, how it works, and how to run your first job.
Read More AI & TechnologyWhat Is PySpark? Python's Gateway to Big Data
PySpark is the Python API for Apache Spark, letting developers process massive datasets across many machines using familiar Python syntax. This guide covers what PySpark does, its core components, and when to reach for it.
Read More