Apache Oozie

Definition

Apache Oozie is a workflow scheduler system for coordinating and running dependent jobs on a Hadoop cluster, letting engineers define multi-step pipelines of MapReduce, Spark, Hive, Pig, and shell actions as a directed acyclic graph in XML. It handles time- and data-availability-based triggering through its coordinator engine, tracks job status, and retries or fails workflows according to configured rules, functioning as the native scheduling layer for Hadoop before general-purpose orchestrators became common.

Overview

Hadoop pipelines are rarely a single job; a typical pipeline ingests data, transforms it through several dependent steps, and loads results somewhere else, and each step needs to run only after its predecessor succeeds. Apache Oozie was built to express and execute exactly that kind of dependency graph natively within the Hadoop ecosystem, rather than relying on cron scripts or ad hoc shell orchestration outside the cluster. A workflow in Oozie is defined as an XML document describing a directed acyclic graph of action nodes, where each node can run a MapReduce job, a Spark application, a Hive or Pig script, a Java program, or a shell command, connected by control-flow nodes that handle branching and joining. The Oozie server, running as its own service, parses this definition and submits each action to the cluster in the correct order, tracking success and failure and re-running or halting the workflow according to configured error handling. Beyond single workflow runs, Oozie's coordinator engine can trigger a workflow on a recurring schedule or when specified input data becomes available in HDFS, and its bundle feature groups multiple coordinators together for managing related pipelines as a unit. Oozie predates and differs from more general-purpose, code-first orchestrators like Apache Airflow, which express dependencies as Python code rather than XML and are not tied specifically to the Hadoop ecosystem. Where Airflow treats Hadoop as one of many possible systems to orchestrate, Oozie is built assuming a Hadoop cluster as its execution environment, which made it a natural default in Hadoop distributions but also ties its usability closely to that ecosystem's continued relevance. In practice, teams running on-premises or cloud Hadoop clusters use Oozie to chain together ingestion, transformation, and reporting jobs, triggering a nightly ETL pipeline once new source data lands, or coordinating a sequence of Hive scripts that build a data warehouse's daily tables. Oozie's web console and REST API let operators monitor running workflows, inspect failures at the specific action that failed, and manually rerun from a failed step rather than restarting an entire pipeline. The most common criticism of Oozie is its XML-based workflow definition, which is verbose and harder to iterate on than the code-based DAGs offered by newer orchestrators, and its UI and feature set have not kept pace with tools built for broader, non-Hadoop-specific orchestration. Many teams migrating off legacy Hadoop stacks toward cloud-native or general-purpose data platforms replace Oozie with Airflow or a managed workflow service rather than continuing to maintain XML-defined pipelines.

Key Features

• XML-defined directed acyclic graph workflows of Hadoop actions
• Coordinator engine for time- and data-availability-based scheduling
• Bundles for grouping related coordinators into a single unit
• Native support for MapReduce, Spark, Hive, Pig, and shell actions
• Web console and REST API for monitoring and rerunning workflows
• Retry and failure handling configurable per workflow action

Use Cases

Alternatives

Frequently Asked Questions

Related in Database