Luigi
By Spotify
Luigi is an open-source Python framework for building and running batch data processing pipelines as a graph of dependent tasks. Originally developed at Spotify, it lets developers define each pipeline step as a Python class specifying its…
Definition
Luigi is an open-source Python framework for building and running batch data processing pipelines as a graph of dependent tasks. Originally developed at Spotify, it lets developers define each pipeline step as a Python class specifying its inputs, outputs, and dependencies, and Luigi's scheduler resolves the resulting dependency graph, running tasks in the correct order and skipping ones whose output already exists.
Overview
Luigi was built to address the recurring problem of chaining many interdependent batch jobs — extracting data, transforming it, loading it into a warehouse, generating reports — where a plain sequence of scripts run by cron makes it hard to know what has already run successfully, retry failed steps, or avoid redundant work when a downstream step fails and the whole chain needs re-running. Mechanically, a Luigi pipeline is expressed as a set of Task classes, each defining a requires method describing upstream dependencies, a run method containing the actual processing logic, and an output method describing where the task's result is written. Luigi treats a task as complete if its declared output already exists, which lets a rerun of the pipeline skip already-finished steps and resume only from the point of failure. A central scheduler process, along with a visualizer, tracks task state across a cluster of worker processes, giving visibility into which tasks have run, failed, or are pending. Luigi also supports parameterized tasks, where the same task class can produce different output targets depending on arguments such as a processing date, which is how a single pipeline definition can be reused to backfill many days or partitions of historical data. Luigi predates and is often compared to Apache Airflow, which emerged with a broadly similar goal of representing pipelines as directed acyclic graphs of tasks. Airflow has since become more feature-rich, with a scheduler that runs on defined time intervals, a larger ecosystem of pre-built integrations, and a more actively developed web UI, while Luigi remains comparatively lighter-weight and closer to a plain Python library, appealing to teams that want dependency-based pipelining without adopting a heavier orchestration platform. In practice, Luigi is used for recurring batch ETL pipelines, machine learning feature and training pipelines, and any multi-step data processing job where steps depend on each other's output and simple idempotent re-runs are valuable, particularly in Python-centric data engineering teams. The main limitation is that Luigi's scheduler is not natively designed around fixed time-based scheduling the way Airflow is, so teams typically still pair it with cron or another external trigger, and it has a smaller plugin ecosystem and less active development momentum than Airflow, which has become the more common default choice for new data pipeline projects. Teams already invested in Luigi's simpler model sometimes stay with it deliberately rather than take on Airflow's larger operational footprint for pipelines that do not need it.
Key Features
- Task classes defining dependencies, execution logic, and output targets in Python
- Automatic skipping of tasks whose declared output already exists
- Central scheduler and visualizer tracking task state across workers
- Dependency graph resolution ensuring tasks run in correct order
- Idempotent design supporting resumable pipeline re-runs after failure
- Lightweight, library-like footprint compared to full orchestration platforms
- Originally built and open-sourced by Spotify for internal data pipelines