Apache Airflow
By the Apache Software Foundation
Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data and workflow pipelines, defined in Python as Directed Acyclic Graphs (DAGs) of tasks.
Definition
Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data and workflow pipelines, defined in Python as Directed Acyclic Graphs (DAGs) of tasks.
Overview
Airflow was originally created at Airbnb in 2014 to manage the company's increasingly complex data pipelines, and was later donated to the Apache Software Foundation, where it graduated to a top-level project in 2019. Its core idea is to let engineers define a pipeline as Python code — a DAG describing tasks and their dependencies — rather than as a tangle of cron jobs, giving pipelines version control, testability, and a visual web UI for monitoring runs. Each Airflow deployment consists of a scheduler that triggers DAG runs, a set of workers (executed via Celery, Kubernetes, or other executors) that actually run the tasks, and a web server providing visibility into successes, failures, retries, and historical run times. Airflow 2.x added the TaskFlow API for writing pipelines with plain Python functions and decorators, plus dynamic task mapping for generating tasks at runtime. Airflow integrates with a huge ecosystem of external systems through 'provider' packages, connecting to cloud platforms, data warehouses like Snowflake and BigQuery, transformation tools like dbt, and container platforms like Docker and Kubernetes for isolated task execution. It is one of the most widely adopted orchestrators in modern data engineering, and is covered in detail in SkillVeris's Apache Airflow & Orchestration course.
Key Features
- Python-native DAG authoring for pipelines as code
- Scheduler with cron-like and event/sensor-based triggering
- Large ecosystem of provider packages for cloud, data, and DevOps tools
- Web UI for monitoring runs, retries, logs, and historical performance
- TaskFlow API for writing pipelines with plain Python functions
- Dynamic task mapping for generating tasks at runtime
- Pluggable executors, including Celery and Kubernetes
- Built-in retry logic, alerting, and SLA tracking