Declaring Dependencies
Dependencies in Airflow are declared using the bitshift operators >> (set downstream) and << (set upstream), or explicitly with .set_upstream() / .set_downstream() methods. Writing extract >> transform >> load tells Airflow that transform cannot start until extract finishes successfully, and load cannot start until transform finishes. This is purely about execution order; it does not automatically pass data between tasks unless you also use XCom or the TaskFlow API, so a task can depend on another without ever touching its output.
Cricket analogy: The >> operator is like the strict batting order on the team sheet: the No. 3 batter cannot walk in until the opener is dismissed, an order of play that says nothing about whether runs were actually shared between them.
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.models.baseoperator import chain, cross_downstream
import pendulum
with DAG(
dag_id="dependency_examples",
schedule="@daily",
start_date=pendulum.datetime(2026, 1, 1),
catchup=False,
) as dag:
extract = PythonOperator(task_id="extract", python_callable=lambda: None)
validate = PythonOperator(task_id="validate", python_callable=lambda: None)
transform = PythonOperator(task_id="transform", python_callable=lambda: None)
load = PythonOperator(task_id="load", python_callable=lambda: None)
notify = PythonOperator(task_id="notify", python_callable=lambda: None)
# Simple linear chain
extract >> validate >> transform >> load >> notify
# Equivalent using the chain() helper for readability with many tasks
chain(extract, validate, transform, load, notify)Fan-out, Fan-in, and chain()
Real pipelines are rarely a single straight line. You can fan out from one task to several with extract >> [validate_schema, validate_row_count], and fan back in with [validate_schema, validate_row_count] >> transform, meaning transform waits for both validation tasks to finish. For long or complex sequences, airflow.models.baseoperator.chain() reads more cleanly than a long line of >>, and cross_downstream() lets you connect every task in one list to every task in another list in a single call, useful when several independent extracts must all complete before several independent loads can begin.
Cricket analogy: Fan-out is like a captain deploying multiple fielders to cover different boundary zones simultaneously off one delivery, and fan-in is like the umpire needing signals from both square-leg and third umpire to confirm before raising the decision.
Trigger Rules
By default, every task uses trigger_rule='all_success', meaning all directly upstream tasks must succeed before it runs. Other trigger rules change this: all_failed runs only if every upstream task failed (useful for cleanup-on-failure logic), one_failed runs as soon as any upstream task fails, none_failed_min_one_success runs if no upstream failed and at least one succeeded (tolerant of skips), and all_done runs regardless of upstream success or failure, commonly used for notification or teardown tasks that must always execute.
Cricket analogy: all_success is like a partnership needing both batters at the crease to survive the over for the team total to advance normally, while all_done is like the innings-closing formalities (handshakes, scorecard signing) happening no matter how the innings ended.
Branching and Short-Circuiting
BranchPythonOperator lets a task return the task_id (or list of ids) of the specific downstream path to follow, causing Airflow to mark all other sibling branches as skipped rather than running them; this is how you build if/else logic into a DAG, like choosing between a 'send_alert' path and a 'proceed_normally' path. ShortCircuitOperator is simpler: it returns a boolean, and if False, all downstream tasks are skipped entirely (or optionally marked as failed with ignore_downstream_trigger_rules=False), useful for stopping a pipeline early when an upstream check determines there's nothing to process.
Cricket analogy: BranchPythonOperator is like a captain choosing between an aggressive fielding setup or a defensive one based on the pitch report, only one full plan gets deployed and the other stays entirely unused for that innings.
Dependencies are defined at DAG-parse time, forming a static graph structure that the scheduler evaluates for each DAG run; runtime branching (via BranchPythonOperator) changes which paths actually execute, but it does not change the graph's structure itself.
Because a DAG must remain acyclic, you cannot create a dependency where task B (directly or transitively through other tasks) depends back on task A after A already depends on B. Airflow will raise a cycle-detection error at parse time if you attempt this, so loops must instead be modeled with separate scheduled DAG runs or dynamic task mapping, not a literal cycle.
- Bitshift operators >> and << declare execution order between tasks; they do not automatically pass data.
- Fan-out connects one task to many, fan-in connects many tasks to one; chain() and cross_downstream() simplify complex wiring.
- The default trigger_rule is all_success; other rules like all_failed, one_failed, and all_done control behavior on partial failure.
- all_done is commonly used for notification or cleanup tasks that must run regardless of upstream outcome.
- BranchPythonOperator picks one downstream path and skips the sibling branches, implementing if/else logic in a DAG.
- ShortCircuitOperator skips all downstream tasks when its callable returns False, useful for early pipeline termination.
- The dependency graph is fixed at DAG-parse time and must remain acyclic; Airflow detects and rejects cycles.
Practice what you learned
1. What does writing `extract >> transform` actually guarantee in an Airflow DAG?
2. Which trigger_rule would you use for a notification task that must run whether the upstream pipeline succeeded or failed?
3. How does BranchPythonOperator implement conditional logic in a DAG?
4. Why can't you create a genuine cycle between tasks A and B in an Airflow DAG?
Was this page helpful?
You May Also Like
DAG Definition Basics
Learn how Apache Airflow DAGs are structured as Python files, parsed by the scheduler, and how to define one correctly with dag_id, schedule, and default_args.
Operators
Understand what Airflow operators are, the major built-in categories, and how to configure or subclass them to build reliable tasks.
The TaskFlow API
Learn how Airflow's @dag and @task decorators let you write DAGs as plain Python functions, with automatic XCom passing and dynamic task mapping.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics