What Are Airflow Variables?
Airflow Variables are key-value pairs stored in the metadata database, managed via the UI (Admin > Variables), the CLI (airflow variables set), or programmatically, and used to store runtime configuration that shouldn't be hardcoded into DAG source — things like a target S3 bucket name, a feature flag, or an environment-specific threshold. They're accessed in code with Variable.get('key') or, for JSON-valued variables, Variable.get('key', deserialize_json=True), though each call to Variable.get() inside top-level DAG code triggers a database query every time the DAG file is parsed, which can slow down scheduler performance if overused.
Cricket analogy: It's like a team's settings sheet pinned in the dressing room — today's designated powerplay overs or the ground's boundary rope distance — updated per match without rewriting the team's playbook.
Jinja Templating in Task Parameters
Many operator parameters (like bash_command in BashOperator or sql in SQL-based operators) are templated fields, meaning Airflow runs them through the Jinja2 templating engine at task execution time, substituting built-in template variables like {{ ds }} (execution date as YYYY-MM-DD), {{ ds_nodash }}, {{ data_interval_start }}, and {{ params.some_key }} before the operator actually runs. This is what lets a single DAG definition process a different date's data on every run without any code change — the templated string is resolved fresh for each specific DAG run's logical date.
Cricket analogy: It's like a stadium's scoreboard template showing '{{ team_name }} vs {{ opponent }}' that automatically fills in the actual team names for whichever match is currently being played.
from airflow.decorators import dag, task
from airflow.operators.bash import BashOperator
from airflow.models import Variable
from datetime import datetime
@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def templated_pipeline():
target_bucket = Variable.get("analytics_export_bucket", default_var="default-bucket")
extract = BashOperator(
task_id="extract",
# {{ ds }} resolves to the DAG run's logical date, e.g. 2026-07-10
bash_command=(
"aws s3 cp s3://raw-events/{{ ds_nodash }}/events.json "
f"s3://{target_bucket}/staged/{{{{ ds }}}}/events.json"
),
)
@task
def log_run(**context):
print(f"Processed logical date {context['ds']}")
extract >> log_run()
templated_pipeline()Params and Custom Jinja Filters
DAGs and tasks can accept dag-level or task-level params — user-supplied values validated optionally against a JSON schema — which are then accessible in templates as {{ params.key }}, letting a DAG be triggered manually with different runtime inputs (e.g., a specific customer_id to backfill) without editing code. Airflow also supports registering custom Jinja filters via a DAG's user_defined_filters or a plugin, so teams can add project-specific template helpers like {{ ds | to_fiscal_quarter }} that aren't part of Airflow's built-in macro set.
Cricket analogy: It's like a broadcaster's graphics template accepting a manually entered 'player of the match' name for that specific game, layered on top of the auto-filled score data.
Common built-in Jinja macros beyond {{ ds }} include {{ data_interval_start }}, {{ data_interval_end }}, {{ dag_run.conf }} (for manually triggered runs with custom config), and macros like {{ macros.ds_add(ds, 7) }} for date arithmetic — all available inside any templated field without extra imports.
Avoid calling Variable.get() at the top level of a DAG file outside of a task or template context. Because DAG files are re-parsed by the scheduler on a short interval, every top-level Variable.get() call adds a database round-trip on every parse cycle across every worker, which can meaningfully degrade scheduler performance at scale. Prefer accessing Variables inside task callables, or via Jinja templating with {{ var.value.my_key }}.
- Variables store runtime key-value configuration in the metadata database, accessed via Variable.get().
- Templated operator fields are rendered through Jinja2 at task execution time, not at DAG parse time.
- {{ ds }}, {{ ds_nodash }}, {{ data_interval_start }} are common built-in macros for the logical run date.
- Params let a DAG accept user-supplied runtime inputs, accessible via {{ params.key }}.
- Custom Jinja filters can be registered via user_defined_filters or a plugin.
- Avoid calling Variable.get() at DAG top level to prevent excessive database queries during parsing.
- Use {{ var.value.key }} in templates instead, which is resolved lazily at render time.
Practice what you learned
1. When are templated operator fields like bash_command actually rendered?
2. What does the {{ ds }} template variable represent?
3. Why should Variable.get() generally be avoided at the top level of a DAG file?
4. How do DAG-level or task-level params make runtime input accessible in templates?
Was this page helpful?
You May Also Like
XComs
Learn how Airflow tasks exchange small pieces of data at runtime using XComs, and when to avoid them in favor of external storage.
Branching with BranchPythonOperator
Learn how to build conditional paths in a DAG using BranchPythonOperator, and how Airflow's trigger rules interact with skipped tasks.
Hooks and Connections
Understand how Airflow Connections store credentials for external systems and how Hooks provide a Python interface to interact with them.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics