What Is an Operator?
An operator is a Python class that defines a single, reusable unit of work; when you instantiate it inside a DAG with a task_id, that instance becomes a task in the graph. Airflow ships with dozens of built-in operators, such as PythonOperator (runs a Python callable), BashOperator (runs a shell command), and provider-specific ones like S3CreateBucketOperator or SnowflakeOperator, each encapsulating the details of talking to a particular system so DAG authors don't have to write that boilerplate themselves.
Cricket analogy: An operator is like a specialist role card in a fantasy cricket app: a 'death bowler' card encapsulates specific stats and behavior, and you slot it into your XI (the DAG) to perform one job well.
Categories of Built-in Operators
Operators broadly fall into three families. Action operators do work directly, like PythonOperator or BashOperator. Transfer operators move data between two systems, such as S3ToRedshiftOperator or GCSToBigQueryOperator, and typically need connections configured for both the source and destination. Sensors are a special subclass of operator (BaseSensorOperator) that wait for a condition to become true, like FileSensor waiting for a file to land, or ExternalTaskSensor waiting for a task in another DAG to finish, and they support poke mode (blocking a worker slot) or reschedule mode (releasing the worker slot between checks).
Cricket analogy: Action operators are like a batter facing a delivery right now; transfer operators are like a mid-pitch run between wickets moving something (the strike) from one end to the other; sensors are like a fielder at the boundary rope watching and waiting for the ball to arrive before acting.
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
from airflow.sensors.filesystem import FileSensor
import pendulum
def load_to_warehouse(**context):
ds = context["ds"]
print(f"Loading partition for {ds} into warehouse")
with DAG(
dag_id="operator_examples",
schedule="@daily",
start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
catchup=False,
) as dag:
wait_for_file = FileSensor(
task_id="wait_for_extract",
filepath="/data/incoming/sales_{{ ds_nodash }}.csv",
poke_interval=60,
timeout=60 * 30,
mode="reschedule",
)
validate = BashOperator(
task_id="validate_csv",
bash_command="wc -l /data/incoming/sales_{{ ds_nodash }}.csv",
)
load = PythonOperator(
task_id="load_to_warehouse",
python_callable=load_to_warehouse,
)
wait_for_file >> validate >> loadConfiguring Operator Behavior
Every operator inherits common parameters from BaseOperator: task_id (unique within the DAG), retries, retry_delay, execution_timeout, on_failure_callback, and trigger_rule. Operator-specific parameters, like bash_command for BashOperator or python_callable and op_kwargs for PythonOperator, control what the task actually does. Many parameters accept Jinja templating, such as '{{ ds }}' in a Bash command string, letting the same operator instance behave differently for each scheduled run without any extra Python logic.
Cricket analogy: task_id is like a player's unique jersey number on the team sheet, while retries mirrors a batter getting a review via DRS, a limited number of extra chances before the decision is final.
Building Custom Operators
When no built-in operator fits, you subclass BaseOperator and implement an execute(self, context) method containing the task logic; this is common for wrapping a proprietary internal API or bundling repeated multi-step logic (e.g., 'call this API, validate the response, then write to S3') into one reusable class. Custom operators typically pair with a custom Hook, a thin wrapper around a connection (Connection object) and its client library, so the operator's execute method stays focused on orchestration logic while the hook handles authentication and API calls.
Cricket analogy: Writing a custom operator is like a franchise designing its own bespoke fielding drill not in any standard coaching manual, built once by the analytics team and reused every training session thereafter.
Operators are designed to be idempotent and atomic wherever possible; a well-built operator's execute() method should be safe to re-run for the same context without corrupting state, since retries and manual re-runs are a normal part of Airflow's operational model.
Avoid using PythonOperator to run CPU- or memory-heavy processing directly inside the Airflow worker process. Airflow is an orchestrator, not a compute engine; heavy transformations should be delegated to something like Spark, dbt, or a Kubernetes job via KubernetesPodOperator, with the PythonOperator only triggering and monitoring that external work.
- An operator is a Python class encapsulating one unit of work; instantiating it with a task_id creates a task.
- Built-in operators fall into action, transfer, and sensor categories, each solving a different orchestration need.
- Sensors wait for a condition and support poke mode (blocks a worker slot) or reschedule mode (frees it between checks).
- Common BaseOperator parameters include task_id, retries, retry_delay, execution_timeout, and trigger_rule.
- Jinja templating on operator parameters lets one operator instance behave differently for each scheduled run.
- Custom operators subclass BaseOperator and implement execute(), often paired with a custom Hook for connection handling.
- Airflow workers should orchestrate heavy compute externally rather than run it directly inside a PythonOperator.
Practice what you learned
1. Which of the following best describes a sensor operator in Airflow?
2. What is the key difference between a sensor's poke mode and reschedule mode?
3. What is the recommended approach for heavy data processing in an Airflow pipeline?
4. When would you write a custom operator by subclassing BaseOperator?
Was this page helpful?
You May Also Like
DAG Definition Basics
Learn how Apache Airflow DAGs are structured as Python files, parsed by the scheduler, and how to define one correctly with dag_id, schedule, and default_args.
Task Dependencies
Learn how to wire tasks together with bitshift operators and the chain() helper, and how trigger rules control execution when upstream tasks fail or skip.
The TaskFlow API
Learn how Airflow's @dag and @task decorators let you write DAGs as plain Python functions, with automatic XCom passing and dynamic task mapping.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics