What Are Connections?
A Connection in Airflow is a stored record — identified by a conn_id — that holds everything needed to reach an external system: host, port, login, password/secret, schema, and an extra JSON field for provider-specific options like a role ARN or a warehouse name. Connections can be created via the Airflow UI (Admin > Connections), via environment variables (AIRFLOW_CONN_<CONN_ID> in URI form), or via a configured secrets backend such as AWS Secrets Manager or HashiCorp Vault, which keeps credentials out of the metadata database entirely.
Cricket analogy: It's like a team's contact card for a stadium groundskeeper — one record with the ground's address, gate code, and the specific person to call, so any player doesn't need to memorize it themselves.
What Are Hooks?
A Hook is a Python class that wraps the low-level client library for a specific system (Postgres, S3, Snowflake, HTTP APIs, etc.) and knows how to read a Connection's conn_id to authenticate and open a session. Hooks expose high-level, reusable methods — for example, PostgresHook.get_pandas_df(sql), S3Hook.upload_file(), or HttpHook.run() — so DAG authors don't need to hand-roll boto3 or psycopg2 calls inside every operator; most provider-built operators (like S3ToRedshiftOperator) actually use a Hook internally to do the real work.
Cricket analogy: It's like a translator who takes the coach's instructions and speaks fluent groundstaff-language to get the pitch prepared correctly — the coach doesn't need to know maintenance jargon directly.
from airflow.decorators import task
from airflow.providers.postgres.hooks.postgres import PostgresHook
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
@task
def export_orders_to_s3():
pg_hook = PostgresHook(postgres_conn_id="warehouse_pg")
df = pg_hook.get_pandas_df("SELECT * FROM orders WHERE status = 'shipped'")
csv_path = "/tmp/shipped_orders.csv"
df.to_csv(csv_path, index=False)
s3_hook = S3Hook(aws_conn_id="aws_default")
s3_hook.load_file(
filename=csv_path,
key="exports/shipped_orders.csv",
bucket_name="analytics-exports",
replace=True,
)Hooks vs. Operators
Operators define what a task does at the DAG level and are what you actually place in a DAG's task graph; Hooks are the lower-level connectivity layer that operators call into to talk to external systems. Many provider packages ship both — for example, PostgresOperator (for running SQL as a task) internally uses PostgresHook — so as a DAG author you reach for a Hook directly (usually inside a @task-decorated Python function) when you need custom logic that no existing operator covers, and reach for a pre-built operator when your use case matches an existing one exactly.
Cricket analogy: It's like the difference between a pre-set fielding plan (an operator — a ready-made play for a known situation) and a fielder's individual skill of throwing accurately (a hook — a reusable capability used inside any plan).
Managing Connections Securely
Storing passwords in plaintext inside the Airflow UI's metadata database is acceptable for local development but discouraged in production; instead, teams typically configure a secrets backend (e.g., AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault) via the [secrets] section in airflow.cfg, so that Hooks transparently fetch credentials from the external vault at task runtime instead of the database. Environment-variable-based connections (AIRFLOW_CONN_MY_CONN_ID=postgres://user:pass@host:5432/db) are also common in containerized deployments since they can be injected securely by an orchestration layer like Kubernetes secrets without ever touching the Airflow metadata DB.
Cricket analogy: It's like a team keeping the dressing-room safe combination with the team manager (a vault) rather than writing it on a whiteboard everyone can see.
Connection URIs follow the pattern <conn_type>://<login>:<password>@<host>:<port>/<schema>?param1=value1. When set as an environment variable, prefix the conn_id with AIRFLOW_CONN_ and uppercase it, e.g. AIRFLOW_CONN_WAREHOUSE_PG for conn_id 'warehouse_pg'.
Never hardcode credentials directly inside a DAG file or a custom operator's source code, even temporarily for testing. DAG files are often stored in version control and synced across multiple workers, so a hardcoded secret can leak far beyond its intended scope. Always reference credentials through a conn_id and let a Hook resolve them at runtime.
- A Connection (conn_id) stores everything needed to reach an external system: host, login, password, schema, and extra JSON options.
- Connections can be defined via the UI, environment variables (AIRFLOW_CONN_*), or a secrets backend like Vault or Secrets Manager.
- A Hook is a Python class that wraps a system's client library and authenticates using a Connection's conn_id.
- Hooks expose reusable high-level methods (get_pandas_df, upload_file, run) so DAG authors avoid hand-rolling low-level client code.
- Operators define what a task does in the DAG graph; many built-in operators use a Hook internally to do the actual work.
- Use a Hook directly inside a @task function for custom logic that no existing operator covers.
- Never hardcode credentials in DAG code — always resolve them through a conn_id at runtime.
Practice what you learned
1. What does an Airflow Connection primarily store?
2. What is the role of a Hook in Airflow?
3. How would you set a Connection via an environment variable for conn_id 'my_api'?
4. Why is it discouraged to hardcode credentials directly in a DAG file?
Was this page helpful?
You May Also Like
XComs
Learn how Airflow tasks exchange small pieces of data at runtime using XComs, and when to avoid them in favor of external storage.
Sensors
Learn how Airflow Sensors wait for external conditions to become true before downstream tasks proceed, and how to avoid worker-slot starvation.
Variables and Templating
Learn how Airflow Variables provide runtime configuration and how Jinja templating lets task parameters be computed dynamically at execution time.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics