DAG

Lazy Evaluation and the DAG

How Spark defers execution of transformations into a logical plan and only runs work when an action forces it, using a DAG of stages for both optimization and…

Building an ETL DAG

A hands-on walkthrough of designing and implementing a complete Extract-Transform-Load pipeline as an Airflow DAG, including failure handling and local testing.

DAG Definition Basics

Learn how Apache Airflow DAGs are structured as Python files, parsed by the scheduler, and how to define one correctly with dag_id, schedule, and default_args.

DAG Versioning and CI/CD

How Airflow tracks DAG structure changes over time via DAG Versioning, and how to build a GitOps-style CI/CD pipeline that tests and deploys DAGs safely.

Dynamic DAG Generation

Techniques for programmatically generating multiple Airflow DAGs from configuration files or metadata instead of hand-writing each one, and the performance tra…

Testing DAGs

How to validate Airflow DAGs at three layers — import integrity, structural correctness, and task logic — using pytest, DagBag, and dag.test().

The Scheduler

How Airflow's scheduler parses DAGs, evaluates dependencies, and triggers task instances to keep pipelines running on time.

Your First DAG

A hands-on walkthrough of writing, testing, and triggering your first Apache Airflow DAG, covering DAG anatomy and task dependencies.

What is Topological Sort?

Topological sort produces a linear ordering of the nodes in a directed acyclic graph such that for every directed edge from node A to node B, A appears before…

What are Strongly Connected Components?

A strongly connected component (SCC) is a maximal set of vertices in a directed graph where every vertex can reach every other vertex in the set via a directed…

What is Dynamic Programming on Graphs?

Dynamic programming on graphs solves problems like shortest paths, longest paths, and counting paths by defining a DP state per node (or per node-and-extra-dim…