Orchestrating dbt Runs

Analogy categoryAnalogies

Orchestration is the layer that decides when dbt commands execute, in what order relative to upstream data loads and downstream consumers, and how failures are handled — dbt itself only resolves dependencies within its own DAG, it has no native awareness of whether Fivetran finished syncing or whether a Looker dashboard is waiting on fresh data. In production, a typical orchestrated run is a sequence: check source freshness, run dbt source freshness, trigger dbt run for all or selected models, run dbt test to validate the output, and only then signal downstream systems that new data is ready.

🏏

Cricket analogy: It's like a match day schedule that sequences the pitch inspection, toss, and innings in strict order — you don't start bowling before the toss decides who bats, just as dbt run shouldn't start before source freshness is confirmed.

Selectors and Partial Runs

Rather than rebuilding an entire project on every schedule, orchestration typically triggers scoped runs using dbt's node selection syntax: dbt run --select tag:hourly runs only models tagged for hourly refresh, dbt build --select state:modified+ (used with dbt Cloud's deferred state or a stored manifest.json) rebuilds only models that changed plus their downstream dependents, and dbt run --select finance.+ rebuilds a folder and everything downstream of it. This selective execution is what makes orchestrating dozens of pipelines with different SLAs practical — a marketing mart that needs hourly freshness and a finance mart that only needs daily refresh can run on independent schedules against the same project.

🏏

Cricket analogy: It's like a coach running targeted net sessions for just the fast bowlers before a pace-friendly pitch rather than putting the entire squad through identical drills — selectors target only the models that need rebuilding.

Orchestration Tooling Choices

dbt Cloud's native job scheduler covers cron-based and event-based (API-triggered) scheduling with built-in retry logic and is the simplest option when dbt is the only orchestration concern. Teams with broader pipelines spanning ingestion, ML training, and reverse ETL typically orchestrate dbt as one task within Airflow (via the cosmos or dbt-airflow provider packages, or a simple BashOperator calling dbt build), or within Dagster using dagster-dbt, which can translate the dbt manifest into individual Dagster assets so each dbt model becomes independently observable in the asset graph alongside non-dbt steps.

🏏

Cricket analogy: It's like choosing between a franchise's own team manager who only schedules training versus a national selector who coordinates across multiple squads and tournaments — dbt Cloud handles just dbt, while Airflow coordinates dbt alongside everything else.

python

# Example: orchestrating dbt with Airflow using the Cosmos provider
from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig
from airflow import DAG
from datetime import datetime

with DAG("dbt_daily_run", start_date=datetime(2026, 1, 1), schedule="0 6 * * *") as dag:
    transform = DbtTaskGroup(
        group_id="transform_and_test",
        project_config=ProjectConfig("/opt/dbt/my_project"),
        profile_config=ProfileConfig(profile_name="my_dbt_project", target_name="prod"),
        # renders each dbt model as its own Airflow task, respecting the dbt DAG
    )

A common orchestration mistake is scheduling dbt run purely on a wall-clock cron without checking dbt source freshness first. If an upstream Fivetran sync is delayed, the dbt run will silently build models on stale data and pass all tests, because the tests validate internal consistency, not whether the source data actually refreshed.

Orchestration decides when dbt runs relative to upstream loads and downstream consumers; dbt's own DAG only sequences models within itself.
A typical production sequence is: check source freshness, dbt build/run, dbt test, then signal downstream consumers.
Node selectors (tag:, state:modified+, folder.+) enable scoped, partial runs instead of rebuilding the whole project every time.
dbt Cloud's native scheduler is simplest when dbt is the only orchestration concern.
Airflow (via Cosmos or dbt-airflow) and Dagster (via dagster-dbt) integrate dbt as tasks/assets within broader pipelines.
dagster-dbt can translate the dbt manifest into individually observable assets in Dagster's asset graph.
Skipping source freshness checks before dbt run risks building on stale data without any test failure to flag it.