Orchestrating dbt Runs
Orchestration is the layer that decides when dbt commands execute, in what order relative to upstream data loads and downstream consumers, and how failures are handled — dbt itself only resolves dependencies within its own DAG, it has no native awareness of whether Fivetran finished syncing or whether a Looker dashboard is waiting on fresh data. In production, a typical orchestrated run is a sequence: check source freshness, run dbt source freshness, trigger dbt run for all or selected models, run dbt test to validate the output, and only then signal downstream systems that new data is ready.
Cricket analogy: It's like a match day schedule that sequences the pitch inspection, toss, and innings in strict order — you don't start bowling before the toss decides who bats, just as dbt run shouldn't start before source freshness is confirmed.
Selectors and Partial Runs
Rather than rebuilding an entire project on every schedule, orchestration typically triggers scoped runs using dbt's node selection syntax: dbt run --select tag:hourly runs only models tagged for hourly refresh, dbt build --select state:modified+ (used with dbt Cloud's deferred state or a stored manifest.json) rebuilds only models that changed plus their downstream dependents, and dbt run --select finance.+ rebuilds a folder and everything downstream of it. This selective execution is what makes orchestrating dozens of pipelines with different SLAs practical — a marketing mart that needs hourly freshness and a finance mart that only needs daily refresh can run on independent schedules against the same project.
Cricket analogy: It's like a coach running targeted net sessions for just the fast bowlers before a pace-friendly pitch rather than putting the entire squad through identical drills — selectors target only the models that need rebuilding.
Orchestration Tooling Choices
dbt Cloud's native job scheduler covers cron-based and event-based (API-triggered) scheduling with built-in retry logic and is the simplest option when dbt is the only orchestration concern. Teams with broader pipelines spanning ingestion, ML training, and reverse ETL typically orchestrate dbt as one task within Airflow (via the cosmos or dbt-airflow provider packages, or a simple BashOperator calling dbt build), or within Dagster using dagster-dbt, which can translate the dbt manifest into individual Dagster assets so each dbt model becomes independently observable in the asset graph alongside non-dbt steps.
Cricket analogy: It's like choosing between a franchise's own team manager who only schedules training versus a national selector who coordinates across multiple squads and tournaments — dbt Cloud handles just dbt, while Airflow coordinates dbt alongside everything else.
# Example: orchestrating dbt with Airflow using the Cosmos provider
from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig
from airflow import DAG
from datetime import datetime
with DAG("dbt_daily_run", start_date=datetime(2026, 1, 1), schedule="0 6 * * *") as dag:
transform = DbtTaskGroup(
group_id="transform_and_test",
project_config=ProjectConfig("/opt/dbt/my_project"),
profile_config=ProfileConfig(profile_name="my_dbt_project", target_name="prod"),
# renders each dbt model as its own Airflow task, respecting the dbt DAG
)A common orchestration mistake is scheduling dbt run purely on a wall-clock cron without checking dbt source freshness first. If an upstream Fivetran sync is delayed, the dbt run will silently build models on stale data and pass all tests, because the tests validate internal consistency, not whether the source data actually refreshed.
- Orchestration decides when dbt runs relative to upstream loads and downstream consumers; dbt's own DAG only sequences models within itself.
- A typical production sequence is: check source freshness, dbt build/run, dbt test, then signal downstream consumers.
- Node selectors (tag:, state:modified+, folder.+) enable scoped, partial runs instead of rebuilding the whole project every time.
- dbt Cloud's native scheduler is simplest when dbt is the only orchestration concern.
- Airflow (via Cosmos or dbt-airflow) and Dagster (via dagster-dbt) integrate dbt as tasks/assets within broader pipelines.
- dagster-dbt can translate the dbt manifest into individually observable assets in Dagster's asset graph.
- Skipping source freshness checks before dbt run risks building on stale data without any test failure to flag it.
Practice what you learned
1. What does dbt's own DAG resolve?
2. What does dbt run --select state:modified+ do?
3. Why is checking source freshness before dbt run important?
4. What does dagster-dbt provide?
Was this page helpful?
You May Also Like
dbt Cloud vs dbt Core
A practical comparison of dbt's two deployment models — the self-hosted open-source CLI versus the managed dbt Cloud platform — and how to choose between them.
Environments and Deployment
How dbt separates development, staging, and production environments using targets, schemas, and deployment jobs so changes are validated before reaching production models.
CI/CD for dbt Projects
How to build continuous integration and deployment pipelines for dbt projects using slim CI, automated testing, and gated production deploys.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics