dbt Core

OrcheStack's transformation layer. The dbt service container holds an in-browser terminal + docs site; the Airflow image bakes dbt + Cosmos so DAGs can run dbt models with per-model task granularity.

Tier

Cold. The dbt service container provides an in-browser terminal and a dbt-docs site; it stays running while the operator is using it and stops when idle. Cosmos-driven dbt runs from Airflow execute inside the Airflow container (which has its own pre-installed dbt) so they don't depend on the dbt service container being warm.

Role in the pipeline

dbt is the transformation step. Raw data lands in the warehouse (from Airbyte, dlt, custom Python loaders, or any other source); dbt reads from raw schemas, applies SQL transformations defined in your project's models, writes results back. Downstream consumers (Metabase, Tableau, or engineers querying directly) read the transformed tables.

Populating your dbt project

Your dbt project lives in the orchestack-dbt-repo docker volume, mounted into the dbt service container at /usr/app/dbt (read-write) and into the Airflow container at /opt/airflow/dbt-project (read-only). Two ways to populate it:

Paste a Git repo URL into the wizard (DBT_REPO_URL). The dbt service entrypoint clones the repo on next start. Set DBT_REPO_BRANCH if you want a non-main branch. If the repo is private, also fill in the PAT field — see Private project repo: PAT below.
Use the in-browser dbt terminal — open the dbt service tile in the dashboard, click "Open Terminal", and run git clone <your-repo-url> . inside /usr/app/dbt. Iteration via direct edits works for solo development; teams should prefer Git-based flow for review and history.

If neither path is configured, the dbt service container writes a minimal demo project on first start so the dbt-docs server has something to render — one model called warehouse_tables that lists the warehouse's catalog. If your wizard URL was set but clone silently failed (private repo, no PAT), you'll see this demo project rather than yours; check the dbt container logs for the clone-failure block.

Private project repo: creating a GitHub PAT

OrcheStack collects the Personal Access Token as a separate wizard field; the backend concatenates it into the URL at write-time (https://<PAT>@github.com/owner/repo.git). You never hand-craft that URL.

Go to github.com/settings/tokens?type=beta. Click Generate new token.
Token name: orchestack-dbt.
Expiration: 90 days is a reasonable default; rotate from the dashboard's Edit Config page when it expires.
Resource owner: the user or org that owns the dbt repo.
Repository access: Only select repositories → just the dbt repo. Don't grant org-wide.
Permissions: Contents: Read-only under Repository permissions. Nothing else.
Generate → copy the token immediately (GitHub shows it once).
Paste it into the OrcheStack wizard's GitHub Personal Access Token field next to the dbt repo URL. Submit; the backend rewrites + writes to .env; the bare PAT is dropped from the credential payload.

If you also configured Airflow with a DAGs repo from the same GitHub account, you can reuse the same PAT — or generate one per service for finer-grained revocation.

Rotating the PAT later: from the dbt tile's Edit Config page, type a new value into the DBT_REPO_PAT field (always blank on display — typing replaces the embedded PAT in the URL). Save, then Stop + Open the dbt tile to re-clone.

⚠ Permission gotcha — most common cause of "remote: Write access to repository not granted":

GitHub's fine-grained-PAT system rejects clones with HTTP 403 even though git clone is read-only, when the token isn't granted the exact Contents: Read permission on the specific repo. Two things to double-check:

Permissions: under Repository permissions, set Contents: Read-only. Don't grant Write — clone doesn't need it.
Repository access: select "Only select repositories" and pick just your dbt repo. The default "All repositories" is often blocked by org policy. For org-owned repos, the org admin must approve the token from https://github.com/organizations/<org>/settings/personal-access-tokens-pending.

Project layout

A typical dbt project looks like:

your-dbt-repo/
├── dbt_project.yml         # name + profile + paths
├── packages.yml            # dbt_utils, dbt_expectations, etc.
├── models/
│   ├── staging/            # raw → cleaned (one staging model per source table)
│   │   ├── _sources.yml    # source declarations
│   │   ├── stg_orders.sql
│   │   └── stg_customers.sql
│   └── marts/              # cleaned → analytics tables
│       ├── _models.yml     # tests + documentation
│       ├── fct_orders.sql
│       └── dim_customers.sql
├── tests/                  # custom singular tests
└── macros/                 # custom Jinja macros

OrcheStack's only requirement: the profile: name in dbt_project.yml must match what's in your generated profiles.yml. The dbt service entrypoint generates profiles.yml using your project's profile name — set the project's profile: to something like orchestack_warehouse for consistency with the Airflow Connection that Cosmos uses, or any other name as long as it matches what you set in the dashboard.

⚠ DBT_DATABASE and DBT_SCHEMA — what dbt creates and what it doesn't. Operators sometimes set DBT_DATABASE to a name different from WAREHOUSE_DB_NAME in the Configure step (e.g. to separate raw landing from production marts at the database level). That works — but with one strict rule:

dbt creates schemas, never databases. If you set DBT_DATABASE to anything that doesn't already exist as a PostgreSQL database, dbt run fails on first invocation with FATAL: database "<name>" does not exist. dbt's role permissions include CREATE on a database, but the database itself must be pre-existing.
Create the database manually before starting the dbt service. Connect to PostgreSQL as orchestack_admin (via pgAdmin or docker exec orchestack-postgres psql) and run CREATE DATABASE my_analytics_db OWNER warehouse_admin;. Then set DBT_DATABASE to that name in the Configure step (or update .env + restart the dbt service if you're past Configure).
For DBT_SCHEMA, the opposite holds — dbt creates schemas automatically via CREATE SCHEMA IF NOT EXISTS on first run, so any name there works without pre-creation.

The 90% pattern stays: leave DBT_DATABASE = WAREHOUSE_DB_NAME (one database, two schemas — raw and marts). Cross-database separation is an over-engineering for SME scale; cross-schema is sufficient.

Running dbt from Airflow with Cosmos

This is the production pattern. Cosmos parses your dbt project's manifest at DAG-parse time and generates one Airflow task per model and per test, preserving dbt's dependency DAG inside Airflow's. When a model fails, the specific failed-model task lights up red — not a single opaque "dbt build" task.

from pathlib import Path
from airflow import DAG
from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
from cosmos.profiles import PostgresUserPasswordProfileMapping
from datetime import datetime

profile_config = ProfileConfig(
    profile_name="orchestack_warehouse",
    target_name="prod",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="orchestack_warehouse",      # auto-created by OrcheStack
        profile_args={"schema": "public"},
    ),
)

with DAG(dag_id="daily_models", schedule="@daily", start_date=datetime(2026, 1, 1)) as dag:
    dbt_models = DbtDag(
        dag_id_prefix="dbt",
        project_config=ProjectConfig(Path("/opt/airflow/dbt-project")),
        profile_config=profile_config,
        execution_config=ExecutionConfig(dbt_executable_path="/home/airflow/.local/bin/dbt"),
    )

Behaviour:

First DAG run after a manifest change: Cosmos re-parses; new models become new tasks automatically
Failed model: appears as a failed task with the model name; click Clear Task and Downstream to re-run only that model + its downstream models + tests
Tests: each _models.yml test becomes its own task downstream of the model it tests
Selection: pass render_config=RenderConfig(select=["+fct_orders"]) to DbtDag to run only a subset

For complete end-to-end DAGs that combine ingest + Cosmos + BI refresh, see the three patterns on the Compose your first pipeline page.

Running dbt from the terminal (interactively)

For diagnosis, ad-hoc model runs, and exploratory dbt show / dbt debug, use the in-browser dbt terminal. Open the dbt tile in the dashboard, click "Open Terminal", and you're inside /usr/app/dbt with dbt on the PATH.

Common commands:

dbt debug                              # confirm the warehouse connection works
dbt deps                               # install package dependencies (packages.yml)
dbt run --select stg_orders            # run a single model
dbt test --select fct_orders           # test a single model
dbt show --inline 'select * from raw.orders limit 5'   # quick data peek
dbt docs generate && dbt docs serve    # rebuild + serve the dbt-docs site

For production fixes, prefer editing models via your Git repo (changes go through PR review and CI). The terminal is for diagnosis — figuring out why something failed, not for nightly production runs.

Troubleshooting

"Could not find profile named 'X'": the profile: in your dbt_project.yml does not match the top-level key in profiles.yml. The dbt service auto-generates profiles.yml using whatever your project's profile: field says, so the mismatch usually means a stale generated file. Restart the dbt service container to regenerate.

Cosmos can't find the dbt project: confirm the volume orchestack-dbt-repo exists (it's created by the dbt service's compose project on first start) and that the Airflow container is mounting it read-only at /opt/airflow/dbt-project. docker volume inspect orchestack-dbt-repo from the host shows whether the volume has content.

Permission denied on warehouse schema: the dbt_admin role (or the role you configured) lacks CREATE privilege on the target schema. Connect via pgAdmin as the warehouse admin and grant: GRANT CREATE ON SCHEMA marts TO dbt_admin;

SQL error in a model: open the failed-model task in Airflow → Logs. Cosmos passes the full dbt output through. The same error reproduces from the dbt terminal with dbt run --select <the_failing_model> — that's the fastest iteration loop.

← Previous Apache Airflow Next PostgreSQL →