Apache Airflow

OrcheStack ships Airflow 2.10 with dbt + astronomer-cosmos preinstalled. Author DAGs in your own Git repository; Airflow discovers and runs them on the cadence you set.

Tier

Cold. The Airflow webserver + scheduler are stopped when nothing's actively using them; the orchestrator brings them up when an operator opens the tile. Important caveat for scheduled DAGs: a cold Airflow can't fire an 8 AM DAG because it's not running at 8 AM. If you rely on schedule-based runs, pin Airflow from its detail-page Keep warm card — pinned services are excluded from the cold-tier idle-stop reconciler. The dashboard surfaces a yellow banner on Airflow's detail page as a reminder.

What ships in the image

Airflow 2.10.5 with three things baked in at build time so operators never run pip install at task time:

• dbt-core 1.8.0 + dbt-postgres 1.8.0 — for running dbt models against the warehouse. (Pinned to 1.8.0 because dbt 1.9+ requires protobuf ≥5, which conflicts with Airflow 2.10's pinned protobuf 4.25.6. We'll bump when we move back to Airflow 3.x.)
• astronomer-cosmos 1.8.x — turns the operator's dbt project into one Airflow task per model + per test
• Airflow's HTTP provider — for triggering external tools (Airbyte, Fivetran, Metabase, Tableau, etc.) via their REST APIs

Anything else the operator's DAGs need — dlt, requests, pandas, custom Python loaders — gets installed per-task via PythonVirtualenvOperator (see below).

The image runs as the root user (Docker prints a warning about this on start; safe to ignore — Airflow 2's upstream image is intentionally root-by-default in single-container mode).

Connections

Cosmos needs an Airflow Connection named orchestack_warehouse with the warehouse credentials. OrcheStack creates this connection automatically on first Airflow start through the orchestrator's post-start hook — no manual setup needed.

For other connections (Airbyte's API, Metabase's API, Tableau Server, your custom HTTP endpoints), add them in the Airflow UI: Admin → Connections → +. Common ones:

Where your DAGs live

DAG Python files live in /opt/airflow/dags/ inside the Airflow container, backed by the orchestack-airflow-dags docker volume. Two ways to populate it:

1. Paste a Git repo URL into the wizard (AIRFLOW_DAGS_REPO_URL) — the Airflow entrypoint clones the repo on each start. Set AIRFLOW_DAGS_REPO_BRANCH if you want a non-main branch. This is the recommended pattern. If the repo is private, fill in the PAT field too (see next section).
2. Bind-mount a host directory — for single-developer iteration. Replace the airflow-dags volume with a bind mount and Airflow picks up local edits within 30 seconds.

Your dbt project files live in a separate location: /opt/airflow/dbt-project/ inside the Airflow container (read-only). This is a mount of the same volume the dbt service uses, so populating your dbt project on the dbt side makes it visible to Airflow automatically. See the dbt Core page for how to populate it.

Private DAG repo: creating a GitHub PAT

For private GitHub repos, OrcheStack embeds a Personal Access Token in the clone URL (the git client inside the container can't use any other auth flow). The wizard collects it as a separate field; you don't have to hand-craft the https://<PAT>@github.com/... URL — the backend concatenates it for you on save.

1. Go to github.com/settings/tokens?type=beta (fine-grained tokens). Click Generate new token.
2. Token name: orchestack-airflow-dags (or whatever distinguishes it from your other PATs).
3. Expiration: pick what your security policy demands. 90 days is a reasonable default for an on-prem deploy.
4. Resource owner: the user or org that owns your DAGs repo.
5. Repository access: Only select repositories → pick just the DAGs repo. Don't grant org-wide access.
6. Permissions: under Repository permissions, set Contents: Read-only. Nothing else needs to be checked.
7. Click Generate token and copy it immediately (GitHub only shows it once).
8. In the OrcheStack setup wizard, paste the token into the GitHub Personal Access Token field next to your DAG repo URL. Submit. The backend rewrites the URL to https://<PAT>@github.com/owner/repo.git and writes the result to .env; the bare PAT key is never written separately.

If you need to rotate the PAT later: go to the Airflow tile's Edit config page, type a new PAT into the AIRFLOW_DAGS_REPO_PAT field (it's always blank on display — typing replaces the embedded PAT in the URL). Save, then Stop + Open the Airflow tile to re-clone.

Diagnosing clone failures: if Airflow starts but no DAGs appear, check the container logs — they print a multi-line classified diagnostic block on failure (auth / not-found / network). Operator URL: /app/services/airflow → Activity tab → look for the most recent session_autostart_failed event for the orchestrator-side error. The orchestrator also writes a demo orchestack_welcome DAG to the volume on first start so you'll see at least one DAG in the Airflow UI — useful for confirming Airflow itself is functioning even when your repo clone has issues.

Repo layout: OrcheStack respects AIRFLOW_DAGS_REPO_PATH (default dags). DAGs at your-repo/dags/*.py get copied into the container's DAG folder as expected. If your DAGs live at the repo root instead, set AIRFLOW_DAGS_REPO_PATH=. in the wizard or Edit Config.

Running dbt from Airflow with Cosmos

Cosmos's DbtDag reads the operator's dbt project at DAG-parse time and generates one Airflow task per dbt model and per dbt test. The dbt DAG becomes visible inside the Airflow UI, and per-model failures surface with the correct task ID — no more "dbt_build [FAILED], scroll the logs to find which model."

from pathlib import Path
from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
from cosmos.profiles import PostgresUserPasswordProfileMapping

profile_config = ProfileConfig(
    profile_name="orchestack_warehouse",
    target_name="prod",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="orchestack_warehouse",      # auto-created by OrcheStack
        profile_args={"schema": "public"},   # or "marts", "analytics", etc.
    ),
)

dbt_dag = DbtDag(
    dag_id_prefix="dbt",
    project_config=ProjectConfig(Path("/opt/airflow/dbt-project")),
    profile_config=profile_config,
    execution_config=ExecutionConfig(
        dbt_executable_path="/home/airflow/.local/bin/dbt",
    ),
)

For end-to-end DAGs that combine ingest + Cosmos dbt + BI refresh, see the three patterns on the Compose your first pipeline page.

Triggering external tools (HttpOperator)

For any tool with a REST API — Airbyte's connection-sync endpoint, Metabase's dashboard-refresh endpoint, Tableau Server's data-source-refresh endpoint, Fivetran's connector-trigger endpoint — use HttpOperator (and its companion HttpSensor for polling completion).

from airflow.providers.http.operators.http import HttpOperator
from airflow.providers.http.sensors.http import HttpSensor

trigger = HttpOperator(
    task_id="trigger_airbyte",
    http_conn_id="airbyte",                  # set in Airflow UI
    endpoint="api/v1/connections/sync",
    method="POST",
    data='{"connectionId": "{{ var.value.airbyte_connection_id }}"}',
)

wait = HttpSensor(
    task_id="wait_for_airbyte",
    http_conn_id="airbyte",
    endpoint="api/v1/jobs/{{ ti.xcom_pull(task_ids='trigger_airbyte')['job']['id'] }}",
    response_check=lambda r: r.json()["job"]["status"] == "succeeded",
    poke_interval=30,
    timeout=60 * 60,
)

The same pattern works for any tool that exposes an HTTP trigger + a status endpoint to poll. Only the connection ID, endpoint URL, and response-check logic change.

Custom Python ingestion (PythonVirtualenvOperator)

For ingestion logic written in Python — using libraries OrcheStack doesn't bake in (dlt, requests, pandas, your custom loader) — use PythonVirtualenvOperator. Airflow creates an ephemeral virtualenv with the listed dependencies on first task run (~30-60s), caches it, then reuses the cached venv on subsequent runs (sub-second).

from airflow.operators.python import PythonVirtualenvOperator

def load_data():
    # Runs in an isolated venv with `dlt[postgres]` installed.
    import dlt
    pipeline = dlt.pipeline(pipeline_name="github", destination="postgres", dataset_name="raw")
    pipeline.run([{"id": 1, "name": "example"}], table_name="github_repos")

load_task = PythonVirtualenvOperator(
    task_id="load_with_dlt",
    python_callable=load_data,
    requirements=["dlt[postgres]>=1.0"],
    system_site_packages=False,
)

This is how the platform stays unopinionated about ingestion: any Python library that exists on PyPI can ingest into the warehouse without rebuilding the Airflow image.

Composing patterns

The patterns above are independent. Mix and match for your pipeline:

• Airbyte ingest + dbt + Metabase: HttpOperator + DbtDag + HttpOperator
• dlt ingest + dbt + Tableau: PythonVirtualenvOperator + DbtDag + HttpOperator
• Multiple sources in one DAG: two HttpOperators in parallel (or one HttpOperator + one PythonVirtualenvOperator), then DbtDag downstream
• dbt-only schedule: a DAG with just DbtDag if your ingest fires from somewhere else

Three worked-out compositions are on the Compose your first pipeline page — copy whichever matches your stack, edit, run.