Pipelines

A pipeline wakes a sequence of cold-tier services on a schedule. It does not run jobs — it just turns the lights on so each tool's own internal scheduler can fire its own work.

Why pipelines exist

Most data tools have their own scheduler:

• Airflow runs DAGs on a schedule.
• Airbyte runs connections on a schedule.
• dbt + cosmos generates per-model tasks inside Airflow's schedule.

But all of these only fire if the tool's container is running. OrcheStack puts these tools in the cold tier — they sleep after 10 minutes of no activity to save resources. A cold Airflow can't fire an 8 AM DAG because it's not running at 8 AM.

Pinning solves this (a pinned service never sleeps) but it's blunt — you lose the cold-tier savings entirely. Pipelines are the lighter alternative: schedule the wake-up, then let the service go cold again when idle.

Pipelines are NOT a job orchestrator

Pipelines don't run dbt models, they don't trigger Airbyte syncs, they don't shape DAGs. Each tool keeps owning that. The pipeline's only job is "be UP at the right time."

If you need to RUN a job (e.g. "trigger this specific Airbyte connection then run dbt models then refresh Metabase"), use Airflow — write a DAG with HttpOperator + DbtTaskGroup as documented in the Airflow service page. That DAG lives inside Airflow's scheduler. The OrcheStack pipeline's job is just to make sure Airflow itself is awake when its DAG schedule fires.

Trigger types

Steps

A pipeline is an ordered list of steps. Each step has:

• Service — any service in your catalogue. Cold-tier services get woken; hot-tier services (Postgres, Metabase, pgAdmin) are tagged (always-on) in the dropdown and execute as no-ops — they're allowed in the step list so a pipeline reads as a complete dependency chain ("start postgres → start dbt") even though postgres is already up.
• Action — start or stop. The action picker is greyed out for hot-tier picks since the step no-ops regardless.
• Buffer (seconds) — after the action completes, the pipeline waits this long BEFORE running the next step. Lets the service finish booting (Airflow scheduler tick, Airbyte worker register, etc.) so step N+1 can rely on step N being responsive. Default 300 s (5 min). The last step's buffer is ignored (nothing follows it).

Execution semantics

Steps run sequentially, top to bottom.

Bail on first failure. If a step's docker compose up / stop fails, the pipeline stops there — remaining steps don't fire. Operators model pipelines as dependency chains: if "start Airbyte" fails, running "start dbt" after it is meaningless because dbt's data source isn't up. The step_results field of the pipeline_run shows exactly which step blew up.

Manual fire subsumes the pending scheduled fire. If you manually Run a pipeline that's also scheduled for later (cron or once), the upcoming scheduled fire is skipped. For cron triggers, the run advances next_run_at to the cron match after the pending slot; for once triggers, next_run_at is cleared. Prevents the "I ran it at 16:15 and now it's running AGAIN at 16:30" surprise.

No overlap. If a pipeline's previous run is still in progress when its trigger fires, the new fire is skipped (the scheduler's check is NOT EXISTS (... WHERE status='running')). The trigger won't queue — it just waits for the next match.

When stop actions make sense

Cold-tier services already auto-stop after 20 minutes of no sessions (the reconciler handles this). So you usually don't need explicit stop steps. They're useful in two cases:

• Pinned services — pinning bypasses the reconciler's idle-stop. A stop step in a pipeline is the only way to stop a pinned service on a schedule.
• Coordinated shutdown — "stop Airflow at 11 PM and Airbyte at 11:05 PM" — a single pipeline with two stop steps + a 5-min buffer between them. Two stop-pipelines on cron would also work; pick whichever reads cleanest.

Cancelling a running pipeline

The Recent runs table on a pipeline's detail page shows a Cancel button next to any run with status running (admin-only). Clicking it:

1. Marks the run as cancelled in the database immediately.
2. The executor checks this flag between steps AND every 5 seconds during the buffer wait. As soon as it sees cancelled, no further steps fire.
3. The in-flight step (the one currently running) is allowed to complete — docker compose up is already racing against the daemon and there's no clean way to abort it mid-stream. The exact behaviour: cancel after step 1 succeeds + while waiting buffer → step 2 never fires; cancel mid-step-1 → step 1 still completes, step 2 never fires.

Editor walkthrough

1. Pipelines in the sidebar → click + New pipeline.
2. Give it a name (unique, shown everywhere) + optional description.
3. Pick a trigger. For cron, the timezone field accepts any IANA name (UTC, Europe/London, Africa/Lagos, etc.).
4. Click + Add a step to add a service. Set start/stop + buffer.
5. Click Create. You land back on the list page; the new pipeline shows its next-run time + step count.

On the list page each pipeline has three buttons: Run (fire now), Edit (open the config form), Delete (remove + drop history).

Audit trail

Every operation writes an event to platform.audit_log:

• pipeline_created / pipeline_updated / pipeline_deleted
• pipeline_run_started — when a run begins (whether scheduled or manual).
• pipeline_run_succeeded / pipeline_run_failed — when a run completes.
• pipeline_run_cancelled — when an operator clicks Cancel.

The pipeline_run row itself stores step_results (per-step status + error if any) so you can drill from an audit event into the exact failure point.

Current limitations (v0.1.x)

• No inter-pipeline dependencies. Pipelines fire independently. You can't say "fire pipeline B after pipeline A completes." Workaround: stagger their cron times.
• No parallel steps within one pipeline. Steps are strictly sequential. To start two services at the same wall-clock time, create two cron pipelines with the same trigger.
• No conditional steps. Steps always run (until a failure stops the chain). No if-X-then-Y logic. That's by design — pipelines are scheduling, not orchestration.
• Cancel doesn't abort the in-flight compose command. The step that's currently racing against docker compose continues; cancel takes effect at the next inter-step boundary. Usually within seconds.