Architecture

OrcheStack is a three-state system: a minimal base that ships in every install, a hot tier that appears after configuration, and a cold tier that spins up on demand during pipeline runs.

Three states

Traditional data-stack deployments pull every tool upfront — a "worst-case RAM" footprint that wastes resources when only parts of the stack are in use. OrcheStack instead pulls and starts services according to what's actually needed right now.

1. Base install (~2 GB)

Runs from the first docker compose up OrcheStack. Contains only the control plane:

• Reverse Proxy — routes HTTP traffic, terminates TLS, serves the landing page
• Front-facing site — this docs site, plus home, services, and contact pages
• Dashboard — admin UI, HTMX + FastAPI + Tailwind (served after auth)
• PostgreSQL — stores user accounts, roles, permissions, session ledger
• Service Orchestrator — Python process managing hot/cold lifecycles

2. Configured (~4–5 GB)

After the admin picks tools and enters credentials, OrcheStack pulls the selected services and brings up those marked hot to stay running:

• MinIO — data lake for raw files
• Metabase (or Superset / Lightdash) — BI for stakeholders
• Airflow scheduler — the clock that triggers cold services on schedule

3. Active pipeline (~7–10 GB peak)

During a scheduled DAG run or when an engineer manually triggers work, OrcheStack spins up cold-tier services for the duration of the task:

• Airbyte — ingestion window
• Airflow workers — task execution
• dbt Core — SQL transformations
• Great Expectations — data quality checks
• Elementary — dbt observability
• pgAdmin, OpenMetadata — when an engineer opens them

Cold services stop when their task completes and the reference-counted session count reaches zero.

Why three states matter

A naive deployment would keep every service running constantly. For a 10-tool stack, that's ~15 GB of RAM sitting idle between pipeline runs. OrcheStack's three-state model means:

• Your baseline resource cost is ~2 GB — what the control plane needs
• Steady-state cost is ~4–5 GB — what stakeholders need to query BI
• Peak cost is bounded to pipeline runtime — typically minutes per day

Reverse proxy as unified entry

Every service with a native web UI is reachable through the same domain via the reverse proxy:

/                → Front-facing site (landing, services, contact)
/docs/*          → This docs site
/app             → Dashboard (after auth)
/app/metabase    → Metabase (hot)
/app/airflow     → Airflow UI (hot)
/app/airbyte     → Airbyte (cold — spun up on click)
/app/openmeta    → OpenMetadata (cold)
/app/pgadmin     → pgAdmin (cold)

Clicking any cold-service URL in the dashboard triggers the orchestrator to spin it up, then redirects you to the proxy route once a health check passes.

Storage layout

OrcheStack uses a single PostgreSQL instance with multiple schemas to avoid database sprawl:

• platform — user accounts, roles, permissions, sessions, audit log
• raw — landed data from Airbyte, waiting for dbt
• marts — dbt outputs consumed by Metabase
• airflow and openmetadata schemas for those services' metadata

MinIO holds raw files as an alternate substrate for AI / ML / DS consumers who prefer objects over rows.