Airbyte

Connector-based ingestion. Pulls data from 300+ sources into MinIO and PostgreSQL on the schedule you configure.

Tier

Cold. Airbyte runs during the ingestion window (scheduled by Airflow) and stops when the sync completes. Memory during a run: ~1.5–2 GB depending on dataset size.

Role in the pipeline

Airbyte is the first step of every scheduled run. It reads from your external source (database, API, CSV drop, SaaS tool) and writes landed records to two destinations in parallel:

• MinIO under s3://OrcheStack/raw/<source>/<timestamp>/ — the data lake archive for ML/DS consumers.
• PostgreSQL in the raw schema — the staging area for dbt.

Configuration

OrcheStack collects Airbyte's internal PostgreSQL credentials during setup — Airbyte uses its own database for tracking sync state. This is separate from your warehouse PostgreSQL.

• AIRBYTE_DB_HOST — usually postgres
• AIRBYTE_DB_USER, AIRBYTE_DB_PASSWORD
• AIRBYTE_DB_NAME — typically airbyte

Source credentials (for the systems Airbyte pulls FROM) are entered inside Airbyte's own UI after the first sync is configured.

Adding a new source

Click Open Airbyte from the dashboard. This spins up the container (if cold), then opens Airbyte's web UI at /app/airbyte.

1. Click Sources → + new source.
2. Choose the connector type (PostgreSQL, Google Sheets, Stripe, etc.).
3. Enter the source credentials.
4. Configure sync frequency and tables.
5. Airbyte writes to both MinIO and PostgreSQL automatically — OrcheStack pre-wires the destinations.

Troubleshooting

• Sync fails with "connection refused" — the source system is unreachable from the Docker network. Check firewall rules and whether the source needs a VPN.
• "Schema change detected" — the source added or removed a column. Airbyte's behaviour depends on your sync mode; see the Airbyte docs for schema evolution.
• Airbyte won't start — its internal PostgreSQL credentials are wrong. Edit the config in the dashboard.