Database migrations

Every Scrydon upgrade may include database migrations. This runbook covers what runs, how to verify, and how to handle a failed migration.

How migrations run

The Helm chart assigns each schema one migration Job owner. On first install, auth uses the ordinary auth-bootstrap-<revision> Job so it can run alongside the Deployments; API Platform replicas wait for its schema sentinel. On upgrade, auth-migration-<revision> runs as a weighted pre-upgrade hook. Other services retain their chart-defined migration hooks.

Migrations are forward-only by design. Down migrations are not provided; if you need to roll back, restore from backup.

What a migration does

Migrations are typically:

• Adding a new column (ADD COLUMN ... NULL).
• Backfilling derived data.
• Adding an index in CONCURRENTLY mode.
• Soft-deprecating a column (renamed, then read from new + old, then read-from-new only, then drop).

Migrations that would block writes for more than a few seconds are split across multiple releases — the platform never ships a single-step lock-the-table migration on a large table.

Verifying a migration ran

# Find the migration version
kubectl exec -it deploy/api-platform -n scrydon-platform -- \
  psql "$DATABASE_URL" -c "SELECT version, applied_at FROM platform_migrations ORDER BY applied_at DESC LIMIT 5;"

Each service has its own migrations table.

Handling a migration failure

If a migration fails, its Job fails and the migration error appears in the Job logs. A dependent pod may remain in Init or NotReady while it waits for the required schema.

Recovery procedure:

1. Read the failing pod's logs.
2. Identify the migration version and the SQL statement that failed.
3. Decide between three paths:
   • Fix forward: apply a hot-patch to the failing migration. Restart the pod.
   • Skip the migration (if you've decided it's safe): mark the migration applied in the migrations table manually and restart.
   • Restore from backup: roll back to the pre-upgrade state. Restore PostgreSQL from the snapshot you took before the upgrade.

Always take a PostgreSQL snapshot before starting an upgrade. The upgrade runbook calls this out explicitly.

Migration tracking desync

The migration job verifies its tracking table against the actual database schema on every run. If the tracking table claims migrations are applied whose objects do not exist, the job fails with:

[migrate-bootstrap] FATAL: the migration tracking table is out of sync with
the actual database schema — it claims migrations are applied whose objects
do not exist. This is a tracking desync, not a migration bug.

When the message offers MIGRATE_REPAIR_TRACKING=1 ("Re-run with MIGRATE_REPAIR_TRACKING=1 to rewind tracking from ..."), the state is provably auto-repairable: the affected migrations left no objects behind, so re-applying them cannot conflict with existing data. Enable the repair via the matching service's values flag:

# values override for the affected service only — auth, analytics,
# cortex, apiOntology, or agentic
analytics:
  migration:
    repairTracking: true

Then re-run helm upgrade. The migration job rewinds the falsely-tracked rows and re-applies the migrations in order.

If the failure message instead reports a partial or non-contiguous state, the auto-repair refuses to run by design — contact support before touching the tracking table manually.

Long-running backfills

Some upgrades introduce a backfill (populating a new column from existing data). For large datasets, the backfill runs as a separate background job, not blocking the rolling update.

The progress is visible in the audit log as MIGRATION_BACKFILL_* events. The job can be paused, resumed, or restarted from where it left off.

What helm upgrade doesn't touch

helm upgrade orchestrates both migration Jobs and the workload rollout. This means:

• A Helm rollback doesn't un-apply migrations.
• A failed rollout may already have applied its pre-upgrade migrations.
• The schema owner and its migration history remain service-specific even though Helm launches the Jobs.

Migrations never ran on a fresh install

On a first install, auth migration runs as the ordinary auth-bootstrap-<revision> Job. API Platform waits for the authorization evidence relations before starting, so helm install --wait cannot skip auth migration behind a post-install readiness gate. If the bootstrap fails, API Platform remains in Init; inspect the Job directly:

kubectl -n <ns> get jobs -l app=auth-bootstrap
kubectl -n <ns> logs job/auth-bootstrap-<revision>

Other services that still use post-install migration hooks can be skipped when an unrelated Deployment never becomes Ready. Their service logs then show relation "…" does not exist.

Confirm the schema is missing:

kubectl -n <ns> exec deploy/db -c postgres -- psql -U postgres -d auth -c '\dt' | head
# "Did not find any tables." => migrations never ran

Recover. First fix whatever kept a pod from becoming ready (see Install troubleshooting). Then re-run the migrations one of two ways:

1. Re-run via Helm (preferred). The migration hook is pre-upgrade, which runs before the readiness wait — so it completes even if a slow service is still settling:

   helm -n <ns> upgrade <release> <chart> --reuse-values

2. Run the migration jobs manually (if you can't run helm upgrade). Auth bootstrap is in helm get manifest; the remaining install migrations are in helm get hooks. Extract only the relevant Job manifests and apply them as ordinary Jobs:

   helm -n <ns> get manifest <release> > /tmp/manifest.yaml
   helm -n <ns> get hooks <release> > /tmp/hooks.yaml
   # extract auth-bootstrap-* and the relevant *-migration-* Job manifests, then:
   kubectl -n <ns> apply -f /tmp/migrations.yaml
   kubectl -n <ns> wait --for=condition=complete job -l app.kubernetes.io/name=scrydon --timeout=300s

The jobs are idempotent — they detect a fresh database and apply every migration from scratch, and become a no-op once the schema is present.

Related

• Install troubleshooting — first-install failures and recovery.
• Upgrade runbook — full upgrade procedure.
• Backup & restore — the rollback path.