Database migrations

Every Scrydon upgrade may include database migrations. This runbook covers what runs, how to verify, and how to handle a failed migration.

How migrations run

Each database has exactly one migration Job, and the Helm chart runs it for you:

The lakekeeper database belongs to the bundled Iceberg catalog, which migrates its own schema when it starts.

On an upgrade every Job runs as a pre-upgrade hook, named <job>-<revision>, before any new pod starts; Upgrades gives the order. On the first install, every Job runs as an ordinary, non-hook Job named <service>-bootstrap-1, so no schema waits for a workload to become Ready. The collaboration schema's Job keeps its name, collaboration-migration-1.

With collaboration.enabled, collaboration-migration-1 is an ordinary Job on the first install because the homeservers only become Ready once API Platform has provisioned their organizations on its schema: a post-install hook runs only after helm install --wait (or a Zarf deployment) has seen every pod Ready, so it would never run. On the bundled PostgreSQL, the database's first start also creates the scrydon_tenant and scrydon_system roles the Job requires. API Platform can start before the Job has created the collaboration schema. It does not wait for it: it retries provisioning the collaboration organizations with a growing delay and provisions them as soon as the Job completes, without a restart. It never gives up: once the delay reaches five minutes it keeps retrying every five minutes for as long as the schema is missing, so fixing and re-running the Job is enough and API Platform does not need a restart. While an organization keeps failing, API Platform logs Chart Matrix tenant still refused ensure_failed; retrying at most once an hour for it.

Migrations are forward-only by design. Down migrations are not provided; if you need to roll back, restore from backup.

What a migration does

Migrations are typically:

• Adding a new column (ADD COLUMN ... NULL).
• Backfilling derived data.
• Adding an index in CONCURRENTLY mode.
• Soft-deprecating a column (renamed, then read from new + old, then read-from-new only, then drop).

Migrations that would block writes for more than a few seconds are split across multiple releases — the platform never ships a single-step lock-the-table migration on a large table.

An explicitly announced product retirement can remove a table after the release's operator export gate. Treat those migrations as exceptions to the usual additive rollout pattern and follow their release-specific procedure.

Legacy Agentic public-chat removal

Migration 0102_remove_legacy_chat drops the retired public.chat table. Before the migration Job runs, record the row count and create a protected data export:

psql "<agentic-database-url>" \
  -c 'SELECT count(*) AS legacy_chat_rows FROM public.chat;'

pg_dump "<agentic-database-url>" \
  --data-only --table=public.chat --column-inserts \
  > legacy-agentic-chat.sql

This export supports inventory and controlled archival; it is not a down migration. The supported rollback point is a complete snapshot of the Agentic database taken before the upgrade. The migration is a pre-upgrade hook, so rolling Helm back after it succeeds does not recreate the table, and the previous Agentic application cannot serve its legacy public-chat routes against the new schema. Restore the complete database snapshot and previous application revision together if a full rollback is required.

The release also removes the hidden legacy chat_trigger workflow block type. Inventory workflow definitions before upgrading:

psql "<agentic-database-url>" \
  -c "SELECT count(*) AS legacy_chat_trigger_blocks, count(DISTINCT workflow_id) AS affected_workflows FROM public.workflow_blocks WHERE type = 'chat_trigger';"

If either count is non-zero, replace each legacy block with the unified Start block and redeploy the workflow before upgrading. Do not update the block type directly in PostgreSQL: immutable deployment versions and execution snapshots can contain separate copies of the workflow definition. After the upgrade, Agentic treats chat_trigger as an unknown retired block and removes it during workflow sanitization.

The removal does not affect Cortex conversation tables, A2A publications, workflow checkpoints, copilot feedback, or workflow-editor conversation state.

Form-publication removal

0141_drop_form_publication drops public.form_publication and the form_auth_type enum. It runs DROP TABLE without CASCADE on purpose: an object of yours that depends on the table stops the upgrade instead of being dropped silently. Both statements use IF EXISTS, so applying the migration again changes nothing.

Inventory dependent objects before upgrading. The query lists every object that depends on the table or on its enum: views, foreign keys, columns (arrays included), domains, functions and constraints. Each row is the object's description, for example column a of table c2 or rule _RETURN on view v, which is what you drop:

SELECT DISTINCT pg_describe_object(d.classid, d.objid, d.objsubid) AS dependent_object
FROM pg_depend d
WHERE d.deptype = 'n'
  AND ((d.refclassid = 'pg_class'::regclass AND d.refobjid = to_regclass('public.form_publication'))
    OR (d.refclassid = 'pg_type'::regclass AND d.refobjid IN (to_regtype('public.form_auth_type'), to_regtype('public.form_auth_type[]'))))
  AND NOT (d.classid = 'pg_class'::regclass AND d.objid = to_regclass('public.form_publication'))
  AND NOT (d.classid = 'pg_attrdef'::regclass AND d.objid IN (SELECT oid FROM pg_attrdef WHERE adrelid = to_regclass('public.form_publication')));

No rows means the migration will apply. On a database that no longer has the table the query also returns no rows, without an error. Export the rows first; Upgrades → Removed: Form publications has the commands.

Verifying a migration ran

Every database tracks its applied migrations in drizzle.__drizzle_migrations. On the bundled PostgreSQL, list the five most recent in one database:

kubectl exec -it deploy/db -n scrydon-platform -- \
  psql -U postgres -d auth \
  -c "SELECT id, hash, to_timestamp(created_at / 1000) AS applied_at FROM drizzle.__drizzle_migrations ORDER BY id DESC LIMIT 5;"

Replace auth with events, agentic, analytics, cortex or ontology for the others. With an external database, run the same query with psql against that database's URL.

Handling a migration failure

If a migration fails, its Job fails, helm upgrade reports the release as failed, and the previous version keeps serving traffic. The error is in the Job's logs:

kubectl get jobs -n scrydon-platform | grep -E 'migration|bootstrap'
kubectl logs job/<job-name> -n scrydon-platform

On a first install a pod that needs the schema stays in Init or NotReady until the Job succeeds.

Then pick one of three paths:

• The error names a data precondition. Some migrations check your data first and stop with a message that says what to fix. Fix the data, then run the same helm upgrade again. The next attempt replaces the failed Job with a fresh one.
• The error is the tracking desync below. Follow Migration tracking desync.
• Anything else. Restore the databases from the snapshot you took before the upgrade, roll back with helm rollback, and send the Job logs to Scrydon support.

Always take a PostgreSQL snapshot before starting an upgrade. The upgrade runbook calls this out explicitly.

Migration tracking desync

The migration job verifies its tracking table against the actual database schema on every run. If the tracking table claims migrations are applied whose objects do not exist, the job fails with:

[migrate-bootstrap] FATAL: the migration tracking table is out of sync with
the actual database schema — it claims migrations are applied whose objects
do not exist. This is a tracking desync, not a migration bug.

When the message offers MIGRATE_REPAIR_TRACKING=1 ("Re-run with MIGRATE_REPAIR_TRACKING=1 to rewind tracking from ..."), the state is provably auto-repairable: the affected migrations left no objects behind, so re-applying them cannot conflict with existing data. Enable the repair via the matching service's values flag:

# values override for the affected service only: auth, agentic, analytics,
# cortex or apiOntology (events has no repair flag)
analytics:
  migration:
    repairTracking: true

Then re-run helm upgrade. The migration job rewinds the falsely-tracked rows and re-applies the migrations in order.

If the failure message instead reports a partial or non-contiguous state, the auto-repair refuses to run by design — contact support before touching the tracking table manually.

What helm upgrade doesn't touch

helm upgrade orchestrates both migration Jobs and the workload rollout. This means:

• A Helm rollback doesn't un-apply migrations.
• A failed rollout may already have applied its pre-upgrade migrations.
• The schema owner and its migration history remain service-specific even though Helm launches the Jobs.

Migrations never ran on a fresh install

On a first install, every service's migrations run as an ordinary Job, not a Helm hook: auth-bootstrap-1, events-bootstrap-1, agentic-bootstrap-1, analytics-bootstrap-1, cortex-bootstrap-1 and api-ontology-bootstrap-1. None of them waits for any workload to become Ready, so a slow or failing Deployment cannot hold a schema back. Install with --wait --wait-for-jobs and Helm also fails the release when a bootstrap Job exhausts its retries. If one fails, the services that read its database stay unready; inspect the Job directly:

kubectl get jobs -A -l 'app in (auth-bootstrap,events-bootstrap,agentic-bootstrap,analytics-bootstrap,cortex-bootstrap,api-ontology-bootstrap)'
kubectl -n <ns> logs job/<service>-bootstrap-1

Confirm the schema is missing:

kubectl -n <ns> exec deploy/db -c postgres -- psql -U postgres -d auth -c '\dt' | head
# "Did not find any tables." => migrations never ran

Recover. First fix whatever kept a pod from becoming ready (see Install troubleshooting). Then re-run the migrations one of two ways:

1. Re-run via Helm (preferred). The migration hook is pre-upgrade, which runs before the readiness wait — so it completes even if a slow service is still settling:

   bash

   helm -n <ns> upgrade <release> <chart> --reuse-values

2. Run the migration jobs manually (if you can't run helm upgrade). The bootstrap Jobs are in helm get manifest; on a 1.3.25 or older release the agentic, analytics, cortex and ontology migrations are in helm get hooks instead. Extract only the relevant Job manifests and apply them as ordinary Jobs:

   bash

   helm -n <ns> get manifest <release> > /tmp/manifest.yaml
   helm -n <ns> get hooks <release> > /tmp/hooks.yaml
   # extract the relevant *-bootstrap-* (or, on 1.3.25 and older, *-migration-*) Jobs, then:
   kubectl -n <ns> apply -f /tmp/migrations.yaml
   kubectl -n <ns> wait --for=condition=complete job/<job-name> --timeout=300s

The jobs are idempotent — they detect a fresh database and apply every migration from scratch, and become a no-op once the schema is present.

Related

• Install troubleshooting — first-install failures and recovery.
• Upgrade runbook — full upgrade procedure.
• Backup & restore — the rollback path.