Upgrade runbook
The order of operations for an in-place Scrydon upgrade.
This runbook covers an in-place upgrade of a running Scrydon cluster. For major-version upgrades that include breaking changes, follow the version's release notes in addition to this runbook.
Pre-upgrade
- Read the release notes. Note any breaking changes, required Helm values changes, or schema migrations that need manual steps.
- Verify the artefacts. Run
cosign verifyagainst the new images and chart. See Supply-chain verification. - Take a PostgreSQL snapshot. If anything goes wrong, this is your rollback path. See Backup & restore.
- Check the license. Confirm your license has at least 90 days remaining via Settings → License in the platform UI. If not, rotate it first. See License rotation.
- Confirm the audit forwarder is healthy. Upgrades emit a flood of events; you want them captured.
If the target release includes 0102_remove_legacy_chat, count and export public.chat from the Agentic database before running Helm. This destructive pre-upgrade migration retires the legacy Agentic hosted chat. Follow the required export and rollback procedure.
If the target release includes 0171_ba175_account_issuer_relax (Better Auth 1.7.5), run this read-only query against the auth database before running Helm, and again after the rollout completes:
SELECT provider_id, account_id, count(*) FROM account
GROUP BY provider_id, account_id HAVING count(*) > 1;It must return no rows. Each row is a sign-in identity the new version refuses to resolve, so the affected user cannot sign in with that provider until the duplicate is removed. Keep the account the user actually signs in with. Do not merge accounts on email address. This release can only be rolled back by restoring the pre-upgrade snapshot (see Rollback below).
Upgrade command
Scrydon ships as a single Helm chart (oci://scrydonops.azurecr.io/scrydon/charts/scrydon). One helm upgrade rolls every service. The chart's per-service pre-upgrade hook Jobs handle migration ordering (db-ensure-databases at weight 1, auth-migration and events-migration at weight 5, every other migration Job at weight 6) — operators do not have to sequence sub-charts by hand.
helm upgrade scrydon oci://scrydonops.azurecr.io/scrydon/charts/scrydon \
--version <new-version> \
--namespace scrydon-platform \
--values values.customer.yaml \
--wait --timeout 15m
# Verify rollout
kubectl rollout status deployment/api-platform -n scrydon-platform
kubectl rollout status deployment/agentic -n scrydon-platform # (or scrydon-agentic if you split namespaces)
kubectl rollout status deployment/analytics -n scrydon-platform # if analytics.enabledIf you've explicitly split namespaces via
namespaces.*overrides, point eachkubectl rollout statusat the matching namespace. The default chart layout keeps everything inscrydon-platform.
Watch the migration
Migrations run as Helm pre-upgrade hook Jobs (auth-migration-<rev>, events-migration-<rev>, agentic-migration-<rev>, analytics-migration-<rev>, cortex-migration-<rev>, api-ontology-migration-<rev>). They are BackoffLimit: 3 and write structured logs.
kubectl get jobs -n scrydon-platform | grep migration
kubectl logs job/auth-migration-<rev> -n scrydon-platformThe migration tracking table inside each database is drizzle.__drizzle_migrations:
kubectl exec -it deploy/db -n scrydon-platform -- \
psql -U postgres -d auth \
-c "SELECT id, hash, created_at FROM drizzle.__drizzle_migrations ORDER BY id DESC LIMIT 5;"If a migration fails, see Database migrations → Handling a migration failure.
Post-upgrade verification
After all rollouts complete, verify the platform is healthy:
- Sign in. Confirm SSO still works.
- Open a workflow. Confirm the editor loads and the integration list is correct.
- Run a known-good workflow. Confirm the run completes with the same output as before.
- Check the audit log. Confirm new events are landing.
- Check the SIEM. Confirm forwarding caught up after the rollout.
- Check metrics. Compare the post-upgrade dashboards to pre-upgrade — significant changes need investigation.
Rollback
Schema migrations are not down-migrated. If the upgrade applied a migration, rolling the application back is necessary but not sufficient — you may need to restore the database snapshot you took pre-upgrade. In particular, 0102_remove_legacy_chat drops a table and makes the previous legacy public-chat code unusable until the complete Agentic snapshot is restored. After 0171_ba175_account_issuer_relax, the previous version cannot find accounts created by the new one, so restore the auth snapshot rather than rolling back the application alone.
For a clean rollback:
# Roll the single Scrydon release back to the previous revision
helm rollback scrydon <previous-revision> -n scrydon-platform
# Confirm the revision moved
helm list -n scrydon-platformIf migrations changed schema in a way the previous app version can't read, restore from the pre-upgrade PostgreSQL snapshot.
Always validate the rollback with the same post-upgrade verification steps before declaring the rollback complete.
Related
- Upgrades — the higher-level upgrades page.
- Database migrations — migration internals.
- Backup & restore — the disaster-recovery path.