Upgrades

This page covers upgrading Scrydon, both connected (Helm) and air-gapped (Zarf). You decide when to upgrade: Scrydon runs no in-cluster agent, so nothing changes until you run the upgrade.

Before you upgrade

• Read the release notes. Each release ships with notes covering breaking changes, required configuration changes, and any manual steps. Release notes are not published on a public site — request them from your Scrydon account team for the target version before upgrading.
• Back up your databases. Migrations only go forward, so the backup is your rollback point. A default install has seven databases: auth, agentic, events, analytics, cortex, ontology and lakekeeper. See Backup and restore.
• Check your license. Open Settings → Organization → License and confirm the status is valid and the expiry is after the upgrade window. An upgrade never touches the license: it lives in the platform database. See Licensing.
• Note your current chart version. Run helm list -n scrydon-platform so you have a known-good revision to roll back to if needed.
• Confirm registry credentials still work. helm registry login scrydonops.azurecr.io --username <acr-token-name> should succeed before you start the upgrade.

Database migrations

You never run migrations by hand. Every upgrade runs them as Helm pre-upgrade hook Jobs, in this order, before any new application pod starts:

The vector update runs against the PostgreSQL that is serving before the upgrade, so a release that bumps the PostgreSQL image moves the extension on the following upgrade.

<revision> is the Helm release revision, so every attempt gets a fresh Job. A Job that succeeds is deleted straight away; a failed one stays for you to inspect and is replaced on the next attempt. (On the very first install, every Job runs instead as an ordinary, non-hook Job named <service>-bootstrap-1; the collaboration schema's Job keeps its name, collaboration-migration-1.)

If a migration fails, its Job ends in BackoffLimitExceeded and helm upgrade reports the release as failed. The previous version keeps serving traffic untouched. Read the Job's logs:

bash

kubectl get jobs -n scrydon-platform | grep migration
kubectl logs job/<job-name> -n scrydon-platform

Some migrations also include data integrity preconditions — these fail fast with a clear message if your data does not satisfy them, and require operator action before retrying. See Recover From a Failed Pre-Upgrade Hook below.

Guided upgrade (recommended)

curl -fsSL https://get.scrydon.com | bash

Run against a cluster that already has Scrydon and the installer detects the existing release and switches to an upgrade. It asks only which version to move to — everything else is read back from the cluster:

This matters more than it sounds. The install interview asks about roughly ten settings; a running release commonly carries dozens — branding, email, passkey, storage, StarRocks sizing, CORS. An upgrade must not reduce your configuration to the answers of a questionnaire, so it does not re-run one.

Before changing anything it renders the new version against your current configuration server-side, so a version that needs a setting you do not have fails preflight by name instead of half-way through:

FAIL    v1.4.0 needs settings your configuration does not have
   auth.secrets.NEW_THING must contain at least 32 characters

It also refuses to move backwards without an explicit confirmation, and exits early if you are already on the version you picked.

To replace your configuration with fresh answers instead of keeping it, run with --reconfigure. That discards any setting the interview does not ask about, so reach for it deliberately.

The manual helm upgrade steps below remain fully supported.

Connected Upgrades (Helm)

Discover Available Versions

The installable versions are published at get.scrydon.com/versions.json, which the guided installer reads for you (bash setup.sh --list-versions prints the same list). Your registry credential still cannot enumerate tags itself: it is a pull-only ACR scoped token (content/read on a fixed set of repositories), so it can helm pull a known tag but cannot enumerate what is published. Both the chart source and the release notes live in a private Scrydon repository.

To find out what to install, ask your Scrydon account team. They will share the current "latest production" tag, any newer pre-release tags, and the matching release notes.

Production tags use the vMAJOR.MINOR.PATCH format (for example v1.3.6). Tags ending in -rc.N, -alpha.N, -beta.N, or -staging.N are pre-release and should not be deployed to production.

Once you have a tag from your account team, you can sanity-check that it is currently published to ACR by fetching just its Chart.yaml:

helm show chart oci://scrydonops.azurecr.io/scrydon/charts/scrydon --version v1.3.6

A 404 means the tag is not currently in the registry — older production tags age out on a retention policy. Ask your account team to have it republished if you need a tag that no longer resolves.

Upgrade

helm upgrade scrydon oci://scrydonops.azurecr.io/scrydon/charts/scrydon \
  --version <new-version> \
  --values values.customer.yaml \
  -n scrydon-platform \
  --wait --timeout 15m

Helm performs a rolling update. Old pods remain running until new pods are healthy. Use the same values.customer.yaml that was used for the previous install — overrides carry forward.

Tag format: Scrydon chart tags keep the leading v (v1.3.6), unlike most OCI registries. Pass --version v1.3.6, not 1.3.6.

Roll Back

If the upgrade introduces a problem, roll back to the previous release:

helm rollback scrydon -n scrydon-platform

To roll back to a specific revision:

# List revision history
helm history scrydon -n scrydon-platform

# Roll back to revision N
helm rollback scrydon <revision-number> -n scrydon-platform

Note: Rolling back a Helm release does not undo database migrations. If a migration added columns or tables, they remain after rollback. Scrydon migrations are designed to be backward-compatible, so the rolled-back application version will continue to function.

Exception: Explicitly announced destructive retirements are not backward-compatible. The legacy Agentic public-chat removal below drops its table before the new pods roll out; a Helm rollback alone cannot restore that feature or its data.

Exception: From 0171_ba175_account_issuer_relax (Better Auth 1.7.5) on, accounts are keyed the way the previous version cannot read. A rolled-back version does not find accounts created after the upgrade, so roll back by restoring the pre-upgrade auth snapshot. See the upgrade runbook for the check to run before and after this upgrade.

Recover From a Failed Pre-Upgrade Hook

If the upgrade fails during a pre-upgrade migration (status failed in helm history), the workloads are untouched and the previous revision is still active. To retry:

# 1. Inspect the failed job
kubectl get jobs -n scrydon-platform | grep migration
kubectl logs job/<job-name> -n scrydon-platform

# 2. Resolve the error (typically a data precondition — see the release notes
#    for any data fixes required by this version).

# 3. Retry the upgrade. It removes the failed Job and runs a fresh one.
helm upgrade scrydon oci://scrydonops.azurecr.io/scrydon/charts/scrydon ...

Air-Gapped Upgrades (Zarf)

Receive the New Package

Scrydon ships a new Zarf package for each release:

zarf-package-scrydon-amd64-<new-version>.tar.zst

Transfer the new package to your air-gapped environment via your approved transfer method.

Verify the New Package

Verify it on the connected side exactly as for a first install, with cosign verify-blob against the release workflow's identity, then check SHA256SUMS inside. The steps are in Air-gapped deployment → Pull and verify.

Deploy Over the Existing Installation

Deploy the new package over the existing installation, with the same --set-variables you used for the first install:

zarf package deploy zarf-package-scrydon-amd64-<new-version>.tar.zst --confirm \
  --set-variables DOMAIN=app.example.gov \
  --set-variables TLS_ISSUER=internal-ca

Each deploy starts from the package's defaults. A variable you leave out falls back to its default: without DOMAIN the platform moves to app.scrydon.com.

Zarf will load the new images into the in-cluster registry and update the Helm release. Existing pods are replaced using a rolling update strategy.

Roll Back (Air-Gapped)

To roll back an air-gapped upgrade, redeploy the previous Zarf package:

zarf package deploy zarf-package-scrydon-amd64-<previous-version>.tar.zst --confirm

Keep the previous package archive until the new version is stable.

Post-Upgrade Verification

After any upgrade, verify the deployment is healthy. Default chart layout puts every service in scrydon-platform; only adjust if you've overridden namespaces.*:

# Check all pods are running (single-namespace default)
kubectl get pods -n scrydon-platform
# If you split namespaces via `namespaces.agentic` / `namespaces.analytics`,
# also check those namespaces.

# Check rollout status of key deployments
kubectl rollout status deployment/api-platform -n scrydon-platform
kubectl rollout status deployment/agentic -n scrydon-platform   # (or scrydon-agentic if split)

# Check the license: Settings → Organization → License should still say `valid`

# Confirm the running chart version matches what you intended
helm list -n scrydon-platform

Open https://<your-host>/platform, sign in, and check that your workflows load.

Breaking Changes

Changed: the signed-sources key is renamed

auth.requireSignedPackSources is now auth.requireSignedExtensionSources. Nothing breaks: the chart still reads the old key, and either one set to true enforces signed extension sources. Rename it in your values.customer.yaml when convenient.

Earlier versions of the extension sources guide named auth.requireSignedExtensionSources before the chart read it. On those chart versions that key was accepted and did nothing, so a deployment that set only it was not enforcing signatures. Upgrading makes it take effect: unsigned-allowed sources then stop reconciling and their catalog content cannot be activated. Migrate those sources to cosign-keyless or cosign-key first.

Changed: the bundled Postgres password is generated and kept

infra.db.credentials.password used to default to the literal postgres. It now defaults to empty, and a live helm install generates a random password and keeps it in the secrets-infra Secret, which survives helm uninstall alongside db-pvc. Postgres fixes the password into its volume the first time it starts, so on every later helm install/helm upgrade the password in the live secrets-infra wins over whatever your values say.

What this means for an existing install:

• Direct helm upgrade, the installer, Zarf, Flux. Nothing to do. The upgrade reads the password your database already uses out of secrets-infra, whether that is postgres, the example files' REPLACE-WITH-SECURE-DB-PASSWORD, or your own value.
• GitOps and other renders without cluster access (ArgoCD, Kustomize's Helm inflator, helm template | kubectl apply, helm diff, --dry-run). Such a render cannot read secrets-infra, so it keeps the old behaviour exactly: it uses infra.db.credentials.password as written, placeholder included, or postgres when the value is empty. An install that never set the value keeps working on postgres. A new GitOps install should set its own value from your secret store (URL-safe, e.g. openssl rand -hex 16); see Rendering without a live cluster.

Rotating the bundled Postgres password

Because the live Secret wins, changing the value in your values file is ignored on a live upgrade, and the install notes print a warning saying so. The rotation is three steps, in this order:

NS=scrydon-platform   # your infra namespace (namespaces.infra)
NEW="$(openssl rand -hex 16)"

# 1. Change the password inside Postgres (postgres is the default
#    infra.db.credentials.user; use yours if you changed it).
kubectl exec -n "$NS" deploy/db -c postgres -- \
  psql -U postgres -c "ALTER USER postgres PASSWORD '$NEW'"

# 2. Record it in the Secret the chart reads back.
kubectl patch secret secrets-infra -n "$NS" --type merge \
  -p "{\"stringData\":{\"PG_DATABASE_PASSWORD\":\"$NEW\"}}"

# 3. Upgrade. Every DSN, migration Job and consumer pod picks up the new value.
helm upgrade scrydon <chart> -n "$NS" --reuse-values

For a GitOps install, do steps 1 and 2, then set infra.db.credentials.password to the new value in your secret store and let the next sync render it.

The install notes also warn while the password is still a well-known value (postgres or the example placeholder). Rotate it with the steps above.

Changed: the Gateway API replaces Ingress (v1.3.24)

From v1.3.24 the chart exposes Scrydon through the Kubernetes Gateway API — one Gateway named gateway-frontdoor plus one HTTPRoute per app — and renders no Ingress objects at all. A cluster that ran an earlier version on an Ingress controller needs three cluster changes and one values change before helm upgrade. Your current Ingress keeps serving through the first three, so they can be applied ahead of the maintenance window.

1. Install the Gateway API CRDs. They are not part of a stock cluster, and recent Traefik charts no longer ship them. Use the version your controller supports — for Traefik v3.7 (Traefik chart 40.x) that is v1.5.1:

kubectl apply --server-side \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

2. Turn on your controller's Gateway provider and publish its GatewayClass. For a Traefik release you installed yourself, add to its values and helm upgrade it at the same chart version:

providers:
  kubernetesGateway:
    enabled: true
gateway:
  enabled: false   # the Scrydon chart owns the Gateway; Traefik's own would compete with it
gatewayClass:
  enabled: true    # publishes the `traefik` GatewayClass that gateway.className names

Leave Traefik's Ingress provider on (the default). Your current Ingress keeps serving until the Scrydon upgrade replaces it, and an existing cert-manager ClusterIssuer with an http01.ingress solver keeps renewing certificates through the new Gateway. Confirm the class is accepted before continuing:

kubectl get gatewayclass traefik   # ACCEPTED must be True

3. Turn on cert-manager's Gateway API support — config.enableGatewayAPI: true, see TLS prerequisites. Without it no certificate is ever requested for the Gateway.

4. Rename ingress: to gateway: in your values file. The keys under it are unchanged. The chart's values schema no longer accepts ingress, so a file that still uses it fails validation before anything in the cluster changes.

# before                     # after
ingress:                     gateway:
  tls:                         tls:
    enabled: true                enabled: true

Then run the upgrade as usual. The Gateway takes over the same tls-frontdoor certificate Secret, so HTTPS is uninterrupted.

5. Confirm cert-manager adopted the certificate. Renewal depends on a Certificate owned by the Gateway:

kubectl get certificate tls-frontdoor -n scrydon-platform \
  -o jsonpath='{.metadata.ownerReferences[0].kind}{" "}{.status.renewalTime}{"\n"}'

The expected output is Gateway and a renewal time. If there is no Certificate at all, cert-manager saw the Gateway while the old Ingress's Certificate still existed, declined to take over a certificate it did not own, and was not prompted to look again after the Ingress was removed. HTTPS keeps working from the existing Secret, but nothing will renew it. Restart the cert-manager controller so it reconciles every Gateway again:

kubectl -n cert-manager rollout restart deployment/cert-manager

helm rollback to a pre-v1.3.24 revision re-creates the Ingress; the Gateway API CRDs and the controller settings from steps 1–3 can stay in place.

Changed: the GitOps example's secret placeholder no longer renders

values.gitops.example.yaml ships every secret you must supply as REPLACE-ME-WITH-A-REAL-VALUE-FROM-YOUR-SECRET-STORE. Most of those secrets are only checked for length, and the placeholder is long enough, so a copy of the example used to render and deploy with each of them set to a value that ships with the chart. The chart now refuses to render while any value is still the placeholder, surrounding whitespace included, and the error names the values path to replace:

agentic.secrets.INTERFACE_VISITOR_TOKEN_SECRET is still the GitOps example's placeholder ...

Replace every one from your secret store before you upgrade. To list the ones left in your values file:

grep -n 'REPLACE-ME-WITH-A-REAL-VALUE-FROM-YOUR-SECRET-STORE' <your-values-file>.yaml

If one of those values already reached your cluster, it was a public value: rotate that secret, not only the line in your values file.

api-agentic also checks INTERFACE_VISITOR_TOKEN_SECRET before it starts. It now refuses to start when the key is missing, blank, shorter than 32 characters or the placeholder, unless NODE_ENV is exactly development or test. The chart sets NODE_ENV from global.environment, so staging and every other name follow the production rules. A chart install is not affected: the chart generates the key when you do not set one. A deployment that runs api-agentic outside the chart must now provide the key (openssl rand -hex 32), where it used to fall back to BETTER_AUTH_SECRET under any NODE_ENV other than production.

Removed: Form publications

The migration 0141_drop_form_publication permanently drops public.form_publication and the form_auth_type enum from the Agentic database. The deploy sheet's form publishing, /api/forms/... and the form-publication management API are removed. The Form trigger replaces them: recreate each published form as a workflow with a Form trigger.

Before upgrading, inspect and export the table even if you expect it to be empty:

psql "<agentic-database-url>" \
  -c 'SELECT count(*) AS form_publications FROM public.form_publication;'

pg_dump "<agentic-database-url>" \
  --data-only --table=public.form_publication --column-inserts \
  > form-publications.sql

Protect form-publications.sql as sensitive material: rows can hold password hashes and allowed embed origins.

The migration does not use CASCADE. If you created a view, a foreign key, a column of type form_auth_type or another object that depends on form_publication or its enum, the upgrade stops with a dependency error: drop your object first. The migration uses IF EXISTS, so running it again once it has applied changes nothing. Rolling back needs the complete pre-upgrade database snapshot together with the previous application revision; helm rollback does not recreate the table. See Database migrations → Form-publication removal.

Removed: Legacy Agentic Public Chat

The migration 0102_remove_legacy_chat permanently drops public.chat from the Agentic database. The public hosted chat page, /api/chat and /api/chat/:identifier routes, chat-management APIs, and the legacy SDK stream client are removed. Cortex chat, A2A publications, workflow execution, and client.chat.completions remain supported.

Before upgrading, inspect and export the retired table even if you expect it to be empty:

psql "<agentic-database-url>" \
  -c 'SELECT count(*) AS legacy_chat_rows FROM public.chat;'

pg_dump "<agentic-database-url>" \
  --data-only --table=public.chat --column-inserts \
  > legacy-agentic-chat.sql

Protect legacy-agentic-chat.sql as sensitive material: historical rows can contain access configuration, allowed email addresses, and password hashes. Store it only in your approved encrypted backup location with access logging and retention controls.

The table export is an archival aid, not a supported down migration. A complete pre-upgrade Agentic PostgreSQL snapshot is the authoritative rollback point. helm rollback or redeploying an older Zarf package does not recreate the table or its data, and old Agentic pods cannot serve the retired routes after the pre-upgrade hook has run. Restoring the old feature requires restoring the complete pre-upgrade database snapshot together with the previous application revision.

Requests to removed routes receive the platform's normal 404; there is no compatibility shim. Plan the upgrade as a breaking removal and migrate callers first. See Database migrations → Legacy Agentic public-chat removal.

The hidden serialized chat_trigger workflow block type is removed as part of the same retirement. Self-hosted operators must inventory public.workflow_blocks, replace any matching blocks with the unified Start block, and redeploy those workflows before upgrading. The workflow editor's conversation panel remains supported through the unified Start block; Cortex and A2A are unchanged. Follow the inventory query and replacement guidance in Database migrations → Legacy Agentic public-chat removal.

Removed: license.product Helm Value

The license.product Helm value was removed. If your values.customer.yaml still contains it, remove it before upgrading: the chart's schema rejects it, and the upgrade fails validation with license: Additional property product is not allowed.

# Remove this line from your values file if present:
# license:
#   product: "scrydon-agentic"   # ← remove this

The license now carries resource entitlements (CPU, RAM and VRAM) instead of a product name. See Licensing.