Backup & restore

Scrydon's state lives in three stores: PostgreSQL for relational state, object storage (SeaweedFS or S3) for blobs, and an OLAP warehouse (StarRocks) for analytics rows. This runbook covers backing up and restoring all three.

What's where

Backup cadence

The recommended baseline:

For regulated environments, extend retention to whatever your compliance framework requires.

PostgreSQL backup

If you're using a managed Postgres (Azure Database for PostgreSQL, Amazon RDS, …), use the provider's PITR + snapshot tools.

If you're using the platform-bundled Postgres (not recommended for production), the recommended pattern is pg_basebackup for full backups + WAL archive for PITR. The Helm chart documents the volume mounts.

For a simpler logical-backup baseline, the chart can retain pg_dump files on db-backup-pvc and ship them to Azure Blob. Create a Secret containing a scoped SAS token, then enable both jobs:

infra:
  db:
    backup:
      enabled: true
      retentionDays: 30
      offCluster:
        enabled: true
        endpoint: https://ACCOUNT.blob.core.windows.net
        container: scrydon-backups
        objectPrefix: production/
        egressCidrs:
          - 10.40.12.7/32 # Azure Blob private endpoint
        credentialSecret:
          name: scrydon-backup-azure
          key: sas-token

Each run dumps every database the chart creates for the enabled apps: seven on a default install. To dump fewer, list them in infra.db.backup.databases.

Before the databases, each run also writes globals-<timestamp>.sql.gz: the cluster roles, from pg_dumpall --globals-only, without passwords. pg_dump works on one database at a time and leaves roles out, but every row-level security policy is granted to the scrydon_tenant or scrydon_system role, so a restore needs them first. See step 2 of the full restore.

Both jobs run on the node that hosts the db pod. db-backup-pvc is ReadWriteOnce and the db pod keeps it mounted, and a cloud disk attaches to one node at a time, so a backup pod on any other node could never start. Leave room on that node for the backup pod (it requests 50m CPU and 128Mi memory).

The shipper uploads each .sql.gz file and an adjacent .sha256 checksum. The SAS token is read from the existing Secret and never placed in Helm values. When NetworkPolicies are enabled, egressCidrs is required and opens only TCP/443 from the shipper. Prefer a private endpoint with stable, approved CIDRs; do not use 0.0.0.0/0 for a public Blob endpoint. Dumps become visible to the shipper only after an atomic rename, so an overlapping run cannot upload a partially written archive. Configure Azure lifecycle/immutability rules on the container separately; the chart controls local PVC retention, not remote deletion. Test a restore from the remote object regularly.

Object storage backup

If you're using SeaweedFS (the bundled blob store), set up volume replication to a secondary location. SeaweedFS supports cross-region replication via its filer's replication feature.

If you're using S3 / Azure Blob / GCS, the provider's cross-region replication is straightforward.

StarRocks backup

The platform-bundled StarRocks is a single-pod image with a snapshot mechanism for individual tables. For production HA, use the StarRocks Kubernetes operator with native FE/BE backup.

Restore — full disaster recovery

Order of operations for a full DR scenario:

1. Provision the target cluster with the same Kubernetes version and Scrydon version.

2. Restore PostgreSQL from your most recent backup (or PITR target). Restore every database together: auth, agentic, events, analytics, cortex, ontology and lakekeeper on a default install. The license, organizations, integrations, workflows and ontologies are joined across them, and lakekeeper holds the catalog for the Iceberg tables in object storage. Restore the roles first, then each database:

   bash

   gunzip -c globals-<timestamp>.sql.gz | psql -U postgres -d postgres
   createdb -U postgres auth
   gunzip -c auth-<timestamp>.sql.gz \
     | pg_restore -U postgres -d auth --no-owner --no-privileges
   # ...the same for every other database in the set

   role "postgres" already exists from the first command is expected. Every pg_restore must then finish without errors. role "scrydon_tenant" does not exist means the roles were not restored first: each row-level security policy was skipped, so the tables keep row-level security switched on with no policy, and PostgreSQL then denies every row to the application. Running the migrations does not put the policies back, because the restored databases record those migrations as applied. Drop the databases and restore again in this order.

3. Restore object storage to the same paths.

4. Restore StarRocks snapshots.

5. Apply the Helm chart pointing at the restored stores.

6. No license re-application step. The license lives in the restored auth database's platform_config row — once Postgres comes back, api-platform reads it on the next request. If you need to rotate the license post-restore, use Settings → License → Update license in the UI (see License rotation).

7. Validate by signing in and confirming workflows / tables / knowledge bases are visible.

Dapr crypto master key

The Dapr crypto component encrypts every secret value the platform writes (OAuth client secrets, vendor API keys, encrypted env vars passed to integration providers) before it lands in Postgres. The encryption key lives in a Kubernetes Secret named master-key in the Scrydon release namespace.

The chart auto-generates this key on fresh install via the Helm lookup function — but it is never part of any subsequent Helm release, which means a routine helm uninstall + helm install cycle, or any restore that re-applies the chart against a fresh namespace, will generate a new key. Any encrypted-secret rows in the restored Postgres are then unrecoverable.

Back up the Secret out-of-band and restore it before applying the chart:

# Capture the key into a YAML file (run this on the source cluster).
kubectl get secret master-key -n scrydon-platform -o yaml \
  | grep -v '^\s*resourceVersion\|^\s*uid\|^\s*creationTimestamp' \
  > master-key.backup.yaml

# Store master-key.backup.yaml off-cluster — encrypted at rest. Treat it
# with the same protection class as your Postgres backups.

# Restoring on the target cluster (BEFORE running helm install):
kubectl apply -f master-key.backup.yaml -n scrydon-platform

Alternatives if your team already runs one of these:

• Sealed Secrets — seal the master-key Secret with the cluster's controller key; the sealed file is safe to commit. Re-apply on disaster recovery before the chart.
• External Secrets Operator — source master-key.key from your cloud key manager (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault). The chart's lookup-based auto-generate path will not overwrite a Secret that already carries the Helm ownership annotations.
• etcd snapshots — captures every Kubernetes Secret including master-key. Useful for full-cluster DR; less useful for cross-cluster restore.

Compliance mapping. ISO 27001:2022 A.5.16 / A.8.24 — cryptographic-key management requires a documented backup, rotation, and recovery procedure for any key that protects production data. The master-key rotation procedure is covered in the internal Dapr ADR; for customer purposes, treat this backup as a hard prerequisite to a working PostgreSQL restore.

Restore — single store

If you only need to restore one store (e.g. a corrupted knowledge-base blob), do the targeted restore against that store. Other stores stay live.

For PostgreSQL, restoring a single table requires a logical dump that targets it specifically (pg_dump -t).

Verifying a backup

A backup that hasn't been restored isn't a backup. Run a monthly drill:

1. Download yesterday's PostgreSQL dumps and their .sha256 companions from the off-cluster store, verify the checksums, and restore into a scratch cluster: the globals file first, then every database. Each pg_restore must finish without errors.
2. Sign in with a known admin account.
3. Open a workflow, a knowledge-base document, and a managed table.
4. Confirm the audit log shows yesterday's events.
5. Tear the scratch cluster down.

If any step fails, escalate immediately — a silently failing backup is a known operational hazard.

Related

• Upgrade runbook — restore is one of the rollback paths.
• Database migrations — schema-aware aspects of restore.
• Licensing — reapplying the license post-restore.