BYO Database

Scope: replacing the in-cluster Postgres deployed by infra.db with a customer-managed instance. Covers AWS RDS, GCP Cloud SQL, Azure Database for PostgreSQL (Flexible Server), Supabase, Neon, CloudNativePG, the Zalando Postgres Operator, or any other Postgres ≥ 15 with the extensions required by your enabled components.

Related: Backup & Restore (the bundled backup CronJob is scoped to the in-cluster DB only — managed-DB users should rely on the provider's native PITR).

Why

The bundled infra.db is a single-pod Postgres backed by a PVC — fine for proofs of concept and single-node clusters, but it lacks point-in-time recovery, synchronous replication, automated failover, and regional snapshot replication. Production customers and most compliance audits (SOC 2, ISO 27001 A.8.13, SecNumCloud 3.2, NIST CP-9/CP-10) expect a managed service or a dedicated DBA-operated cluster.

Prerequisites

Pre-create databases and extension

Run once, from any client that can reach your managed DB. The default chart enables every component, which maps to six databases; drop the lines for any service you've disabled in your values file.

psql "$ADMIN_URL" <<'SQL'
CREATE DATABASE auth;
CREATE DATABASE agentic;
CREATE DATABASE analytics;   -- only if analytics.enabled (default true)
CREATE DATABASE cortex;      -- only if cortex.enabled (default true)
CREATE DATABASE ontology;    -- only if apiOntology.enabled (default true)
CREATE DATABASE lakekeeper;  -- only if infra.lakekeeper.enabled (default true)

\c auth
CREATE EXTENSION IF NOT EXISTS vector;

\c agentic
CREATE EXTENSION IF NOT EXISTS vector;

\c cortex
CREATE EXTENSION IF NOT EXISTS vector;

\c lakekeeper
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS btree_gin;
CREATE EXTENSION IF NOT EXISTS btree_gist;
SQL

The bundled chart creates the databases and pgvector automatically via an initdb script; Lakekeeper's migration init container creates its own extensions. For managed DBs, pre-create everything above so installation fails before deployment if the provider or migration role cannot create an extension. analytics and ontology do not require pgvector; auth, agentic, and cortex do.

Why this must happen before helm install: the chart runs the auth Drizzle migration as the ordinary auth-bootstrap-<revision> Job and the remaining per-service migrations as post-install hooks against DATABASE_URL directly — drizzle-kit migrate does not create the database itself. Lakekeeper also runs a lakekeeper migrate init container. A missing database surfaces as failed migration Jobs or a Lakekeeper pod stuck during init shortly after install.

Configuration

The five app databases support two modes. Pick one per app — when both are set for an app, infra.db.external.existingSecrets.<app> wins over infra.db.external.<app>Url. Lakekeeper currently supports inline mode only: set infra.db.external.lakekeeperUrl whenever both infra.db.enabled=false and infra.lakekeeper.enabled=true.

Mode A — external Secrets (recommended)

Keep app DSNs out of values files and CI logs. Pre-create one Secret per app, each in the app's target namespace, each carrying a single DSN under the key that app expects. The chart adds each as a second envFrom on the corresponding deployments.

1. Create one Secret per enabled app database (five with chart defaults):

NS=scrydon-platform   # adjust if you use split namespaces

# auth → key DATABASE_URL → auth DB
kubectl create secret generic scrydon-db-auth -n "$NS" \
  --from-literal=DATABASE_URL='postgres://scrydon:REDACTED@pg.example.com:5432/auth?sslmode=require'

# agentic → key DATABASE_URL → agentic DB
kubectl create secret generic scrydon-db-agentic -n "$NS" \
  --from-literal=DATABASE_URL='postgres://scrydon:REDACTED@pg.example.com:5432/agentic?sslmode=require'

# analytics → key DATABASE_URL_ANALYTICS → analytics DB
kubectl create secret generic scrydon-db-analytics -n "$NS" \
  --from-literal=DATABASE_URL_ANALYTICS='postgres://scrydon:REDACTED@pg.example.com:5432/analytics?sslmode=require'

# cortex → key DATABASE_URL_CORTEX → cortex DB
kubectl create secret generic scrydon-db-cortex -n "$NS" \
  --from-literal=DATABASE_URL_CORTEX='postgres://scrydon:REDACTED@pg.example.com:5432/cortex?sslmode=require'

# ontology → key DATABASE_URL_ONTOLOGY → ontology DB
kubectl create secret generic scrydon-db-ontology -n "$NS" \
  --from-literal=DATABASE_URL_ONTOLOGY='postgres://scrydon:REDACTED@pg.example.com:5432/ontology?sslmode=require'

Why one Secret per app: each app reads its DSN env var (DATABASE_URL, DATABASE_URL_ANALYTICS, DATABASE_URL_CORTEX, DATABASE_URL_ONTOLOGY) via envFrom, which merges every key of the referenced Secret into the container env. Sharing one Secret would force apps that read DATABASE_URL (auth, agentic) to see the same value and point at the same database — wrong. Per-app Secrets keep each app pinned to its own database while allowing all apps to share a namespace.

2. Install the chart:

# values.customer.yaml
infra:
  db:
    enabled: false
    tls:
      enabled: false
    backup:
      enabled: false
    external:
      existingSecrets:
        auth: scrydon-db-auth
        agentic: scrydon-db-agentic
        analytics: scrydon-db-analytics
        cortex: scrydon-db-cortex
        ontology: scrydon-db-ontology
      lakekeeperUrl: "postgres://scrydon:REDACTED@pg.example.com:5432/lakekeeper"

helm upgrade --install scrydon oci://scrydonops.azurecr.io/scrydon/charts/scrydon \
  --version <version> \
  --namespace scrydon-platform \
  -f values.customer.yaml

Lakekeeper is the exception to Mode A: its DSN must currently be supplied as lakekeeperUrl. The chart appends sslMode and writes the same DSN to Lakekeeper's read and write variables in the chart-managed secrets-lakekeeper Secret, alongside its encryption key. The value therefore appears in values files and helm get values.

If you manage app secrets via External Secrets Operator, Sealed Secrets, or SOPS, make sure the resulting Secret has the correct key (DATABASE_URL, DATABASE_URL_ANALYTICS, DATABASE_URL_CORTEX, or DATABASE_URL_ONTOLOGY) in the right namespace — the chart never inspects how the Secret got there.

The agentic database also backs the shared Dapr execution-security state used by Agentic and API Platform. If namespaces.agentic and namespaces.platform differ, create the same named Agentic database Secret, with the same DATABASE_URL, in both namespaces. Dapr components are namespace-local; the chart renders one component in each namespace and scopes each copy only to its owning app.

You may also mix modes per app (e.g. existingSecrets.auth for auth, agenticUrl for agentic) — Mode A wins per app when both are set. Lakekeeper still requires lakekeeperUrl.

Mode B — inline DSNs

Simpler — one values block, no pre-created Secrets. DSNs end up in the chart-managed secrets-<app> Secret inside the cluster. Fine for staging; avoid in production (DSNs sit in values files, CI logs, and helm get values output).

# values.customer.yaml
infra:
  db:
    enabled: false
    tls:
      enabled: false
    backup:
      enabled: false
    external:
      authUrl:      "postgres://scrydon:REDACTED@pg.example.com:5432/auth"
      agenticUrl:   "postgres://scrydon:REDACTED@pg.example.com:5432/agentic"
      analyticsUrl: "postgres://scrydon:REDACTED@pg.example.com:5432/analytics"
      cortexUrl:    "postgres://scrydon:REDACTED@pg.example.com:5432/cortex"
      ontologyUrl:  "postgres://scrydon:REDACTED@pg.example.com:5432/ontology"
      lakekeeperUrl: "postgres://scrydon:REDACTED@pg.example.com:5432/lakekeeper"
      sslMode: require   # appended as ?sslmode=require to each DSN

What still needs to stay in-cluster

Migrations

The chart runs one Helm-hook Job per enabled component — auth-migration, agentic-migration, and (when enabled) analytics-migration, cortex-migration, api-ontology-migration — as post-install / pre-upgrade hooks. Each one sources its DSN from the same Secret the runtime uses, so they apply against your managed DB.

When Lakekeeper is enabled, its lakekeeper-migrate init container applies catalog migrations using lakekeeperUrl before the catalog server starts.

The wait-for-db init container that ping-tests the in-cluster service FQDN is skipped when infra.db.enabled=false — external DB readiness is the provider's responsibility.

To re-run all migrations manually after a values change:

kubectl delete job -l 'app in (auth-bootstrap,auth-migration,agentic-migration,analytics-migration,cortex-migration,api-ontology-migration)' -n scrydon-platform
helm upgrade scrydon oci://scrydonops.azurecr.io/scrydon/charts/scrydon \
  --version <version> \
  --namespace scrydon-platform \
  -f values.customer.yaml

Verification

# Apps should be Ready with Dapr sidecars injected (2/2). The chart defaults
# every service to scrydon-platform; add your namespaces.* targets if you split.
kubectl get pods -n scrydon-platform

# No in-cluster Postgres should exist
kubectl get deploy db -n scrydon-platform
# Expected: Error from server (NotFound)

# Migration Jobs should have run successfully
kubectl get jobs -n scrydon-platform
# auth-bootstrap-<rev> on install (auth-migration-<rev> on upgrade),
# agentic-migration-<rev>, and (when enabled)
# analytics-migration-<rev>, cortex-migration-<rev>, api-ontology-migration-<rev> — Complete

# Lakekeeper's migration init container should have exited successfully
kubectl logs deploy/lakekeeper -n scrydon-platform -c lakekeeper-migrate

# A running app pod should have DATABASE_URL pointing at the managed host
kubectl exec deploy/api-platform -n scrydon-platform -c api-platform -- \
  sh -c 'echo "$DATABASE_URL" | sed -E "s#//[^@]+@#//REDACTED@#"'

Provider notes

AWS RDS / Aurora

• Parameter group: set rds.force_ssl=1.
• Extension: pgvector available from Postgres 15.3 on RDS and Aurora. Install via CREATE EXTENSION vector; from an rds_superuser-owned role.
• Role: create a least-privilege role (CREATE ROLE scrydon LOGIN PASSWORD '…' CREATEDB;) and grant it per-DB ownership.
• IAM auth: not supported by the chart today — use password auth.

GCP Cloud SQL

• cloudsql.iam_authentication=on flag is optional; chart uses password auth.
• pgvector enabled by default on Postgres 15+ Enterprise Plus; on standard tiers run CREATE EXTENSION vector; once.
• Private IP: run the cluster in the same VPC as your Cloud SQL instance, or front it with a Cloud SQL Auth Proxy sidecar.

Azure Database for PostgreSQL (Flexible Server)

• Server parameter: add VECTOR to azure.extensions before the first CREATE EXTENSION.
• Networking: private DNS zone + VNet peering is recommended; alternatively open firewall rules for the AKS egress IP.
• CA bundle: Azure publishes a combined CA bundle. Mount it and use sslmode=verify-full&sslrootcert=/etc/ssl/azure-ca.pem for full verification.

Supabase / Neon

• pgvector enabled by default.
• Connection string format (pooled, transaction mode): postgres://postgres.<project>:<pass>@aws-0-<region>.pooler.supabase.com:6543/postgres. Verify your migrations run via the pooled connection; some DDL may require the direct (5432) connection.
• Neon's auto-suspend will freeze idle branches — set a keepalive or use a paid tier for platforms that idle between workflow runs.

Restore onto managed Postgres

To restore a pg_dump --format=custom archive produced by the bundled backup CronJob (see Backup & Restore):

# decompress if you used gzip
gunzip -k /backups/auth-20260416T020000Z.sql.gz

# restore into the managed DB (run from any client with network access)
pg_restore -d "$DATABASE_URL" --no-owner --no-privileges \
  /backups/auth-20260416T020000Z.sql

--no-owner --no-privileges is important — role names on the managed DB usually differ from the bundled Postgres (postgres vs scrydon / rds_superuser / cloudsqlsuperuser).

After restore, roll the app pods so they pick up fresh connections:

kubectl rollout restart deployment -n scrydon-platform

Troubleshooting

CREATE EXTENSION vector fails — the managed provider does not have pgvector on your current Postgres version / tier. Upgrade the instance; no workaround.

App pods CrashLoopBackOff with ECONNREFUSED or SSL required — DSN host/port wrong, TLS mandatory but sslmode missing, or network policy blocks egress to the DB. kubectl logs the pod, check the DSN value with the kubectl exec command under Verification.

Migrations hang — the wait-for-db init container is skipped, so the Job goes straight to migrating. If it hangs, the DSN is wrong or the role lacks CREATE privileges. Inspect kubectl logs job/auth-bootstrap-<rev> on install or kubectl logs job/auth-migration-<rev> on upgrade.

Lakekeeper stays in Init — verify lakekeeperUrl, network access, the lakekeeper database, and the five Lakekeeper extensions. Inspect kubectl logs deploy/lakekeeper -c lakekeeper-migrate.

Dapr workflow state table not found — the Dapr Workflows component auto-creates its schema on first start. If you are replacing the bundled DB with an existing managed DB that was used elsewhere, ensure the tablePrefix values (dapr_workflow_state, analytics_dapr_workflow_state) do not collide.

Compliance mapping

• ISO 27001:2022 A.8.13 (information backup) — managed provider's native backup satisfies this; disable infra.db.backup.enabled.
• ISO 27001:2022 A.8.24 (TLS in transit) — sslmode=require minimum, verify-full preferred.
• NIST CP-9 / CP-10 (backup & restore) — managed PITR coverage; document the provider's RPO/RTO in your BCP.
• SecNumCloud 3.2 (continuity) — requires regional or multi-AZ replication, which the bundled in-cluster Postgres cannot deliver. BYO managed Postgres is the supported path.