Deploying messaging

The Matrix workspace needs a messaging server (a Matrix homeserver) for each organization that uses it. The Helm chart runs one per organization, under the collaboration values. This page covers how to add one, how it is backed up and restored, and what your users see when the platform refuses one of their messages.

Each organization's homeserver:

• serves one DNS name, for example acme.chat.example.com, on port 443 of the platform's Gateway;
• signs users in through Scrydon, its only identity provider;
• does not federate with other Matrix servers;
• asks Scrydon to admit every message and room change before storing it. When Scrydon refuses, or cannot be reached, the event is not stored.

Requirements

The chart refuses to render collaboration.enabled: true unless all of these hold:

• gateway.enabled and gateway.tls.enabled. Matrix clients and the homeserver's sign-in discovery are https only.
• auth.enabled, dapr.enabled and auth.dapr.enabled. The homeserver reaches Scrydon only through Dapr, over mutual TLS.
• On the bundled PostgreSQL (infra.db.enabled: true), cortex.enabled.
• With backups on (the default), a deployment default store (storage.deploymentDefault, or the bundled SeaweedFS). Set collaboration.backup.enabled: false to run without scheduled backups.
• With at least one organization, auth.replicas: 1 (the default). The platform API holds every organization's message-reading and Cortex devices, and their encrypted keys, on one volume that only one pod can mount.
• No auth.secrets.DATABASE_URL_COLLABORATION. The chart refuses that key. If you upgrade a release that sets it, move the DSN first, as described in Bring your own database.

The data and media volumes are ReadWriteOnce. A ReadWriteOncePod storage class is not supported, because the backup job mounts the same volumes.

The platform API gets one more ReadWriteOnce volume, api-platform-matrix-devices (1Gi; set collaboration.devices.storage.size and .className to change it). It holds the encryption keys of each organization's devices. Back it up: the homeservers already know those devices, and a lost volume cannot recreate them. Because only one pod can mount it, the platform API is replaced rather than rolled during an upgrade, so it is briefly unavailable. The chart keeps the volume when you uninstall the release.

Add an organization

1. Choose a server name and point DNS at the Gateway

Pick one DNS name per organization. It becomes the organization's Matrix server name and appears in every user ID (@alice:acme.chat.example.com), so it cannot be changed later. It must:

• be lower case, with no port;
• not be a hostname the chart already serves (the platform, Cortex and the other apps);
• resolve to the Gateway's external address, the same address as your platform hostname.

The chart adds one HTTPS listener per organization to the Gateway, with its own certificate. cert-manager issues it into the Secret tls-collab-<organizationId> (override with tlsSecret, or share gateway.tls.existingSecret). If that certificate comes from a private CA, also set collaboration.oidcIssuerCa: see Collaboration homeservers behind a private CA.

The listener routes only the Matrix client API, the sign-in pages and /.well-known/matrix/{client,server}. Federation and administration paths are never routed.

2. Generate the organization's Secret

Each homeserver reads its credentials from one Kubernetes Secret. Generate it with the script in the Scrydon repository. It prints the Secret to standard output, so no value is written to disk.

The script needs a checkout of the Scrydon source repository and Bun on the machine where you run it, plus kubectl access to the release namespace. The chart does not ship it. If you install from the chart alone, ask Scrydon support for access to the repository.

bun packages/matrix-appservice/scripts/new-org-secrets.ts \
  --organization-id <organization uuid> \
  --server-name acme.chat.example.com \
  --registration-id <new uuid> \
  --appservice-url http://svc-api-platform.scrydon-platform.svc.cluster.local \
  --namespace scrydon-platform \
  | kubectl create -f -

• --registration-id is a new UUID you choose for this organization. Use the same value in the chart values.
• --appservice-url is the in-cluster address of the platform API in the release namespace. helm template prints the exact value as SCRYDON_COLLABORATION_APPSERVICE_URL in the api-platform Deployment.
• --name defaults to collab-<organization id>.
• The Secret includes the client secret users sign in with (oidc-client-secret). You do not register a client anywhere: at startup the platform registers the homeserver's sign-in client for the organization (client ID collab-<organization id>, redirect URI https://<serverName>/_continuwuity/oidc/complete). The client is a system app, so it is not shown under Settings → Organization → Registered Apps and cannot be deleted there.

A Secret created by an older version of the script may have an empty oidc-client-secret. Users cannot sign in to that homeserver, and the platform logs Chart Matrix homeserver has no OIDC client secret at startup. Fill that one key, then restart the platform API and the homeserver:

bun packages/matrix-appservice/scripts/new-org-secrets.ts --only-key oidc-client-secret \
  | kubectl patch secret collab-<organization id> -n scrydon-platform \
      --type merge --patch-file /dev/stdin
kubectl -n scrydon-platform rollout restart deploy/api-platform statefulset/collab-<organization id>

Fill an empty key only. Once the platform has stored a client secret, a different value is refused with secret_mismatch.

3. Add the organization to the values

collaboration:
  enabled: true
  organizations:
    - organizationId: 7f9c0d2e-…            # lower-case UUID of an existing organization
      serverName: acme.chat.example.com     # step 1
      existingSecret: collab-7f9c0d2e-…     # step 2
      registrationId: 5b0f…                 # step 2, the same UUID
      instanceGeneration: 1                 # bump only after a restore
      # oauthClientId: ""                   # default collab-<organizationId>
      # tlsSecret: ""                       # default tls-collab-<organizationId>
      # storage: { data: 20Gi, media: 50Gi, className: "" }

Organization IDs, server names, registration IDs and Secrets must each be unique. The chart refuses a duplicate. Two Secrets must also never hold the same admission credential. The platform checks this at startup and takes every organization that shares one out of service until an administrator fixes it.

4. Upgrade the release

Run helm upgrade. Adding an organization:

1. restarts the platform API, which records the organization's homeserver, stores a copy of its credentials and registers its sign-in client;
2. starts the organization's homeserver pod. The homeserver waits until the platform can admit its events, then starts. On a first install it waits for the collaboration database migration as well;
3. once the homeserver answers, the platform creates the organization's devices: the one that reads its rooms, and Cortex. Until then, chat shows no rooms or people.

Check that it came up:

kubectl -n scrydon-platform get pods -l scrydon.io/collaboration-organization=<organization id>
kubectl -n scrydon-platform logs deploy/api-platform | grep "Chart Matrix tenant"

Chart Matrix tenant provisioned names the organization once it is in service, and Chart Matrix managed devices provisioned once its devices exist. While the homeserver is not reachable, the log says Chart Matrix managed devices could not be provisioned; retrying, and the platform keeps trying. The first time someone in the organization opens Cortex's chat, the platform registers Cortex with the homeserver and logs Chart Matrix homeserver browser client registered. If the homeserver is not reachable yet, chat shows messaging as unavailable, the log says Chart Matrix homeserver browser client unavailable, and the next visit tries again.

Users click nothing on the homeserver. It sends them straight to the platform's sign-in, and it approves Cortex without a consent page. The approval covers only Cortex's own chat address, which is the address the platform registered.

Chart Matrix tenant refused gives a reason:

Backup

Each organization gets a nightly CronJob, collab-<organization id>-backup. It is on by default. Configure it under collaboration.backup:

Each run takes an online backup of the homeserver's database, then copies it and the media volume to the deployment default store under:

operator/00_organization/collaboration/<organizationId>/<YYYYMMDDTHHMMSSZ>/
  database/…
  media/…
  manifest.json

The filesystem's own lost+found directory at the root of the media volume is not copied: it holds nothing the messaging server wrote.

manifest.json is written last. It lists every object of the run with its size and SHA-256. A run without manifest.json is incomplete (the run stopped part-way) and must not be restored. Retention never counts it as a kept copy.

A backup is a complete copy of the organization's messaging server. Restrict access to the store accordingly.

The chart ships no alert rules. Build your own from three metrics, which the backup job and the platform API export over OTLP (observability.otlpHttpEndpoint), each labelled with organization_id:

We recommend two alerts:

• Backup failed: the failures counter increased.
• Backup stale: an organization is expected (scrydon_collaboration_backup_expected is 1 now and was 1 27 hours ago), and its successes counter has not increased in the last 27 hours. This also catches a run that never reports, such as a pod that is killed or never scheduled. It needs the platform API's metrics, so it cannot fire while those are not exported.

To take a backup now:

kubectl -n scrydon-platform create job \
  --from=cronjob/collab-<organization id>-backup collab-<organization id>-backup-manual

Agent turn retention

Agent answers in messaging rooms are kept for the organization's chat retention period, 180 days by default, and then deleted together with their stored answer and progress files. A room or organization under legal hold is kept. The chart schedules the deletion hourly; configure it under collaboration.turnRetention:

Each deletion is recorded in the organization's Audit Log before anything is removed. An organization whose retention setting cannot be read is skipped, never deleted at the default.

Restore

Restoring is a manual procedure. It replaces the organization's homeserver with the state of one backup. Messages sent after that backup are lost.

1. Pick a complete run. Choose a run prefix that has a manifest.json. Download the whole prefix, then check every object against the manifest:

   bash

   jq -r '.objects[] | "\(.sha256)  \(.key)"' manifest.json | sha256sum -c -

   Stop if any object is missing or fails the check.

2. Stop the homeserver and its backups. kubectl -n scrydon-platform scale statefulset/collab-<organization id> --replicas=0, then suspend the backup CronJob, which mounts the same volumes: kubectl -n scrydon-platform patch cronjob collab-<organization id>-backup -p '{"spec":{"suspend":true}}'

3. Rebuild the database. Mount the claim data-collab-<organization id>-0 in a maintenance pod. Move the current database files aside (everything at the volume's root except backups/). Then follow the upstream online-backup restore with the run's database/ objects: copy each shared_checksum/<id>_….sst to <id>.sst at the volume's root, then copy the contents of private/<n>/ there as well.

4. Restore media. Copy the run's media/ objects into the claim media-collab-<organization id>-0, replacing its contents.

5. Bump instanceGeneration for the organization in your values, and run helm upgrade. The upgrade sets the StatefulSet back to one replica, so this step starts the homeserver on the restored data.

6. Check the homeserver. Confirm that the pod becomes Ready, then resume the backup CronJob ("suspend":false) and take a backup now, as above.

Do not delete the claims to start over once the organization has real conversations. They are the only copy besides your backups.

When a message is refused

The platform admits a message or room change only when all of these hold:

• the homeserver is matched to an organization in service;
• the sender is a member of the room, on that organization's own server;
• the room carries a readable classification;
• the sender's clearance covers the room's classification, and any new classification the sender sets.

Agents and Scrydon's own service accounts are checked for membership and classification, but not for clearance: their output is designed to be checked against the room's classification before it is sent.

When one fails, or the platform cannot decide, the event is refused. The sender sees Message could not be sent with Try again beside the message. The reason is not shown to the sender. A refused message is never resent automatically: only a network failure is. Trying again is refused the same way until the cause changes, for example a clearance is granted.

Each refusal records one collaboration.admission.denied event in the audit log (Settings → Platform → Audit Log). The event's resource is the room, and its details carry the refusal code, the sender, the event type and the server name:

A refusal the platform could not match to an organization is recorded outside any organization's audit log, so one organization's homeserver can never write into another's.