Install troubleshooting

This runbook covers the failures you are most likely to hit on a first helm install and how to recover from each. Most of them share a single root cause: on a fresh install the Helm chart applies the Dapr control plane, the database-migration jobs, and every application at the same time, so anything that prevents a pod from becoming Ready can stall the rest.

Throughout, replace <ns> with your release namespace (the platform namespace — e.g. scrydon-platform).

Quick triage

# Overall pod health — look for anything not Running/2-of-2
kubectl -n <ns> get pods -o wide

# Helm release state — "failed" means --wait gave up on a not-ready resource
helm -n <ns> history <release>

# What blocked the install (the message names the offending resource)
helm -n <ns> status <release>

A pod count of 1/2 or 0/2 (app + Dapr sidecar), a release STATUS: failed, or pods in CrashLoopBackOff / ContainerStatusUnknown all point at one of the cases below.

"Auth server unreachable" in the platform UI

The platform shows a full-screen "We're reconnecting…" message:

We're reconnecting… — Scrydon couldn't reach the authentication service. It usually comes back on its own — we'll keep trying and restore your session the moment it does.

The screen reconnects on its own: it re-checks api-platform's health on a gentle backoff (3s → 5s → 8s → every 15s, shown as a countdown) and restores the session automatically the moment the service is healthy again — no user action needed. Retry now forces an immediate attempt.

Expand "Details for administrators" on that screen for a one-line diagnosis of the cause — e.g. secret store unavailable (the Dapr sidecar case below), pending migrations, or can't connect. The diagnosis updates automatically while the screen keeps checking, so you can watch an outage move through categories (e.g. connecting → pending migrations) as the stack comes up.

This almost always means one of two things: the auth server (api-platform) is reachable but a database-backed call returned a 500 — on a fresh install because the database was never migrated (the schema is empty) — or api-platform came up without its Dapr sidecar so it can't decrypt secrets (see Pods stuck without a Dapr sidecar).

Confirm

# api-platform is up and serving (these should be 200)
kubectl -n <ns> logs deploy/api-platform -c api-platform --tail=20   # "Auth server is ready to accept requests."

# The setup-status call the platform makes — a 500 here is the smoking gun
kubectl -n <ns> run authchk --rm -it --restart=Never --image=curlimages/curl:8.7.1 -- \
  curl -s -m 6 -X POST -H "Content-Type: application/json" -d '{}' \
  http://svc-api-platform.<ns>.svc.cluster.local/api/auth/config/get-setup-status

If api-platform logs contain relation "user" does not exist (or any relation … does not exist), the schema is missing.

Cause

On current charts, first-install auth migration runs as the ordinary auth-bootstrap-<revision> Job, and API Platform waits for its evidence-table sentinel before starting. A missing auth schema therefore means the bootstrap Job failed or could not reach PostgreSQL; it is no longer hidden behind a post-install readiness hook. Older chart releases used a post-install,pre-upgrade auth hook and could skip it when another Deployment never became Ready.

Fix

1. Inspect kubectl -n <ns> logs job/auth-bootstrap-<revision> and fix the reported database, credential, or migration error.

2. Re-run the migrations. The cleanest recovery is to let Helm's auth-migration-<revision> pre-upgrade hook run them—it executes before the readiness wait:

   bash

   helm -n <ns> upgrade <release> <chart> --reuse-values

3. Verify the schema exists, then reload the platform:

   bash

   kubectl -n <ns> exec deploy/db -c postgres -- psql -U postgres -d auth -c '\dt' | head

See Database migrations → Migrations never ran on a fresh install for the manual job-based recovery if you cannot run helm upgrade.

Pods stuck at 1/1 with no Dapr sidecar

Right after a fresh install, some pods come up 1/1 (no daprd sidecar) while others are 2/2. Restarting the 1/1 pods makes them 2/2.

Cause

Dapr injects its sidecar through a mutating webhook (dapr-sidecar-injector). The webhook only injects into pods created after it is registered and Ready. A single helm install creates the Dapr control plane and the app Deployments together, so any app pod scheduled before the injector is ready gets no sidecar. Services that depend on Dapr then fail their readiness checks.

Fix

1. Confirm the control plane is healthy. With the bundled Dapr (dapr.installControlPlane: true, the default) it runs in the release namespace, not dapr-system:

   bash

   kubectl -n <ns> get pods | grep dapr     # dapr-operator, dapr-sidecar-injector, dapr-sentry, dapr-placement, dapr-scheduler
   kubectl get mutatingwebhookconfigurations | grep dapr

2. Once the injector is Ready, roll the app deployments so they get sidecars:

   bash

   kubectl -n <ns> rollout restart deployment

api-table stuck at 1/2 (readiness failing)

api-table stays 1/2 (its daprd sidecar is ready, the app container is not). Its readiness probe gates on three dependencies — Postgres, StarRocks, and the Dapr sidecar.

Diagnose

POD_IP=$(kubectl -n <ns> get pod -l app=api-table -o jsonpath='{.items[0].status.podIP}')
kubectl -n <ns> run rtest --rm -it --restart=Never --image=curlimages/curl:8.7.1 -- \
  curl -s -m 8 http://$POD_IP:7500/api/table/health/ready
# => {"status":"not_ready","checks":{"postgresql":true,"starrocks":false,"dapr":true,"opa":true,...},"stalled":[]}

The checks object names the failing dependency. starrocks:false is the most common on a fresh install.

stalled lists the checks that did not answer at all within the per-check budget (READINESS_CHECK_TIMEOUT_MS, default 3000 ms). A dependency in stalled is reported as false too, but it means "never answered" rather than "answered no": look for a wedged connection pool or a hung backend, not a misconfiguration. If the probe itself times out at the kubelet (Readiness probe failed: ... context deadline exceeded in the pod events) instead of returning a 503, raise the probe's timeoutSeconds above the budget; the budget exists so the probe can name the dependency instead of timing out silently.

Fix StarRocks (starrocks:false)

The chart ships a single-pod StarRocks (infra.starrocks.enabled: true, the allin1 image). Two things commonly keep it from serving queries:

• Replication factor. STARROCKS_REPLICATION_NUM defaults to "1", matching the single-pod backend. Only if it was overridden above the alive BE count do table writes/queries fail with "tablet replica … not enough" or "replication num should be less than or equal to the number of available BE nodes" — set it back to 1 for the single-pod deployment (or to your real BE count for an external cluster):

  yaml

  apiTable:
    starrocks:
      replicationNum: "1"   # single-pod StarRocks has one backend

• Cold start. StarRocks takes longer to accept connections than api-table. Confirm the FE is up and a backend is alive, then let api-table re-probe (it retries every 10s):

  bash

  kubectl -n <ns> logs deploy/starrocks-fe --tail=30 | grep -i 'alive\|backend'

If StarRocks is genuinely not needed, disable Managed Tables with apiTable.enabled: false and infra.starrocks.enabled: false.

daprd crash-loops with address already in use

A pod is 1/2 and its daprd container restarts with:

Failed to listen for gRPC server on TCP address :50002 with error: listen tcp :50002: bind: address already in use

and a helm upgrade --wait that rolled the pod ends in context deadline exceeded.

Cause

The app container and daprd share one network namespace. daprd listens on 50001 and 50002, which sit inside Linux's default ephemeral port range (32768–60999). If the app opens an outbound connection (for example to Postgres) before daprd binds, the kernel can give that connection local port 50002, and daprd cannot start.

Fix

Charts released after v1.3.25-rc.36 reserve those ports in every Dapr-enabled pod (dapr.reserveSidecarPorts: true, the default). On an older chart, or with the setting turned off, delete the affected pod. Its replacement gets fresh ephemeral ports. To confirm which process holds the port, read /proc/net/tcp in the app container: C352 is port 50002 in hex.

Workflow cleanup logs workflow is stalled

The workflow retention sweep logs, on every run, for the same instances:

Error purging workflow <instance-id>: 2 UNKNOWN: failed to purge workflow state: workflow is stalled

Cause

Dapr marks a workflow instance stalled when the code now deployed cannot replay its history. The usual reason is a deploy that changed the workflow's code (PATCH_MISMATCH). The instances in this log had already finished: an event woke them after they completed, they replayed on the new code, and they stalled. Dapr still reports them as finished, but it refuses to purge a stalled instance unless the purge is forced, and the Dapr JavaScript SDK cannot send a forced purge. So the sweep failed on these instances on every run, and their state was never removed.

Fix

Releases after v1.3.25-rc.39 force the purge, but only when Dapr refuses it because the instance is stalled. A purge Dapr refuses for any other reason, such as a workflow that is still running, is never forced. Each forced purge logs Workflow <instance-id> is stalled; purging it with force and records the usual workflow.purged audit event. No manual step is needed. The errors stop on the first sweep after the upgrade.

Helm release shows failed

helm history reports failed with a message like:

Release "scrydon" failed: resource Deployment/<ns>/api-table not ready

Cause

helm install --wait waited for every resource to become Ready and one Deployment never did (commonly api-table waiting on StarRocks, or a pod that was OOM-killed — see below). When the wait fails, the release is marked failed. Every service's first-install migrations run as ordinary bootstrap Jobs rather than hooks, so they are not skipped — check them with kubectl get jobs -A. On charts up to and including 1.3.25 the agentic, analytics, cortex and ontology migrations were post-install hooks, which a failed wait skips; those apps then fail with relation "…" does not exist until they run.

Fix

1. Resolve the readiness failure named in the message (the sections above cover the usual ones).

2. Re-run helm upgrade to reconcile the release to deployed. Its pre-upgrade migration hooks run the skipped migrations, and do nothing for a schema that is already current:

   bash

   helm -n <ns> upgrade <release> <chart> --reuse-values

Pods OOM-killed or crash-looping (single small node)

Symptoms: pods in CrashLoopBackOff or ContainerStatusUnknown, exit code 137, every pod failing readiness at once shortly after install.

Cause

The whole stack (all apps + the Dapr control plane + Postgres + StarRocks + object storage) was scheduled onto one undersized node. When every pod starts at once, memory limits overcommit and the kubelet OOM-kills pods, producing a cold-start storm.

Fix

• Meet the Resource Requirements — 8 vCPU / 40 GB minimum for the full default stack — and prefer at least two nodes so the scheduler can spread the load and the control plane isn't competing with the apps.

• On Azure, scaling a node pool can fail with InsufficientVCPUQuota. Check and raise your regional vCPU quota for the VM family before scaling:

  bash

  az aks nodepool scale --cluster-name <cluster> --resource-group <rg> --name <pool> --node-count 3
  # ERROR: (ErrCode_InsufficientVCPUQuota) … request a quota increase in the Azure portal

• To run on a smaller footprint, trim optional components — see Helm → Trimming for low-resource installs.

First install never converges on a small cluster

Symptoms: helm install --wait times out. db (or another pod) sat Pending with 0/N nodes are available: … Insufficient cpu or Insufficient memory, the <service>-bootstrap-<revision> Jobs failed with DeadlineExceeded, and api-platform is stuck in Init.

kubectl -n <ns> get jobs
kubectl -n <ns> get events --field-selector reason=FailedScheduling

Cause

On a first install the schema-bootstrap Jobs run at the same time as the workloads, and they need db. When the nodes cannot hold both, the scheduler places whichever pods it reaches first. If db is left waiting, the Jobs cannot reach it and fail at their 10-minute deadline. db schedules once they are gone, but the schemas the other services wait for were never created. The Resources page gives the peak a first install needs.

Fix

1. Add node capacity, or turn off components you do not need — see Helm → Trimming for low-resource installs. The migration Jobs in the next step need room to schedule too.

2. Run helm upgrade. Its pre-upgrade migration hooks create the missing schemas:

   bash

   helm -n <ns> upgrade <release> <chart> --reuse-values

Cannot reach /setup (redirects to sign-in)

If /setup bounces straight to /auth/sign-in on a brand-new install where no admin exists yet, the auth server is reporting setup as already complete. Verify with the get-setup-status call shown above. Migration-seeded system+…@scrydon.internal service actors do not complete setup; the expected response is {"setupCompleted":false} until the first human administrator exists. A true response with no human user indicates an inconsistent auth database or mixed application versions. Contact support — do not edit the auth tables by hand.