Observability

Scrydon emits metrics, logs, and OpenTelemetry traces. This page documents what's available and the SLOs you should care about.

Metrics

The services push OpenTelemetry metrics over OTLP/HTTP; nothing is scraped. Metrics and application traces are off until you give the chart a collector:

observability:
  otlpHttpEndpoint: http://otel-collector.monitoring.svc.cluster.local:4318   # OTLP/HTTP, not gRPC 4317

Every series is named scrydon_<area>_<what>, with _total for counters and _ms for durations. Services export every 15 seconds.

A series that never appears usually means otlpHttpEndpoint is empty or points at the gRPC port.

Suggested SLOs

These are guideline numbers — adjust to your workload.

Authorization evidence troubleshooting

Every permit, deny, and indeterminate result is returned only after API Platform atomically commits its Audit Log evidence and idempotency ledger. Any increase in scrydon_authorization_evidence_append_failures_total or sustained authorization 503s is an incident; the protected operation does not run. Restore OPA or the evidence database, then retry with the same stable decision key. An identical retry deduplicates, while conflicting reuse returns 409.

The workflow Event Backbone is a separate collaboration and domain-event transport. Its health does not prove that authorization evidence is complete, and authorization evidence does not depend on enabling that backbone.

Tracing

OpenTelemetry traces are emitted for:

• Every API request (entry to exit).
• Every workflow run (parent span) with child spans per block execution.
• Every LLM call through Cortex (with model, provider, latency, cost attributes).
• Every managed-table read with the policy decision as an attribute.

A request that crosses services is one trace. Each service sends the W3C traceparent header on its calls to other services, so every hop appears as a child span of the caller. When you also export Dapr spans (below), the sidecar spans join that same trace.

Configuring Dapr distributed traces

Scrydon uses Dapr for service-to-service communication. By default, Dapr spans are written to the sidecar logs (exporter: stdout) — they are not shipped to a collector.

To send Dapr distributed traces to an OTLP-compatible collector (SigNoz, Grafana Tempo, Jaeger, an OpenTelemetry Collector, Honeycomb, Datadog OTLP endpoint, etc.), set the dapr.tracing values in your scrydon Helm chart:

dapr:
  tracing:
    samplingRate: "1"   # "1" = 100%; "0" = disabled; fractional values supported
    exporter: otel      # stdout (default, spans to sidecar logs) | otel (ship to collector)
    otel:
      endpointAddress: "my-otel-collector.observability.svc.cluster.local:4317"
      protocol: grpc    # grpc | http
      isSecure: false   # true if the collector endpoint requires TLS

Important: exporter: otel with an empty endpointAddress is an error at chart render time — the chart fails closed to prevent misconfigured tracing silently dropping spans.

Example — send to an OpenTelemetry Collector sidecar/DaemonSet:

dapr:
  tracing:
    exporter: otel
    otel:
      endpointAddress: "otel-collector.monitoring.svc.cluster.local:4317"
      protocol: grpc
      isSecure: false

Example — send to Grafana Tempo:

dapr:
  tracing:
    exporter: otel
    otel:
      endpointAddress: "tempo.monitoring.svc.cluster.local:4317"
      protocol: grpc
      isSecure: false

These examples use in-cluster addresses — adjust endpointAddress to match wherever your collector is hosted. The exporter setting applies to all Dapr-enabled services in the deployment.

The application-layer spans go to observability.otlpHttpEndpoint, the same endpoint as the metrics above. Point it at your collector: Jaeger, Tempo, Datadog, Honeycomb, Lightstep or any other OTLP/HTTP receiver.

An API request whose caller disconnected before the response was written is recorded with http.response.status_code 499 (client closed request). Its span is not marked as an error. Exclude 499 when you compute a server error rate: nothing failed, and the caller received nothing.

Log access

Logs are structured JSON, emitted to stdout. Configure your log pipeline (Loki, Cloud Logging, Datadog logs, …) to ingest pod logs in the Scrydon namespaces.

Important fields on every log line:

• service — which subsystem emitted it.
• level — info / warn / error / fatal.
• requestId — correlation ID across services.
• actorId (when in a user context) — the user.
• organizationId (when in a tenant context) — the tenant.

Logs never contain secret values or PII — those are redacted at emission. See Redaction.

Related

• SIEM forwarding — for the audit-event side.
• Audit logging — the queryable audit log.