Moderation

Moderation is a callable content-safety capability that classifies text against hate, harassment, self-harm, violence, and other harmful categories. It is complementary to — never a replacement for — the platform's built-in Data Loss Prevention (DLP) layer.

A typical pipeline combines both: DLP strips PII from the input before it leaves your tenant; moderation scores the resulting content against your policy.

Call the capability

With a capabilities:write token (service account or notebook session), send a POST to the unified capability endpoint:

curl -X POST "https://<your-host>/api/v1/capabilities/execute" \
  -H "authorization: Bearer $ACCESS_TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "capability": "moderation",
    "input": {
      "input": "This is the text to classify."
    }
  }'

Batch input: pass an array and results align by index:

curl -X POST "https://<your-host>/api/v1/capabilities/execute" \
  -H "authorization: Bearer $ACCESS_TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "capability": "moderation",
    "input": {
      "input": [
        "First message to classify.",
        "Second message to classify."
      ]
    }
  }'

Optional model override:

{
  "capability": "moderation",
  "input": {
    "input": "...",
    "model": "omni-moderation-latest"
  }
}

412 — not configured

{ "error": "moderation_not_configured" }

This means the organization has no moderation integration enabled. An org admin can add one under Settings → Integrations. See Vendor setup below.

Response shape

{
  "results": [
    {
      "flagged": true,
      "categories": {
        "hate": false,
        "violence": true,
        "self-harm": false
      },
      "categoryScores": {
        "hate": 0.01,
        "violence": 0.93,
        "self-harm": 0.02
      },
      "orgVerdict": "fail",
      "triggeredCategories": ["violence"]
    }
  ],
  "provider": "openai",
  "model": "omni-moderation-latest",
  "usage": { "inputTokens": 8 }
}

Fields

orgVerdict semantics

The orgVerdict is the decision the org policy (see Configure the policy) makes about this result — independent of the vendor's own flagged signal.

The platform evaluates orgVerdict in this exact order, stopping at the first match:

1. No policy configured — verdict mirrors the vendor flagged signal (true → "fail", false → "pass").
2. Threshold hit — any configured category with score ≥ its threshold → "fail" (triggered categories listed in triggeredCategories).
3. Uncovered vendor flag — no threshold fired, but the vendor flagged at least one category that has no configured threshold and defaultFlagBehavior = "fail" (the default) → "fail" (those uncovered categories listed in triggeredCategories). A covered category below its threshold is an org override for that specific category only; an uncovered flagged category can still fail here.
4. Pass — none of the above conditions matched.

Example: policy has { hate: 0.8 }, vendor returns flagged: true with violence scored 0.93 and hate scored 0.4. Step 2 checks hate — 0.4 < 0.8, no hit. Step 3 finds violence is vendor-flagged and has no configured threshold → orgVerdict: "fail", triggeredCategories: ["violence"]. The hate: 0.8 threshold was an org override for hate specifically; violence still failed via defaultFlagBehavior.

Python SDK

From a Marimo notebook or any Python environment with the scrydon package:

from scrydon import moderation

# Classify one string — returns the list of result dicts by default.
results = moderation.moderate("Text to classify.")
for r in results:
    if r["orgVerdict"] == "fail":
        print("Flagged:", r.get("triggeredCategories", []))

# Batch — results align by index.
results = moderation.moderate(["First text.", "Second text."])

# Optional model override.
results = moderation.moderate("Text.", model="omni-moderation-latest")

# Full response (provider, model, usage) with detail=True.
payload = moderation.moderate("Text.", detail=True)
print(payload["provider"], payload["model"])

If the organization has no moderation integration configured, the call raises ModerationNotConfiguredError — an org admin can enable one under Settings → Integrations.

Vendor setup

An org admin adds a moderation integration under Settings → Integrations → Add Integration.

OpenAI

Select OpenAI and enter your API key. Moderation with OpenAI (omni-moderation-latest) is free — the endpoint does not incur usage charges.

The default model is omni-moderation-latest, which classifies text and images. This capability uses text-only input in v1.

Any OpenAI-compatible endpoint

Select Generic → OpenAI Compatible, enter the Base URL (e.g. https://my-server/v1), and tick Moderation in the capabilities step. The platform calls POST {baseUrl}/moderations and supports keyless (no-auth) endpoints.

Configure the policy

IT admins govern how moderation decides for the whole organization, not just which vendor serves it.

Navigate to Settings → Governance → DLP → Moderation tab.

Category thresholds

Add one row per content category you want to tune. When a vendor returns a score for that category, the platform compares it against your threshold:

• Score ≥ threshold → orgVerdict: "fail" (the category appears in triggeredCategories).
• Score below threshold → orgVerdict: "pass" — an explicit org override, even if the vendor flagged that category.

Category keys match the vendor's neutral shape (e.g. "violence", "hate", "self-harm"). Leave a category out of the map to let defaultFlagBehavior govern it.

Default-flag behavior

Controls what happens when a vendor flags a category (flagged: true) but the org has no configured threshold for it.

Governance hits

Every call where at least one result produces orgVerdict: "fail" emits a moderation.policy.hit audit event. These appear in Settings → Governance beside DLP and egress hits, filterable by category.

Automatic moderation of platform LLM calls

In addition to the callable capability above, an org admin can enable the moderation guardrail — an automatic judge that inspects LLM calls the platform makes through its main processing paths (chat, copilot, and most workflow agent steps) against the org's category thresholds. Some platform-internal LLM calls used for knowledge-base ingestion and condensation fall outside the guardrail's scope.

Enable the guardrail

Navigate to Settings → Governance → DLP → Moderation tab.

Click Enable to turn on the guardrail. The platform starts in Audit mode: the judge runs on every call in scope, hits are recorded, and nothing is blocked. Review the Hits tab to understand your traffic before switching to Enforce.

When you are confident in your thresholds, switch to Enforce mode. Blocked calls fail with LLM_BLOCKED_BY_MODERATION (see below). You can return to Audit mode at any time.

Choosing a judge

The guardrail supports two judge backends:

Both backends apply the same org-policy thresholds (categoryThresholds, defaultFlagBehavior) and produce the same verdict semantics.

Audit-first rollout

Enabling the guardrail always starts in Audit mode. In audit mode the judge runs on every call in scope, moderation.policy.hit events are emitted for failing calls, and nothing is ever blocked. Use the DLP Hits tab (filter by Moderation) to observe hit volume and tune your thresholds before committing to Enforce.

The On judge error setting (visible in Enforce mode) controls what happens if the judge itself becomes unreachable or returns a malformed verdict. The default is Block the call — a judge that cannot respond fails closed. Change this to Allow and audit if you prefer availability over enforcement during transient judge failures.

What a blocked call looks like

When the guardrail is in Enforce mode and the judge finds a failing category, the LLM call is blocked before any content is returned to the caller:

HTTP 422 Unprocessable Entity

{
  "error": "llm_blocked_by_moderation",
  "triggeredCategories": ["violence"]
}

• Chat and copilot surfaces receive the error in place of the model reply.
• Workflow agent steps fail the step with this sentinel; the error is visible in the run UI.
• Notebook and SDK callers receive the 422 response.

Streaming and unary replies

The guardrail treats streaming and unary replies differently:

• Unary (non-streaming) replies are judged before the response is delivered to the caller. In Enforce mode, a failing reply is withheld and the caller receives llm_blocked_by_moderation.
• Streaming replies are judged after the full stream completes. In all modes (Audit and Enforce), a failing streamed reply produces an audit hit — it is not blocked and is not interrupted. Mid-stream termination is a planned improvement.

The scope section of the Moderation tab shows this caveat inline for the Output direction.

Hits in the DLP panel

Moderation hits appear in Settings → Governance → DLP → Hits tab alongside DLP and egress hits. Use the Moderation filter chip to view only moderation events.

Each moderation hit row shows direction (Input / Output), action (Audited / Blocked), judge backend, and whether the judged text was truncated or came from a stream. No raw content is stored.

Relationship to DLP

DLP and moderation are complementary and independent:

• DLP is mandatory and auto-applied on every capability call. It scans inputs for secrets, PII, and injection patterns before they reach any vendor.
• Moderation is callable and org-governed. Call it when you specifically need a content-safety score — in a pipeline that processes user-generated content, before surfacing text to a broader audience, or as part of a workflow gate.

When the moderation guardrail is enabled, the ordering of DLP and the guardrail judge depends on which dispatch path handles the call. On the non-streaming (unary) path — which covers most workflow agent steps, copilot, and non-streaming chat — DLP runs on the input before the guardrail judge does, so the judge sees content after secrets and PII have already been handled by DLP. On the streaming reply path that powers streaming chat responses, DLP input handling is not yet applied before the judge; the guardrail audits the completed stream, but against unredacted text. That is a known limitation and a planned improvement.

A single call can produce both a dlp.scan.input audit row (from DLP scanning the input) and a moderation.policy.hit row (from the org policy evaluating the result).