Knowledge Base Ingestion & Content Scanning

Both knowledge-base engines (RAG and Memex) ingest content through the same pipeline: files are uploaded through the platform server, scanned for secrets, and processed asynchronously — the upload dialog returns as soon as every file is queued, and per-document progress shows on the knowledge-base page.

upload → server-side secrets scan → background processing → searchable

Server-routed uploads

Knowledge-base files are always uploaded via the platform server — never directly from the browser to object storage — so every byte can be scanned before it becomes part of your knowledge base.

Upload limits

The per-drop total and the per-file cap are independent: a folder of 400 documents at 500 KB each is well inside both, even though it is far larger than any single file may be. Files over the per-file cap are skipped at the picker with a message naming the limit — the rest of the drop still uploads.

A large folder uploads in two stages, both shown on the progress bar: the bytes go up first, then each document is registered with the knowledge base in batches. The bar only reaches 100% when both stages are done.

The secrets policy (per knowledge base)

Every knowledge base has an ingest secrets policy that controls what happens when uploaded or pasted content contains detectable secrets (API keys, JWTs, email addresses):

The policy applies to every ingestion path: file uploads, pasted/inline text, documents created by workflows, and pages an agent files into a Memex knowledge base (the Knowledge Memex File Page tool). Under Block a filed page is not written: the tool receives HTTP 422 with code: "dlp_blocked".

RAG knowledge bases need an embedding provider

A RAG knowledge base turns each document into vector embeddings. Every organization starts with Scrydon Local embeddings enabled, using the 384-dimensional all-MiniLM-L6-v2 model. Another enabled embedding provider takes priority under automatic selection; you can also pin a provider and model in Settings → Platform → Extensions → Models & defaults. That choice applies to knowledge bases created afterwards: each knowledge base records the provider and model it was created with (for example scrydon/all-MiniLM-L6-v2) and keeps embedding with them, so enabling another embedding provider later does not move an existing knowledge base onto it. If all embedding connections are disabled, the platform blocks the action up front rather than accepting documents that could never be indexed:

• Creating a RAG knowledge base is blocked — the create dialog disables the button and tells you to enable an embedding integration first.
• Uploading to (or adding inline documents to) a RAG knowledge base returns HTTP 412 with code: "EMBEDDING_NOT_CONFIGURED" and a reason of either org_not_configured (no embedding integration enabled for the org) or no_model_selected (the knowledge base has no embedding model).

Enable an embedding integration in Settings → Platform → Extensions, then create the knowledge base or retry the upload. Memex knowledge bases do not embed and are never subject to this check.

An embedding provider that is enabled but cannot be reached is a different case: its endpoint is down, or the organization's egress policy refuses it. Creating a knowledge base without naming a model then returns HTTP 503 with code: "PROVIDER_UNAVAILABLE", retryable: true and the provider's reason, and no knowledge base is created. Fix or disable that provider, or pin another one, then retry.

Asynchronous processing

Processing (parsing, scanning, chunking/embedding for RAG; page derivation for Memex) runs in durable background workflows. A document moves through pending → processing → completed (or failed with a reason); progress and failures are visible per file on the knowledge-base page.

Both engines share one folder/file explorer on the knowledge-base page: files are browsed by folder, each row shows the file's size and upload date alongside its processing status, and a failed file can be reprocessed directly from its row — for RAG this retries indexing, for Memex it re-runs the ingest pipeline from the originally uploaded file. With edit permission you can select multiple files (or select-all) and delete them in one action, or delete a whole folder — its row's Delete folder action removes every file under it and all its subfolders. Deletion cascades into every entry derived from each file and cannot be undone.

Writing text notes

The Explorer is the default knowledge-base view. It uses a file browser on the left and an editor or preview on the right. Users with edit permission can create a Markdown note, edit .md, .mdx, and .txt files, and save them without leaving the explorer. Markdown files use live preview; plain-text files keep Markdown-looking syntax literal. Uploaded or derived sources without an editable canonical text body remain read-only.

On smaller screens, files and the editor are separate views with a Back to files action. On desktop, choose Edit, Preview, or Split. The selected note is retained in the URL, so reload, browser back/forward, and a shared deep link restore the same file. Cmd+N / Ctrl+N creates a note in the current folder; Cmd+S / Ctrl+S saves immediately, while idle changes are saved automatically.

Saving unchanged text is a no-op. RAG notes retain their last searchable index while changed chunks are embedded and the replacement index is committed; chunks whose content hash is unchanged reuse their existing vector. Memex notes update the lexical index synchronously without an AI-model call.

Content and index state are reported separately: Unsaved, Saving, Saved, Indexing, Indexed, Saved — index stale, or Index failed. A stale editor never overwrites another revision: it asks you to load the latest content. Failed RAG indexing can be retried while the last good searchable vectors remain active. Binary and derived span-backed items remain read-only and offer Create Markdown copy. See Knowledge note editing for the complete format and editor behavior.

Beyond the per-file retry, a bulk re-ingest of all failed documents in a RAG knowledge base is available to operators (POST /api/knowledge/rag/{knowledgeBaseId}/reprocess-failed, requires edit permission on the base). It re-runs ingestion for every document currently in the failed state through the same durable pipeline — leaving already-completed documents and their embeddings untouched — which is the fastest way to recover a base whose documents failed before an embedding provider was configured.

For Memex knowledge bases, uploading a file whose content already exists in the knowledge base ends in a terminal skipped state instead: the duplicate is detected during background processing (the content hash is computed after parsing and secret redaction), so the upload itself always returns immediately and the existing pages are left untouched. Unsupported file formats and audio files without a configured speech-to-text integration are still rejected synchronously at upload time.

Tagging & filtering documents (RAG only)

RAG knowledge bases support document tags — typed fields (text, number, date, or boolean) defined per knowledge base and set individually on each document. Tags let you track structured metadata alongside your content — for example, a Region field to record which geography a document covers, a Year field for the publication year, or a Reviewed boolean for a QA workflow.

Defining tag fields

Open the knowledge base and click the Tags button in the explorer toolbar to define, rename, or delete fields for the whole knowledge base. You can also create a new field directly from any document's tag popover (see below) — the field becomes available across all documents in that knowledge base immediately.

Viewing and editing tags in the explorer

The explorer table includes a Tags column. Each document shows its set tags as Name: Value chips; if there are more chips than fit in the column a +N overflow indicator is shown. A document with no tags set shows — add tags —.

Click the chips (or the — add tags — placeholder) to open the inline tag editor for that document. From there you can:

• Set or clear a value for any existing field.
• Create a new field and immediately set its value on the current document.
• Click Save to apply the changes — the row updates in place without reloading the page.

Editing tags requires workspace edit permission. Viewers can see tag values but the editor does not allow changes.

Filtering by tag

The filter bar above the explorer lets you narrow the document list by one or more tag conditions — for example:

• Region equals EU
• Year greater than 2024
• Reviewed equals true

Multiple conditions are ANDed together. While any filter is active the explorer shows a flat, knowledge-base-wide list of matching documents rather than the folder tree — all folders are searched regardless of which one is currently open. Clear all filters to return to the folder tree view.