Managed tables
Upload a file, get a typed governed table — schema inference, lifecycle, and the lifecycle states a table can move through.
A managed table is a Scrydon-owned, typed, governed table. You upload a file, the platform infers a schema on first data arrival, and from that moment the table is queryable, governable, and bindable to the ontology layer.
Creating a table
The default flow is data-first: you upload a file, the table is created from the data.
Click New table and pick a name, classification, and optional description.
Drop a CSV / JSON / JSONL file. The platform stages the file in object storage and runs schema inference on a sample.
Review the inferred columns — names, types, nullability, sample values. If a column was misclassified (e.g. a stringly-typed numeric column), correct it before confirming.
On commit, the platform creates the table in the warehouse and loads the staged rows. The table is now queryable from workflows, the ontology layer, and notebooks.
You can also create an empty table by passing a schema explicitly, but data-first is the common path.
Lifecycle states
A table can move through these states:
| State | What it means |
|---|---|
| Pending | The table exists in the catalog but no schema has been inferred yet. |
| Active | Schema inferred, queryable, accepting writes. |
| Archived | Hidden from default listings, no new writes accepted, existing queries still work. |
| Restored | An archived table that has been brought back to active. |
Archiving is soft: the data isn't deleted, the table just stops appearing in default listings. To recover data from an archived table, restore it.
Writes — what's supported
Managed tables support four write modes:
| Mode | What it does |
|---|---|
| Append | Add rows. The default. |
| Replace | Drop existing rows and load the new file. Schema is unchanged. |
| Upsert | Update rows that match by primary key, insert the rest. |
| Delete | Remove rows matching a predicate. |
Writes go through the same authorisation layer as reads. A workspace member with write access can append; only workspace admins can replace or delete.
Schema evolution
Schema bootstrap is additive. The platform enforces this rule:
- A new file with extra columns → those columns are added as nullable to the schema.
- A new file with missing columns → existing columns remain, those rows have nulls.
- A new file with a renamed column → treated as a new column. Renames are explicit, not automatic.
- A new file with a narrowed type (e.g. previously NUMERIC, now INT only) → rejected. Type widening only.
This is binding: production tables never silently narrow.
See Schema inference for the inference rules and overrides.
Storage
Managed tables are backed by an OLAP engine (StarRocks) for fast queries. Staged uploads land in object storage (SeaweedFS / S3 / Azure) before they're materialised into the warehouse.
You don't interact with the warehouse directly. Reads go through the governance layer; writes go through the same.
Iceberg lakehouse — the only storage backend
All managed tables store their data as open Apache Iceberg Parquet in your organization's own object storage. This is what makes your data portable — you can read it with any Iceberg engine (Spark, Trino, DuckDB, PyIceberg) using a Scrydon-issued token. See Iceberg lakehouse.
Because data lives in your object storage, a supported object store is required to use managed tables:
- S3-compatible — AWS S3, MinIO, SeaweedFS, or any S3-API-compatible store.
- Azure ADLS Gen2 — standard Azure Data Lake Storage Gen2.
If no object store is configured, Analytics is unavailable. Set one up under Platform → Settings → Storage, then return to Analytics. Self-hosted installs ship with an S3-compatible store (SeaweedFS) already configured, so this works out of the box for local and air-gapped deployments.
Every write mode — append, replace, and upsert / delete — runs on the Iceberg lakehouse. Upsert and delete use merge-on-read (Iceberg v2 position deletes on StarRocks 4.1+).
Profile snapshots
Each table can be profiled — the platform produces a snapshot of row counts, null rates, distinct counts, and basic statistics per column. Profiles are visible in the table-detail UI and refreshable on demand.
Profiles are informational. They don't enforce constraints. To enforce constraints (e.g. "this column must never be null"), use the Evaluator block or the Guardrails block at read time.
Where you read from
- Workflow tools — the
scrydon:tablesproduct. See Vendors → Scrydon. - Ontology projections — typed Objects backed by a
silver_tablebinding. See Ontology. - Marimo notebooks — Python notebooks. See Marimo notebooks.
- Raw SQL — authorised, governance-masked. See Querying.
External-engine write policy
When you connect a Spark, PyIceberg, Trino, or other Iceberg-compatible engine to a managed table, the platform controls whether that engine can commit new Iceberg snapshots directly to the table.
Each managed table has a write policy:
| Policy | What external engines can do |
|---|---|
governed_only (default) | External engines may read the table but cannot commit snapshots. All writes must go through the platform's governed write kernel (Analytics ingest, workflow tools, the Python SDK). |
open_append | External engines may commit append-only snapshots (new data, no deletes). The platform still enforces column masking on reads. |
open_full | External engines may commit any snapshot type — appends, overwrites, and deletes. Use only when you fully trust the writing engine. |
To change the write policy, open the table in Analytics → Settings tab →
External write access, or call the PATCH /api/table/tables/:id API with
{ "writePolicy": "open_append" }.
open_full removes the write guard. An external engine could overwrite or
delete data that the platform's governance layer cannot recover. Set this
policy only on tables whose ingestion pipeline you control end-to-end.
The policy is also stamped as an Iceberg table property (scrydon.write-policy)
so other catalog-aware tools can inspect it.
Related
- Schema inference — how types are picked.
- Classification & masking — how access and column masking work.
- Column names — handling of dotted / non-SQL-safe headers.
- Agent-created tables — when a workflow creates its own.
- Connect an external engine — read your managed tables from Spark, Trino, DuckDB, PyIceberg, Snowflake, or Databricks using an engine token.