Iceberg & open data
Scrydon managed tables are backed by Apache Iceberg — a table format you own. Export to any engine, bring your own catalog, and keep full data sovereignty.
Scrydon Analytics stores managed tables in Apache Iceberg — an open table format with standardised REST catalog semantics. This means the structured data you ingest through Scrydon is not locked in a proprietary warehouse: it lives in Parquet files in your object store, described by an Iceberg catalog that any conforming engine can read.
What this means in practice
Sovereignty and portability
Every managed table has an open Iceberg representation. The Iceberg files live in the same object store bucket you already configured for Scrydon — S3, Azure ADLS, or the bundled SeaweedFS storage for self-hosted deployments. If you choose to leave Scrydon, the data and its full schema history travel with you.
The exit test: pip install pyiceberg && pyiceberg --catalog <your-catalog-uri> describe <your-namespace>.<your-table> should return your table definition using only open-source tooling and your own credentials.
A governed REST catalog facade
Scrydon exposes the Iceberg REST catalog protocol at:
https://<your-host>/api/table/iceberg/v1/This surface is governed: every request requires an engine token (a scoped API key issued in the Platform UI), and every request is isolated to the issuing organisation. The facade proxies catalog operations to an internal Lakekeeper instance — an Apache-2.0 Rust Iceberg REST catalog that Scrydon manages on your behalf.
Two modes
| Mode | What it is |
|---|---|
| Managed Iceberg (default) | Scrydon creates and owns the Iceberg catalog entries. Your StarRocks warehouse, any connected BI tool, and your data pipelines all read the same governed table. |
| BYO catalog | Connect an existing Iceberg catalog (REST, AWS Glue, Nessie) as a Scrydon data source. Scrydon reads it using a single service-principal credential you supply; your existing catalog's governance is the outer boundary. |
Two-tier governance model
The governance model below is specific to the managed Iceberg path. BYO catalogs rely on the external catalog's own access controls.
Scrydon enforces data governance at two layers:
| Tier | Enforcer | What it covers |
|---|---|---|
| Catalog access | Scrydon REST facade (/api/table/iceberg/v1/*) | Who can reach the catalog at all, and whether they may write. Engine tokens carry organizationId and an optional canWrite flag. |
| Row-level data | StarRocks read path (the trusted engine) | Column masking, row filters, and DLP — enforced at every SQL read regardless of which tool issued the query. |
External engines — Spark, Trino, DuckDB, PyIceberg — connect through the governed catalog facade. They receive all granted rows through standard Iceberg scan planning. Fine-grained masking (partial redaction, hashing) that Scrydon enforces at the SQL layer is not applied to raw Parquet reads via the Iceberg data-plane. This is an honest property of the open format: if a column is stored as plaintext in Parquet, an engine with object-store access can read it. Scrydon's column masking is enforced at the StarRocks SQL layer; for data that must be masked from all callers including Iceberg engines, use the data classification controls to restrict access at the table level rather than relying on partial masking.
Storage requirements
| Storage | Behaviour |
|---|---|
| AWS S3 | STS credential vending — Iceberg engines receive short-lived credentials per table access. No static keys required in engine config. |
| Azure ADLS | Short-lived SAS credentials vended by Lakekeeper on each access. |
| SeaweedFS (bundled self-hosted) | Remote signing — Lakekeeper signs S3 requests on behalf of the engine. SeaweedFS has no STS service. |
| Custom S3-compatible store | Remote signing (same as SeaweedFS). Path-style access is configured automatically. |
DuckDB's Iceberg extension data-plane reads over remote-signed URLs (SeaweedFS / custom S3-compat stores) may not work in all DuckDB releases. Catalog metadata operations — SHOW TABLES, DESCRIBE — work in all cases. Check the compatibility matrix for the current test status.
Where to go next
Connect an engine
Mint an engine token and configure PyIceberg, DuckDB, Trino, or Spark.
BYO Iceberg catalog
Connect your existing Glue, Nessie, or REST catalog as a Scrydon data source.
Managed tables
How tables are created, governed, and evolved.
Classification & masking
Column masking and row filters — what the Iceberg data-plane does and does not enforce.