Connect an engine

External data engines — Spark, Trino, DuckDB, PyIceberg, and any other Iceberg REST-compatible client — connect to the Scrydon catalog facade at:

https://<your-host>/api/table/iceberg

Step 1 — Mint an engine token

Engine tokens are organisation-scoped credentials a query engine uses to authenticate to the catalog. They carry an optional write flag; read-only tokens cannot perform POST, PUT, DELETE, or PATCH operations against the catalog. The organisation binding is set by the server from your session — you cannot mint a token for an organisation you are not a member of, and write tokens require organisation owner or admin.

From the UI (recommended)

You can mint and manage engine tokens without leaving the app:

• Per table — open any Iceberg table in Analytics and go to its Query tab. Toggle write access if needed, click Generate engine token, and the connection snippets below it are filled with your token, catalog URI, namespace, and table name automatically.
• Org-wide — go to Platform → Settings → Engine tokens (organisation owner/admin) to create, list, and revoke every engine token for the organisation in one place.

The token is shown once — copy it immediately and treat it like a password: it grants catalog-level access to your organisation's Iceberg tables.

From the API

An organisation admin can also mint a token directly against the platform engine-token endpoint:

curl -X POST "https://<your-scrydon-host>/api/auth/iceberg/engine-token" \
  -H "Authorization: Bearer <your-platform-session>" \
  -H "content-type: application/json" \
  -d '{"name":"spark-etl","organizationId":"<your-org-id>","canWrite":true}'
# → { "key": "...", "organizationId": "...", "canWrite": true }

Step 2 — Configure your engine

All engines authenticate using the Bearer token scheme (Authorization: Bearer <engine-token>) or the token configuration property below.

The catalog URI below is your Scrydon host's api-table base URL. In most deployments this is the same host you use for the rest of the API:

https://<your-host>/api/table/iceberg

For local development the default is http://localhost:7500/api/table/iceberg.

from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(
    "scrydon",
    uri="https://<your-host>/api/table/iceberg",
    token="YOUR_ENGINE_TOKEN",
)

# List namespaces
print(catalog.list_namespaces())

# Load a table
tbl = catalog.load_table("my_namespace.my_table")
df = tbl.scan().to_arrow()

-- Install and load the iceberg extension (>=1.0.0 required for REST catalog support)
INSTALL iceberg;
LOAD iceberg;

-- Bearer auth for the Scrydon catalog (OR REPLACE so re-runs don't error)
CREATE OR REPLACE SECRET scrydon_token (TYPE ICEBERG, TOKEN 'YOUR_ENGINE_TOKEN');

-- Attach: the catalog URL is the ENDPOINT option, NOT the ATTACH string.
-- The ATTACH string is the WAREHOUSE — it must be non-empty (your org
-- warehouse, the `prefix` returned by GET /v1/config, i.e. org-<your-org-id>)
-- or DuckDB attaches but lists zero tables.
ATTACH IF NOT EXISTS 'org-<your-org-id>' AS scrydon (
  TYPE ICEBERG,
  SECRET scrydon_token,
  ENDPOINT 'https://<your-host>/api/table/iceberg'
);

-- List tables
SHOW ALL TABLES;

-- Query a table
SELECT * FROM scrydon.my_namespace.my_table LIMIT 10;

-- Time-travel: read a specific snapshot
SELECT * FROM scrydon.my_namespace.my_table AT (VERSION => 1234567890);

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://<your-host>/api/table/iceberg
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.token=YOUR_ENGINE_TOKEN

-- List schemas (namespaces)
SHOW SCHEMAS FROM scrydon;

-- Query a table
SELECT * FROM scrydon.my_namespace.my_table LIMIT 10;

-- Time-travel
SELECT * FROM scrydon.my_namespace.my_table FOR VERSION AS OF 1234567890;

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("scrydon-iceberg")
    .config(
        "spark.sql.extensions",
        "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
    )
    .config("spark.sql.catalog.scrydon", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.scrydon.type", "rest")
    .config(
        "spark.sql.catalog.scrydon.uri",
        "https://<your-host>/api/table/iceberg",
    )
    .config("spark.sql.catalog.scrydon.token", "YOUR_ENGINE_TOKEN")
    # Warehouse hint for /config requests; Scrydon overrides this server-side.
    .config("spark.sql.catalog.scrydon.warehouse", "scrydon")
    .getOrCreate()
)

# List namespaces
spark.sql("SHOW NAMESPACES IN scrydon").show()

# Query a table
spark.sql("SELECT * FROM scrydon.my_namespace.my_table LIMIT 10").show()

# Time-travel
spark.sql(
    "SELECT * FROM scrydon.my_namespace.my_table VERSION AS OF 1234567890"
).show()

org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:<version>

-- Step 1: Create a catalog integration pointing at the Scrydon REST facade.
CREATE OR REPLACE CATALOG INTEGRATION scrydon_catalog
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = 'my_namespace'
  REST_CONFIG = (
    CATALOG_URI = 'https://<your-host>/api/table/iceberg'
    WAREHOUSE   = 'org-<your-org-id>'  -- prefix returned by GET /v1/config
  )
  REST_AUTHENTICATION = (
    TYPE         = BEARER
    BEARER_TOKEN = 'YOUR_READ_ENGINE_TOKEN'
  )
  ENABLED = TRUE;

-- Verify the integration can reach the catalog.
DESCRIBE INTEGRATION scrydon_catalog;

-- Step 2: Create an external volume with direct read access to your
-- org's S3 prefix. The IAM role must have s3:GetObject on the bucket.
CREATE EXTERNAL VOLUME scrydon_vol
  STORAGE_LOCATIONS = ((
    NAME                 = 'scrydon-s3',
    STORAGE_PROVIDER     = 'S3',
    STORAGE_BASE_URL     = 's3://<your-scrydon-bucket>/org-<your-org-id>/',
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::<account-id>:role/<role-name>'
  ));

-- Step 3: Register a table backed by the Scrydon catalog.
CREATE ICEBERG TABLE my_table
  CATALOG           = 'scrydon_catalog'
  EXTERNAL_VOLUME   = 'scrydon_vol'
  CATALOG_TABLE_NAME = 'my_table'
  CATALOG_NAMESPACE  = 'my_namespace';

-- Verify: read the first ten rows.
SELECT * FROM my_table LIMIT 10;

org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.1

# Configure the Scrydon REST catalog for this Spark session.
spark.conf.set(
    "spark.sql.catalog.scrydon",
    "org.apache.iceberg.spark.SparkCatalog",
)
spark.conf.set("spark.sql.catalog.scrydon.type",  "rest")
spark.conf.set(
    "spark.sql.catalog.scrydon.uri",
    "https://<your-host>/api/table/iceberg",
)
spark.conf.set("spark.sql.catalog.scrydon.token", "YOUR_ENGINE_TOKEN")

# Verify: list available namespaces.
display(spark.sql("SHOW NAMESPACES IN scrydon"))

# Query a table.
display(spark.sql("SELECT * FROM scrydon.my_namespace.my_table LIMIT 10"))

# Time-travel to a specific snapshot.
display(spark.sql(
    "SELECT * FROM scrydon.my_namespace.my_table VERSION AS OF <snapshot_id>"
))

-- Create a foreign catalog backed by the Scrydon REST facade.
-- Requires Unity Catalog and Databricks Premium or Enterprise.
CREATE CATALOG scrydon
USING ICEBERG
OPTIONS (
  'rest.uri' = 'https://<your-host>/api/table/iceberg',
  'token'    = 'YOUR_ENGINE_TOKEN'
);

-- Grant access to workspace members (optional).
GRANT USE CATALOG ON CATALOG scrydon TO `user@example.com`;

-- Query tables.
SELECT * FROM scrydon.my_namespace.my_table LIMIT 10;

-- Time-travel.
SELECT * FROM scrydon.my_namespace.my_table VERSION AS OF <snapshot_id>;

What the catalog endpoint returns

On GET /api/table/iceberg/v1/config the facade returns your organisation's warehouse prefix. This is how Iceberg clients discover the correct warehouse without you needing to configure it manually. Most clients call this automatically.

Write access

An external-engine write passes two independent gates — both must allow it:

1. Engine-token write flag. The token must be minted with write access (canWrite: true). Read-only tokens receive 403 Forbidden on any POST, PUT, DELETE, or PATCH.
2. Per-table write policy. Each managed table carries a write policy that governs whether any external engine may commit to it — independent of the token. New tables default to Governed only, which rejects every external commit (even from a write-enabled token) so writes stay on the governed data plane.

Changing a table's write policy

Open the table in Analytics → Settings tab → External write access, then pick a Write policy:

Scrydon's internal StarRocks engine is the only engine that writes Parquet data files directly. External engine writes (via the REST catalog) write table metadata (schemas, snapshots, namespace entries) through the governed facade; the actual data-plane writes go directly to your object store via Lakekeeper-vended or remote-signed credentials.

How governance holds under federation

An external engine connects through two independent credential surfaces, and Scrydon governs each:

1. Catalog metadata (namespace / table / schema / snapshot listings) — authenticated by your engine token, organisation-scoped: a token never sees another org's tables (404 cross-org).
2. Data-plane bytes (the Parquet files) — vended or remote-signed per request by the facade, never handed out wholesale.

A raw external scan reads Parquet as stored, so per-column masks — which Scrydon applies as a read-time transform — are not baked into those bytes. Governance therefore holds a different way at each surface.

Reads — clearance-gated raw access (not per-column masks)

Before vending data-plane credentials for a table, the facade's raw-scan vending gate resolves the engine-token user's effective clearance. If any column's classification exceeds that clearance, raw-scan vending is denied — a user who could only see a column masked in the governed path cannot obtain raw credentials to read it unmasked. This is mandatory access control over discretionary sharing: it denies even org admins who lack the clearance.

• Need per-column masking applied to the data itself? Read through the governed data plane — the scrydon notebook SDK, the Analytics UI, or POST /data/v1/tables/:ref/query — which runs the masked-select kernel and returns masked cells plus a maskedColumns list.
• Direct-S3 engines (Snowflake external volume, Databricks STS) read bytes with their own cloud credentials, outside Scrydon's signer. Scrydon cannot mask or clearance-gate those reads — apply the engine's own row/column policies, and grant those integrations tokens only for tables whose sensitivity you accept sharing wholesale.

Writes — the per-table write policy

Every managed table carries a write policy (see Write access above): governed_only (default) rejects all external commits at the facade; open_append / open_full admit them and the platform reconciles the new snapshot after the fact (audit event, row-count refresh, governance re-stamp). A write-enabled engine token is still 403'd on a governed_only table.

Governance metadata travels with the table

Scrydon stamps governance onto the Iceberg table itself as properties — scrydon.classification, scrydon.write-policy, scrydon.identity-columns, scrydon.column-sensitivity — so any catalog-aware tool inspecting the table sees its data classification and identity columns without calling a Scrydon API.

Compatibility matrix

CI-verified engines are tested against Scrydon's REST facade on every pull request that touches the Iceberg surface (workflow test-iceberg-compat.yml). Community-verified engines have been validated manually but are not yet part of the automated matrix.

Catalog operations (namespace / table / schema / snapshot listing) are the always-verified contract for CI-tested engines. Data-plane reads depend on your storage backend (see footnote 1 below).

Writing in from a streaming engine

To stream or batch data into a managed table from Spark Structured Streaming, Flink, or the Kafka Connect Iceberg sink, see External write-in. It commits through this same catalog facade under an open_append write policy, and Scrydon reconciles the commits under governance after the fact.

Roadmap

Two capabilities are designed and gated on upstream engine support — they are not available yet:

• Interactive Arrow-speed reads (Arrow Flight SQL). A brokered StarRocks Arrow Flight SQL endpoint behind token auth with masked-query handles, so interactive reads reach engine-native Arrow throughput end to end without exposing raw governed SQL or StarRocks credentials. Gated on the Flight SQL endpoint being provisioned in the deployment.
• Iceberg format v3. Deletion vectors (faster merge-on-read) and row lineage (stronger commit-audit provenance). Gated on the compatibility matrix — both StarRocks and PyIceberg must read and write v3 before Scrydon writes any v3 table.

Rotating or revoking an engine token

Tokens can be revoked at any time from Platform → Settings → Engine tokens (organisation owner/admin). Revocation takes effect immediately; existing connections using the token will receive 401 Unauthorized on the next request.