BYO Iceberg catalog
Connect an existing Iceberg catalog — AWS Glue, Nessie, or a REST catalog — as a Scrydon data source. Scrydon reads it as a single service principal using the credentials you supply.
BYO (Bring Your Own) catalog lets you connect an existing Iceberg catalog to Scrydon as a data source. Once connected, tables in that catalog appear alongside your Scrydon-managed tables in Analytics, and are queryable from workflows, notebooks, and the ontology layer.
Supported catalog types:
| Type | Example products |
|---|---|
rest | Any Iceberg REST catalog (Lakekeeper, Polaris, Nessie REST) |
glue | AWS Glue Data Catalog |
nessie | Project Nessie |
How it works — service-principal model
Scrydon connects to your catalog using a single service-principal credential that you supply. This means:
- Your catalog's governance is the outer boundary. Scrydon reads only what that credential is permitted to read. You control which tables and namespaces are visible to Scrydon by managing the credential's permissions in your catalog.
- Scrydon does not impersonate individual users. All reads from this catalog source within Scrydon use the same credential. Per-user attribution happens in the Scrydon audit log — who in Scrydon queried which table — but the request to your catalog always originates from the single service principal.
- Credentials are not stored in Scrydon's application database. They are encrypted and stored in the platform secrets layer. The analytics API never logs or surfaces raw credential values.
Named non-goal: per-user credential passthrough — where each Scrydon user's identity is forwarded to your catalog — is not supported in this version. All access uses the single service-principal credential you register.
Markings
When you connect a BYO catalog, you assign default markings to it. These markings are applied to every table read from that catalog within Scrydon's governance layer. For example, if you mark a catalog as confidential, all tables from it are treated as confidential in Scrydon's classification and access-control policies, regardless of how they are classified in the source catalog.
You can update markings after connection without re-connecting.
Connect a catalog
Navigate to Analytics → Data Sources and select the External Catalogs tab at the top of the page (the Data Sources tab beside it lists your pack-installed data sources).
Click Connect catalog to open the connection form.
| Field | Description |
|---|---|
| Name | A short identifier for this catalog within Scrydon (lowercase alphanumeric + _, max 27 characters). This becomes part of the internal catalog name — it cannot be changed after creation. |
| Catalog type | rest, glue, or nessie. |
| URI | The catalog's endpoint URL. |
| Warehouse | (Optional) Iceberg warehouse name, if your catalog requires it. |
| Default markings | One or more markings to apply to all tables from this catalog. |
| Credentials | Key-value pairs passed to the catalog as authentication properties. The exact keys depend on your catalog type. |
Click Test connection — Scrydon will attempt to list namespaces using the credentials you provided. On success, click Save to persist the catalog.
Common credential configurations
AWS Glue
aws.region = us-east-1
aws.access-key-id = AKIA...
aws.secret-access-key = ...Nessie (bearer token auth)
authentication.type = BEARER
authentication.token = <your-nessie-token>REST catalog (token)
token = <your-rest-catalog-token>Property keys and values must not contain double-quote (") or backslash (\) characters — these are rejected at connection time. Single quotes and parentheses are allowed.
Try it locally (development)
The Scrydon dev stack (docker-compose.analytics.yml) ships a local Lakekeeper REST catalog you can connect to without any external account — useful for exercising the BYO path end to end:
| Field | Value |
|---|---|
| Catalog type | rest |
| URI | http://localhost:8282/catalog |
| Warehouse | the warehouse you bootstrapped in Lakekeeper (e.g. demo) |
| Credentials | none required — the dev Lakekeeper runs with authz-backend=allowall |
This local catalog is for development only; it carries no authentication and must never be used in production.
Querying tables from a BYO catalog
Once connected, tables from your catalog appear in Analytics under the catalog name you chose. You can query them from:
- Workflows — using the standard
scrydon:tablestools (same interface as managed tables). - Notebooks — the catalog's tables are available in the notebook's table picker.
- The ontology layer — you can bind object types to BYO catalog tables, enabling ontology queries over your external data.
All reads go through Scrydon's governance layer: markings are enforced, access is logged, and column masking from Scrydon's classification rules is applied if you have them configured.
Per-user attribution
Even though all requests to your catalog use the single service-principal credential, Scrydon's own audit log attributes each query to the Scrydon user who initiated it. This means you can answer "which Scrydon user read table X from the external catalog at time T" from Scrydon's audit trail.
Attribution inside your catalog's audit log will show the single service-principal identity, not individual Scrydon users.
Removing a catalog
You can remove a BYO catalog connection from Analytics → Data sources → [catalog name] → Remove. Removing a connection does not delete the data in your external catalog — it only removes Scrydon's connection record. Scrydon workflow steps or ontology bindings that reference the catalog's tables will fail after removal.