External write-in (Spark, Flink, Kafka)

Stream or batch data into a Scrydon managed table from Spark Structured Streaming, Flink, or the Kafka Connect Iceberg sink under an open-append write policy — and how the platform reconciles those commits under governance.

External engines can write into a Scrydon managed table, not just read from it. A Spark Structured Streaming job, a Flink pipeline, or a Kafka Connect Iceberg sink commits through the same Iceberg REST catalog facade you use for reads — Scrydon governs the commit after the fact rather than blocking it inline, so your pipeline keeps engine-native throughput.

This page covers streaming and batch write-in. For one-off reads and per-engine connection setup, see Connect an engine.

When to use write-in

Reach for external write-in when a high-volume producer already speaks Iceberg and you want it to land directly in a governed table:

Spark Structured Streaming / batch — micro-batch or bulk appends.
Apache Flink — continuous streaming appends.
Kafka Connect with the Iceberg sink connector — topic → table.

If instead you want each row synchronously classified and audited as it lands, use the governed data plane (the scrydon notebook SDK's tables.write, or POST /data/v1/tables/:ref/write) — not external write-in.

Prerequisites

A write-enabled engine token. Mint it from Platform → Settings → Engine tokens with write access, or via the API with canWrite: true. See Connect an engine → Mint an engine token.

The table's write policy set to Open append. New tables default to Governed only, which rejects every external commit (even from a write-enabled token). Open the table in Analytics → Settings → External write access and choose:

Policy	External engines may…
Open append (recommended for streaming)	Append new rows. Existing rows stay immutable.
Open full	Append, overwrite, and delete. Use only for fully trusted pipelines.

A write on a Governed only table returns 403 Forbidden from the facade even with a write-enabled token. If your job fails with 403, check the write policy first.

Configure the writer

Point the engine at your organisation's catalog exactly as for reads — the write path uses the same catalog URI, warehouse, and token. Set the write token in the engine's Iceberg REST catalog config.

spark = (
    SparkSession.builder
    .config("spark.sql.catalog.scrydon", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.scrydon.type", "rest")
    .config("spark.sql.catalog.scrydon.uri", "https://<your-scrydon-host>/api/table/iceberg")
    .config("spark.sql.catalog.scrydon.token", "<WRITE_ENGINE_TOKEN>")
    .getOrCreate()
)

# Micro-batch append into an open_append table.
(
    streaming_df.writeStream
    .format("iceberg")
    .outputMode("append")
    .option("path", "scrydon.<namespace>.<table>")
    .option("checkpointLocation", "/chk/scrydon-sink")
    .trigger(processingTime="30 seconds")
    .start()
)

CREATE CATALOG scrydon WITH (
  'type' = 'iceberg',
  'catalog-type' = 'rest',
  'uri' = 'https://<your-scrydon-host>/api/table/iceberg',
  'token' = '<WRITE_ENGINE_TOKEN>'
);

-- Streaming INSERT appends continuously into the open_append table.
INSERT INTO scrydon.`<namespace>`.`<table>`
SELECT * FROM source_stream;

{
  "name": "scrydon-iceberg-sink",
  "config": {
    "connector.class": "org.apache.iceberg.connect.IcebergSinkConnector",
    "topics": "events",
    "iceberg.catalog.type": "rest",
    "iceberg.catalog.uri": "https://<your-scrydon-host>/api/table/iceberg",
    "iceberg.catalog.token": "<WRITE_ENGINE_TOKEN>",
    "iceberg.tables": "<namespace>.<table>",
    "iceberg.tables.auto-create-enabled": "false"
  }
}

The catalog URI must not include a /v1 suffix — Iceberg REST clients append /v1/ themselves.

Object-store credentials on the write path. How your engine authenticates its data writes depends on the backing store. On AWS S3 and Azure ADLS, Scrydon vends short-lived STS/SAS credentials and your engine writes Parquet directly. On a self-hosted S3-compatible store (SeaweedFS, MinIO), writes go through Scrydon's REST remote signing — and whether a given engine activates the REST signer on its write path varies by engine and version (reads are broadly supported). If external appends land in the catalog but the data files don't materialise on a self-hosted store, this is the cause — see How governance holds under federation.

How commits are reconciled

External write-in is governed after the fact. Each external commit lands as a normal Iceberg snapshot; Scrydon's external-commit reconciler then, on its next pass (or on a commit event where configured):

Detects the new snapshot (drift against the last snapshot Scrydon recorded).
Re-counts rows under governance and refreshes the table's row_count.
Re-stamps governance properties (scrydon.classification, scrydon.write-policy, identity columns, column sensitivity) onto the table.
Emits a table.external_commit_reconciled audit event describing what the commit added (operation, rows, files, bytes).

So your appends are visible to Iceberg readers immediately, and become fully attributed in the governance and audit trail shortly after.

Conflict behaviour with platform upserts

Iceberg's optimistic-concurrency commit protocol arbitrates concurrent writers: each commit validates against current table metadata and retries on conflict. An external append (new files, no deletes) and a platform merge-on-read upsert (deletes + new files) touch disjoint metadata in the common case, so both commit; a genuine conflict makes the loser retry against the winner's snapshot. No commit is lost.

One rule keeps this safe: an external writer must not delete or overwrite rows a platform upsert owns. Mixed ownership of the same primary keys across an external pipeline and the governed upsert path is unsupported. Under open_append this is enforced by construction — appends never delete. Under open_full, keep external and platform writers on disjoint key ranges.

Reading the results back

Read written rows through the governed data plane to get per-column masking and clearance enforcement applied to the data itself:

The scrydon notebook SDK — tables.read(...).
POST /data/v1/tables/:ref/query — masked, keyset-paginated.
The Analytics table preview.

A raw external scan reads Parquet as stored (no per-column masks) and is clearance-gated at credential-vending time — see How governance holds under federation.