Data Sources

A Declarative Data Source is a poll-mode table source encoded entirely as JSON-serializable configuration — an HTTP request spec, an itemsPath selector, a field-mapping table, and a column list. You author it once with defineDataSource; the platform's generic poll runtime drives it on every tick without executing any customer-supplied code.

An organization-scoped poll, stream or push source may also declare trackProjection to send complete, committed rows to a governed ontology correlated_tracks stream. It names ontology, streamId, the immutable policyRevisionId, a marking, and columns for eventId, sourceEntityId, observedAt, latitude, and longitude. Optional column mappings are commonIdentifier, heading, speed, and quality. Every mapped column must be declared in table.columns. A row whose latitude and longitude are both null reports no position: it is skipped, never sent and never a failure. Only a table that declares both columns nullable can hold such a row; a row with one of the two missing fails the projection. An on-demand, manual, query or projection source that declares trackProjection is refused when the manifest is validated: a projection submits nothing of its own, and an on-demand fetch and a manual upload are not live tracks. Platform submits a poll run's rows once they are written, whenever the run had no fetch or write error, even when it rejected other rows, and a stream or push batch's rows once the batch is committed. A failed submit never rolls back written rows. A poll run records it as the run's failure. A stream or push batch stays committed and its positions are not sent again; a push source shows the failure as a warning on its sync, and a stream source as a warning on the sync of the 30-second window the batch was committed in. The installed source ID and workspace environment come from the platform, not provider rows. The ontology must already declare a matching private track policy that admits this source ID. Source health on the track stream requires the matching installed configuration and a complete poll run, stream window or push batch. A stream window is complete when its connection did not end or go silent, nothing was set aside, the data policy kept no row out, and no track position failed to submit. A push batch is complete when the data policy kept no row out and every track position was submitted; its evidence lasts two stream periods, after which a push source that has had no delivery reads as unknown, not degraded. Track freshness is a separate state.

For a stream source, the same projection also drives a best-effort live tick between its batches, roughly every 250 ms, independent of the batch's own commit (written to the table about once a minute). The map shows what was received: a position offered by a tick may belong to a batch whose commit is later refused, in which case the table never carries it even though it was briefly shown live.

Every position carries the projection's marking. A row that carries row markings (a non-empty markings column) is never sent to the track stream: the table shows it only to readers who hold its markings, and a track stream cannot enforce them. A poll or push source's sync warning counts them, and so does a stream source's window warning.

For a normalized AIS table in the MIK example, add this block to a source whose id is fixture-ais and whose table.columns declare these names:

trackProjection: {
  ontology: "scrydon-pack-mik-north-sea-v2",
  streamId: "north-sea-v3",
  policyRevisionId: "mik-fixture-v3",
  marking: { schemeId: "scrydon.default", rank: 0 },
  columns: {
    eventId: "event_id",
    sourceEntityId: "mmsi",
    observedAt: "observed_at",
    latitude: "latitude",
    longitude: "longitude",
  },
}

The HTTP mapping must supply a stable event_id for each observation; reusing an event ID with changed coordinates is rejected. The MIK example's synthetic replay is a test fixture, not a configured HTTP feed.

A row carrying row markings (the table's markings column) is never projected, even when the source is otherwise healthy: a track stream governs reads by one scheme and rank, with no slug of its own, so a marked row published at the projection's fixed marking would reach a reader the table itself hides that row from. Such rows are set aside and counted in the poll run's sync warning; an unmarked row (no markings, or an empty one) projects exactly as described above.

When to use this SDK

Use the Data Source Authoring SDK when:

• You want to pull rows from a public REST API on a schedule and make them available as a typed table inside Scrydon.
• You need the source definition to live in an Extension so re-uploading the extension also refreshes the source (same idempotency contract as ontologies and process flows).
• You are building a demo or starter Extension and want the data tables to come pre-wired with no manual configuration.

Do not use this SDK if:

• Your ingest needs state carried from one run to the next, or code. A declarative source sends the same request every time: it has no pagination or since_id cursor (the X API's incremental reads need one), no conditional logic, and no custom request signing. An extension cannot ship code instead — the bundler strips a produce() function, below. For a feed that only offers a cursor, poll its latest page and let upsert on the item's id absorb the overlap; anything published faster than one page per interval is missed. See What a declarative source cannot do yet. OAuth is not a reason to reach for code: the platform refreshes the access token before every tick (OAuth Connections).
• The destination is not a table. Non-tabular ingestion paths are out of scope for this SDK today.

Key constraint: extensions carry pure JSON — never functions. The bundler strips any produce() functions during serialization. An extensionable source must therefore be fully declarative — every field in the source definition must survive JSON.parse(JSON.stringify(...)) round-trip intact.

Install

bun add -d @scrydon/sdk-authoring zod

npm install --save-dev @scrydon/sdk-authoring zod

import { defineDataSource } from '@scrydon/sdk-authoring/extensions'

Anatomy of a declarative data source

A declarative data source is made up of five blocks:

At execution time, the runtime resolves and pins the request host to validated public DNS answers. It rejects loopback, private, reserved, link-local, cloud-metadata, mixed public/private, and DNS-rebinding targets; HTTP redirects are not followed. Response reads are bounded by the manifest's byte limit even when the server omits Content-Length, and the request timeout cancels the active stream.

Response formats: JSON, XML, CSV, and JSONL

A poll source is not limited to JSON. The runtime picks how to read the response body from three signals, in this order:

1. The response Content-Type. text/csv and application/csv → CSV. application/x-ndjson and application/jsonl → JSONL. application/json, and any structured-suffix type such as application/geo+json, → JSON. application/xml, text/xml, and any +xml structured-suffix type — which is what makes application/rss+xml and application/atom+xml work with no feed-specific configuration — → XML.
2. The extension on the request URL's path — .csv, .jsonl or .ndjson, .json, .xml or .rss or .atom.
3. The first non-whitespace byte — [ or { is read as JSON, < as XML, anything else as CSV.

response.format defaults to 'auto', which is exactly the signal chain above — a source that already serves a correct Content-Type needs no extra configuration. Naming a format ('csv', 'json', 'jsonl', 'xml') pins the decode and outranks every signal, which is what rescues a vendor that mislabels its own payload. See HTTP Requests.

A response that declares Content-Type: text/html is a web page, not data, and under 'auto' the tick fails with unexpected content type text/html instead of reading it. Some APIs answer an error this way with a 200 status — GDELT sends Timespan is too short. as text/html — and read as CSV that prose would be an empty result nobody notices. If a vendor genuinely serves its data labelled as HTML, name the format.

Whatever the format, the body decodes into a document and itemsPath selects the row entries out of it. What differs is only where those entries sit.

A CSV or JSONL body decodes into a flat array of row objects, and that array is the document — there is nothing nested to select inside it. Such a source declares itemsPath: '$':

ingest: {
  mode: 'poll',
  intervalSec: 3600,
  request: { url: 'https://example.org/exports/airfields.csv' },
  response: { itemsPath: '$' },
  mapping: {
    icaoCode: { path: 'icao' },
    name: { path: 'name', transform: 'trim_to_null' },
    elevationFt: { path: 'elevation_ft', transform: 'number_or_null' },
  },
}

For CSV, the first line is the header row and its cells become the field names your mapping paths resolve against. Quoted fields may contain commas and doubled quotes (""). Every cell arrives as a string — CSV has no null and no numbers — so use parse_number for numeric columns and one of the date transforms for timestamps. Remember that a blank cell is '', which filter.requireNonNull does not drop. A quoted field may not span a newline.

For JSONL, every non-blank line must be a JSON object; blank lines are skipped, and a malformed line fails the tick with its line number.

XML, RSS, and Atom feeds

'xml' is one format covering every XML dialect. There is no 'rss' or 'atom' value, because the parsing is identical in all three cases — the only thing that differs between them is where the entries sit, and itemsPath already says that. For RSS 2.0 they sit at rss.channel.item; for Atom at feed.entry.

A news feed therefore looks like any other source with a nested envelope:

ingest: {
  mode: 'poll',
  intervalSec: 900,
  request: { url: 'https://gcaptain.com/feed/' },   // served as application/rss+xml
  response: { itemsPath: 'rss.channel.item' },      // ← the RSS entry path
  mapping: {
    headline: { path: 'title' },
    url: { path: 'link' },
    externalId: { path: 'guid.#text' },             // <guid isPermaLink="false">…</guid>
    publishedAt: { path: 'pubDate', transform: 'iso_from_rfc822' },   // ← RFC-822, the RSS standard
    author: { path: 'dc:creator' },                 // namespace prefix kept verbatim
  },
}

The same source against an Atom feed changes exactly one line:

response: { itemsPath: 'feed.entry' },
mapping: {
  headline: { path: 'title' },
  externalId: { path: 'id' },
  publishedAt: { path: 'updated' },
  url: { path: 'link.@href' },                      // <link rel="alternate" href="…"/>
},

Three properties of the decoded document drive how you write those mapping paths:

• An attribute becomes @name. <link href="…"/> is { '@href': '…' }, so the path is link.@href.
• An element with attributes and text becomes { '@name': …, '#text': … }, while a bare element is just its string. That is why guid above is guid.#text (it carries isPermaLink) but title is plain title.
• A namespace prefix is kept exactly as written — dc:creator, media:content. There is no namespace resolution, and xmlns declarations arrive as ordinary attributes.
• An element becomes a list only when it repeats. An item with one <category> decodes to "Ports", one with three to ["Defense", "Ports", "Shipping"]. An array column refuses the single string, so map a repeatable element with as_array, which always yields a list: categories: { path: 'category', transform: 'as_array' } into { name: 'categories', dataType: 'array', nullable: true }.

Every leaf is a string; nothing is coerced. Use parse_number for numeric columns, and a date transform for timestamps — mapping a feed's date string straight through does not work for a timestamp column, because the row validator requires an RFC 3339 instant (…Z or …±hh:mm) and no feed format is one (see Dates from feed formats).

The mapping DSL

Each mapping entry either copies a path directly or applies a named transform. The transform set is bounded and auditable — adding a new transform requires a reviewed code change to the platform, not extension data. Arbitrary expressions, eval, and sandboxed code are intentionally not supported.

parse_number, iso_from_rfc822, iso_from_basic, iso_from_go_time and iso_from_epoch convert text (parse_number and iso_from_epoch take a number as well); trim_to_null, number_or_null, value_map and iso_from_epoch_offset read values that are already typed; as_array fixes a value's shape. The text-versus-typed distinction is the single most important thing on this page, so it has its own section.

parse_number vs number_or_null

They look interchangeable and are not. number_or_null requires the value to already be a number — it tests typeof value === 'number' and nulls everything else. A CSV or XML row codec yields a string for every cell, so on those formats number_or_null returns null for every numeric column, a nullable column accepts every hole, and the tick reports success having discarded the data.

That is not hypothetical. A day's GPS-interference grid (GPSJam, 48,461 rows) once landed with every numeric column empty and a green result, because its manifest said number_or_null on a CSV source. A poll could not take that file today anyway: a response that maps to more than 10,000 rows is refused whole, and the file's URL changes every day. It goes in through a manual source, which takes up to 50,000 rows — with parse_number on its numeric columns.

parse_number also draws a line number_or_null does not:

• an empty cell ('', whitespace, a missing element) is a genuine no value → null;
• a value it cannot convert ('n/a', '1,420', '12px', Infinity, NaN, 0x10) fails that row. The row is counted in the tick's rowsRejected, the first one is logged with its column and validation code (never the row or its value), and every other row in the tick still lands.

The second rule is deliberate. Silently writing null for a value the feed clearly did send is how a whole signal disappears behind a green tick; losing one row loudly is strictly better, and losing the whole tick to one bad cell would be strictly worse.

value_map carries the same JSON-only assumption in its passthrough: "number" option, which never fires on CSV or XML. On those formats list the string forms in map (the lookup runs on the string form, so "12000" matches) or use passthrough: "string".

Dates from feed formats

A date or timestamp column is validated as an RFC 3339 instant — 2026-09-16T13:08:53Z or 2026-09-16T13:08:53+02:00, with optional fractional seconds — and stored in UTC: an offset value is converted to its Z form (+02:00 → 11:08:53Z), keeping its fractional digits, so a feed that spells UTC as +00:00 lands exactly as one that writes Z. Most other date formats a real feed emits do not qualify:

iso_from_rfc822 and iso_from_basic return midnight UTC for a date with no time, and read a datetime with no zone designator as UTC. Every date transform refuses rather than guesses — a refusal rejects the row, exactly as parse_number does, so you never get a timestamp that is silently hours wrong.

iso_from_go_time reads the layout Go's time.Time prints: the numeric offset (+0000, -0500) is required and is the only zone it reads; the zone abbreviation after it (UTC, CEST) must be present but is never interpreted. Go drops trailing zeros from the fraction, so 0 to 9 digits are accepted, and they are kept verbatim. A trailing monotonic-clock reading ( m=+0.000123456) is dropped. A reading with no offset, no abbreviation, or more than nine fraction digits is refused.

iso_from_epoch has no default unit, because 1789671802 read as milliseconds is a date in January 1970 that every column accepts. Milliseconds must be a whole number — a fraction there is almost always seconds declared as milliseconds — and seconds may carry a fraction, of which digits past the millisecond are dropped. An instant outside the years 0000–9999, which is where milliseconds declared as seconds land, is refused.

What they deliberately do not parse:

• two-digit years (16 Sep 26) — RFC-822 allowed them; the century is a guess;
• obsolete zone abbreviations (EST, PDT, CEST) — ambiguous in practice, and RFC 2822 §4.3 says to treat them as unknown. GMT, UT, UTC and Z are accepted;
• non-English or full-word month names (septembre, September) — three-letter English only;
• impossible dates (31 Sep) and out-of-range times, which a naive parser silently rolls into the next month;
• ISO week and ordinal dates (2026-W37-5, 2026-254);
• the 14-digit run some feeds emit (20260911054500) — it is not ISO 8601, so it is refused rather than guessed.

The day name in an RFC-822 date is not checked against the date. RFC 2822 says it is not authoritative and plenty of feeds get it wrong; rejecting on it would drop real rows to enforce redundancy.

The iso_from_epoch_offset hard case

Some REST APIs — the ADS-B family is the canonical example — do not return per-row absolute timestamps. Instead, the envelope carries a single now field (epoch milliseconds) and each row carries a seen field (seconds ago). The iso_from_epoch_offset transform bridges the two:

seenAt = new Date(envelope.now - row.seen * 1000).toISOString()

Configure it as:

seenAt: {
  path: "seen",                              // per-row offset field
  transform: "iso_from_epoch_offset",
  args: {
    basePath: "$.now",                       // envelope field — resolved against the response root
    baseUnit: "ms",                          // $.now is epoch milliseconds
    offsetUnit: "s",                         // row.seen is seconds
    direction: "subtract",                   // now − seen → absolute time
  },
},

If row.seen is missing or non-numeric, the runtime defaults the offset to 0 (base time exactly), matching the code-source convention seenSecondsAgo = typeof raw.seen === "number" ? raw.seen : 0.

A complete example

The following is the real adsb-lol-military-declarative source shipped in @scrydon/sdk-authoring. It pulls military aircraft positions from https://api.adsb.lol/v2/mil and maps them to a typed aircraft_position table. The golden parity test in the monorepo asserts this produces byte-for-byte identical rows to the equivalent code source.

import { defineDataSource } from '@scrydon/sdk-authoring/extensions'

export const adsbLolMilitaryDeclarative = defineDataSource({
  kind: "table",
  id: "adsb-lol-military-declarative",
  vendor: "adsb-lol",
  displayName: "ADS-B Lol — Military Aircraft (declarative)",
  scope: "global",
  table: {
    name: "aircraft_position",
    primaryKey: ["icao24", "seenAt"],
    timestampColumn: "seenAt",
    columns: [
      { name: "icao24",           dataType: "string",  isPrimaryKey: true },
      { name: "callsign",         dataType: "string",  nullable: true },
      { name: "registration",     dataType: "string",  nullable: true },
      { name: "aircraftType",     dataType: "string",  nullable: true },
      { name: "category",         dataType: "string",  nullable: true },
      { name: "latitude",         dataType: "decimal" },
      { name: "longitude",        dataType: "decimal" },
      { name: "altitudeFeet",     dataType: "int",     nullable: true },
      { name: "groundSpeedKnots", dataType: "double",  nullable: true },
      { name: "heading",          dataType: "double",  nullable: true },
      { name: "seenAt",           dataType: "timestamp", isPrimaryKey: true },
    ],
  },
  ingest: {
    mode: "poll",
    intervalSec: 60,
    minIntervalSec: 30,
    request: {
      url: "https://api.adsb.lol/v2/mil",
      method: "GET",
      headers: { accept: "application/json" },
    },
    response: { itemsPath: "$.ac" },
    filter: {
      // Drop rows missing hex (ICAO), lat, or lon — mirrors the code source guard.
      requireNonNull: ["hex", "lat", "lon"],
    },
    mapping: {
      icao24:           { path: "hex" },
      // "RCH123 " → "RCH123", "   " → null. Mirrors raw.flight?.trim() || null
      callsign:         { path: "flight", transform: "trim_to_null" },
      // undefined → null. Mirrors raw.r ?? null
      registration:     { path: "r" },
      // undefined → null. Mirrors raw.t ?? null
      aircraftType:     { path: "t" },
      // undefined → null. Mirrors raw.category ?? null
      category:         { path: "category" },
      latitude:         { path: "lat" },
      longitude:        { path: "lon" },
      // "ground" → 0, number → passthrough, else → null. Mirrors parseAltitude()
      altitudeFeet: {
        path: "alt_baro",
        transform: "value_map",
        args: { map: { ground: 0 }, passthrough: "number", default: null },
      },
      // finite number → itself, string/NaN/undefined → null
      groundSpeedKnots: { path: "gs",    transform: "number_or_null" },
      heading:          { path: "track", transform: "number_or_null" },
      // new Date(envelope.now - row.seen * 1000).toISOString()
      seenAt: {
        path: "seen",
        transform: "iso_from_epoch_offset",
        args: {
          basePath: "$.now",
          baseUnit: "ms",
          offsetUnit: "s",
          direction: "subtract",
        },
      },
    },
  },
})

defineDataSource is an identity function at runtime — it validates the manifest via the DataSourceManifestSchema Zod schema and derives a row validator from table.columns. The emitted definition is pure data: no produce() function, no closures, serializable as-is into data-source-<id>/manifest.json.

Table keys

A key column is any column that table.primaryKey names or that isPrimaryKey: true flags. Keep the two in agreement, as the example above does. The platform creates the table with these columns as its primary key, and a table key can only use these types:

A manifest with a key column of the second kind is refused before anything is created: by defineDataSource(), extension build, extension validate, the upload inspector and install. The message names the column and says what to do:

data-source-observations/manifest.json validation failed at "table.columns.1.dataType":
Key column "latitude" has dataType "double", which a table key cannot use. Key columns must be
one of: string, int, bigint, boolean, date, timestamp, geo_point. Use a string or integer key,
or keep "latitude" as a normal column: remove it from table.primaryKey and drop its
isPrimaryKey flag.

Measurements such as coordinates are rarely a good identity anyway: two readings of the same thing seldom agree to the last digit. Key the table on what identifies the row, such as a provider ID or an observation time, and keep latitude and longitude as normal double or decimal columns. If the API gives no identifier, build one as a string column.

The markings column

markings is the column every governed read filters rows on: the marking slugs a reader must hold to see the row. You do not have to declare it — the platform adds an empty one to every table a data source writes. Declare it only when your source marks its own rows, and then declare it as { name: "markings", dataType: "array", nullable: true }. Any other type is refused, by the same checks as a key column, because a governed read could not filter by it and every ontology read of the table would fail. The name is matched without regard to case, so Markings is the same column:

data-source-observations/manifest.json validation failed at "table.columns.2.dataType":
Column "markings" is the row-markings column every governed read filters on, so its dataType
must be "array" (the marking slugs a reader must hold to see the row), not "string". Declare it
as { dataType: "array" }, or give the column another name.

If the API returns a field called markings that is not a list of marking slugs, map it to a column with another name.

Write modes

Stored-table projections

A projection source combines columns from existing managed tables on a schedule. It uses mode: "poll", a positive intervalSec, optional minIntervalSec, and the required writeMode: "replace". API Table executes this variant; the extension HTTP poll runtime does not. A projection is a bounded union of 1–32 inputs, with distinct source IDs and literal namespace keys:

ingest: {
  mode: "poll", intervalSec: 300, writeMode: "replace",
  projection: {
    inputKeyColumn: "provider",
    inputs: [{
      sourceId: "tfl-jamcam", key: "tfl",
      mapping: { cameraId: "id", region: null },
    }],
  },
}

The namespace column must be a nonnullable string in table.primaryKey, together with at least one mapped identity column. Set isPrimaryKey: true on exactly those columns listed in table.primaryKey; the host uses those flags when provisioning the table. Each input must map every other author-owned output column to an exact stored source column name, or explicitly to null only for a nullable nonkey output column. Source IDs use kebab-case; column names use safe SQL identifiers other than __proto__, constructor, and prototype, and must stay unique when compared case-insensitively (the warehouse treats region and Region as the same column). Paths, transforms, SQL, inline credentials, and arbitrary constants are not accepted.

Some target columns belong to the host, not to you. Declare each of them exactly as listed, with an explicit nullable: false (an omitted flag would provision a nullable column), outside table.primaryKey; they cannot be mapped:

markings is the governance-marking column the ontology reads, which is what lets a projection output be bound as an ontology source. Each output row is written with the markings of the one input row it was projected from, so a marked input row stays marked in the output and is readable by exactly the people who can read the input row. An input with no markings column contributes unmarked rows, written as an empty array. To change who can read an output row, mark the input row: the next refresh rewrites the output row's markings from it. classificationRank cannot be declared at all. An input's markings or classificationRank column cannot be used as a mapping source either — that would turn the input's access metadata into ordinary data on the output.

An input is unhealthy for a refresh in either of two cases:

• Its latest poll is not a current, complete answer. The input has never synced, its last poll failed, a poll has failed since, or its last successful sync is older than twice its intervalSec (and older than one minute).
• Its current table holds a row the output cannot hold. The row's key is blank, or cannot be read exactly (for example an integer beyond the exact range). A value cannot be converted exactly to its output column's type, such as a timestamp for a date that does not exist. The row's output identity is blank or repeats another row of the same input. Or the output's row validation refuses it. One such row makes the whole input unhealthy: publishing the input without that row would make the entity it describes disappear from the output.

The refresh does not wait for an unhealthy input. The other inputs refresh normally. The unhealthy input is read again at the last snapshot of its own table that the output recorded, and mapped with the current mapping. Its rows keep that snapshot's sourceSnapshotId and sourceSyncedAt, so sourceSyncedAt shows how old they are. Those rows pass the same checks as a fresh read: if one of them is refused as well, the refresh fails. Values stored in the output table itself are never republished. Which snapshot is read again is taken from the output table's sourceSnapshotId. So anyone allowed to write to the output table can stop the input from being carried forward, or choose which of the snapshots its table still holds is read again. The rows are then still the input's own rows, stamped with their real snapshot and time. There is no expiry: the input stays carried forward, and the age its sourceSyncedAt states keeps growing, until it is healthy again. Each refresh that carries an input forward is logged as a warning that names the input and the reason. When the reason is a refused row, only that warning states it: the input's own sync status still shows its successful poll.

An unhealthy input is never published as zero rows. If there is no recorded snapshot to read it at, the refresh fails and the previous output stays. That happens on the first refresh after install, after the input's key changed or its table was re-created, when the input's last published contribution was empty (an empty contribution records no snapshot), when its rows in the output name more than one snapshot or a snapshot ID that cannot be read, when that snapshot has expired from the input table, and when that snapshot now holds no rows. The refresh then fails with the input's reason: projection_input_unhealthy for a poll, or the code of the check its row failed. The refresh also fails and leaves the output unchanged when no input can be read fresh. It then fails with the first unhealthy input's reason, in the order the inputs are declared. An input whose access or schema changed fails the refresh whether or not it is healthy. So does running out of any refresh budget. The pages and bytes of a refused read still count toward the budgets, because they were read.

Choose the account a projection runs as

A projection refresh runs as a service account that an organization admin selects for that projection, separately in each environment. No platform identity is used instead: until an account is selected, every refresh fails with projection_execution_account_missing, and the output is not published. An admin can only select an account their own clearance covers. Saving one cleared above them is refused. Every refresh checks this again for the admin who last saved the selection. If their clearance no longer covers the account, the refresh fails with projection_execution_account_not_accredited until an admin with enough clearance saves the selection again. These failures count toward auto-pause. The person who added the projection, whom it runs as, must also be allowed to hold the account: otherwise the projection would read data for them that they may not read themselves. If their clearance does not cover the account, the projection stops with runs_as_clearance_lost, like the other run-as stops. An organization admin raises their clearance or selects an account they may hold, and then resumes it.

Selecting an account grants it no table access by itself. The platform mints a short-lived token for the account on each refresh, so you never hand it a client secret. See Service accounts for workspace, environment and permission settings.

Mapped output columns can be string, int, bigint, decimal, float, double, boolean, date, timestamp, or geo_point. json, array, binary, and vector columns are rejected when the manifest is validated, because a projection cannot copy them from an input without changing them. Each mapped column must have the same type as its source column, with two exceptions: an int source can feed a string column, and a string source holding an ISO 8601 timestamp can feed a timestamp column. bigint values are copied exactly, including values beyond JavaScript's safe-integer range.

The output table has its own access policy, which an admin manages explicitly. Changing an input policy affects future refresh reads; it does not automatically revoke access to already materialized output or rewrite that output's policy.

When the output table is bound to an ontology object type, access to those objects is granted through the ontology: an analyst who may read the object type does not also need a grant on the table. An explicit restriction an admin places on the output table does apply to that analyst's ontology and graph reads. A table-wide or column deny policy that matches the analyst returns no rows from that table. A hide or mask policy hides or masks the matching property.

Every tick produces a batch of rows; ingest.writeMode controls how that batch lands in the table. It is install-only — it provisions the StarRocks key model at table-create time and cannot be changed afterwards (changing it requires re-installing the extension with an updated manifest). The default is upsert.

For replace, a producer error, rejected row, inconsistent row counts, or an HTTP poll exceeding the runtime row limit prevents publication. The previous table contents remain available and the tick records a failed sync. Reaching the limit exactly is allowed; rows deliberately excluded by filter.requireNonNull do not count toward it. This completeness check does not change the existing partial-batch behavior of append or upsert sources.

A replace poll that returns zero rows writes nothing: the previous snapshot stays. Without a producer error the tick records a successful sync; with one it records a failed sync. Empty append and upsert polls do not write either.

ingest: {
  mode: "poll",
  intervalSec: 60,
  writeMode: "changed-only", // omit for the "upsert" default
  // ...request, response, mapping
}

Upgrading a source's columns

Shipping a new extension version that adds columns to table.columns just works: on the first sync after the upgrade, the platform compares your declared columns against the existing table and adds the missing ones in place — with the declared type, classification, and description, no re-install and no data loss. Orgs that upgraded before syncing recover automatically on their next scheduled sync.

Two changes are not applied in place, because the warehouse cannot alter them on a live table:

• Changing a column's type. The existing column keeps its type; declare a new column instead.
• Adding or changing a primaryKey column. The key model is fixed at create time (same rule as writeMode). The sync log records the skipped key column; pick a new table.name (which provisions a fresh table) if the identity truly has to change.

Columns you remove from the declaration are left in place in the warehouse — they simply stop receiving values.

After an upgrade you can refresh everything at once: the Analytics → Data Sources view has a Resync all toolbar action that re-runs every installed source's sync (a few at a time), alongside the per-row Sync now. Statuses, last-sync times, and row counts update in the table as each source finishes.

Static reference data (inline rows)

Not every table comes from an HTTP feed. Reference data — base registries, type catalogues, fleet rosters, curated synthetic telemetry — can ship inside the manifest as inline rows. Replace the request/response/mapping trio with a rows array; everything else (scheduler, write modes, retention, the install flow) is identical, because the same poll driver runs the source and simply re-emits the rows on every tick. That makes the backing table idempotent and self-healing: drop a row in the warehouse and the next tick restores it. No network egress, no credentials, no SSRF surface.

import { defineDataSource } from "@scrydon/sdk-authoring/extensions";

export default defineDataSource({
  kind: "table",
  id: "nato-bases",
  vendor: "scrydon",
  displayName: "NATO Air Bases (Eastern Flank)",
  scope: "org",
  table: {
    name: "nato_base",
    primaryKey: ["baseId"],
    timestampColumn: "updatedAt",
    columns: [
      { name: "baseId", dataType: "string", isPrimaryKey: true },
      { name: "name", dataType: "string" },
      { name: "country", dataType: "string" },
      { name: "latitude", dataType: "decimal" },
      { name: "longitude", dataType: "decimal" },
      { name: "updatedAt", dataType: "timestamp" },
    ],
  },
  ingest: {
    mode: "poll",
    intervalSec: 3600,          // reference data — an hourly re-assert is plenty
    rows: [
      { baseId: "EYSA", name: "Šiauliai Air Base", country: "Lithuania", latitude: 55.894, longitude: 23.395 },
      { baseId: "EPPW", name: "Powidz Air Base", country: "Poland", latitude: 52.379, longitude: 17.854 },
      // ... up to 10,000 rows
    ],
    stampColumn: "updatedAt",   // stamped with the tick time on every emit
    writeMode: "upsert",
  },
});

The static flavor is recognized by the presence of rows (the HTTP flavor by request); a manifest cannot mix the two. Field notes:

Retention

For time-series tables that grow continuously, declare table.ttlSec alongside table.timestampColumn. The platform partitions the StarRocks table by day on the timestamp column and keeps only the most recent ceil(ttlSec / 86400) daily partitions — older partitions are dropped automatically.

table: {
  name: "aircraft_position",
  primaryKey: ["icao24", "seenAt"],
  timestampColumn: "seenAt",   // the column partitions are cut on
  ttlSec: 604800,              // keep ~7 days of partitions
  columns: [ /* ... */ ],
}

Because the table is partitioned on it, a timestampColumn used with ttlSec must be a date or timestamp column, and it must be one of the key columns, as seenAt is above. A manifest that declares ttlSec and breaks either rule is refused at build time with a message that names the column. timestampColumn must always name one of table.columns, except that an on-demand source may name the system column __fetchedAt (see Ingest modes), which these rules then check like any other. Without ttlSec nothing is partitioned, so the column can have any type and does not have to be part of the key.

ttlSec is the authoring default. An organization admin can override the effective retention per source in Analytics → Data sources: open the source's detail panel and set Retention (seconds), or reset it to the extension's default. The override is applied live, with no table rebuild; a query source has no managed table and no retention setting. The detail panel also shows the source's write mode (read-only) and its recent sync history, and links straight to the backing table.

On Analytics → Data sources, a source's row menu has Remove data source: it deletes the source, which stops running. Its table and every row in it are kept — the source only stops writing — and the table stays on Analytics → Tables, where deleting it is its own action. The extension that contributed the source stays installed, even when the source was its only content; uninstall it from Settings → Platform → Extensions when nothing else needs it.

Only an organization admin can change a source's retention or remove it. Other members see the retention read-only and are not offered Remove data source.

Sync health and auto-pause

Every scheduled tick records a sync state on the source: ok, error (with the last error and a consecutive-failure count), or paused. A source is paused in two ways:

• Automatically. When a tick fails and the consecutive-failure count reaches the organization's threshold (default 10), the platform pauses the source instead of retrying it every interval forever. An organization admin sets or disables the threshold under Settings → Analytics ("Auto-pause failing data sources"). A tick the provider rate-limits (HTTP 429) is recorded as provider_rate_limited, with the Retry-After the provider asked for, and does not count toward the threshold: the provider answered, and the next interval is the retry. A Retry-After of up to 10 seconds is also honoured once or twice within the tick itself. If a source is rate-limited on every tick, lengthen its interval. A tick whose table is busy with another write is not a failure at all: nothing is recorded, the source keeps its status, and the tick runs again within minutes (never later than its interval).
• Manually. Choose Pause syncing on a source under Analytics → Data sources (Pause copying for a query source that mirrors, Pause live access for one read in place), and confirm.

A query source that mirrors its tables walks the same threshold with its mirror runs, and the automatic pause stops only the copy: the source's live catalog stays registered, so live queries against the database keep working while the mirror is paused. Every organization administrator is notified in-app when it happens (notification type data_source.copy_paused). A mirror run that could not run because another holder had its table — another run, a full query admission gate, or another write on the destination — is not recorded and does not count; it is retried within minutes. A manual pause stops both the copy and live queries.

A paused source keeps its table; scheduled runs skip it without polling the provider or writing rows, and Sync now on it is refused (409 source_paused) with the fix. To bring it back, choose Resume syncing (Resume copying, Resume live access): the source returns to pending with its failure count reset. Resuming a query source whose copy was paused automatically does not re-register its live catalog.

Batch fingerprints

sourceBatchFingerprint(rows) (@scrydon/sdk-authoring/data-sources) is the SHA-256 of a batch's rows in canonical JSON (JCS): the same rows give the same hex string whatever order their keys are in. Scrydon takes it of every batch before your organization's data-protection rules scan it, and uses it to recognise a batch it has already written — so a batch sent again after a rule changed is still recognised. Use it when you need to name a batch the way Scrydon does, for example in a test.

import { sourceBatchFingerprint } from '@scrydon/sdk-authoring/data-sources'

await sourceBatchFingerprint([{ id: 'a', at: '2026-10-02T00:00:00Z' }]) // 64 hex characters

Extension layout

An Extension that ships a data source adds one data-source-<id>/ subdir per source, each containing a single manifest.json. Multiple data sources may coexist in the same Extension — data-source is a repeatable content kind, like workflow.

<extension>.scrydon-extension.tar.gz
├── extension.json                               # ExtensionArchiveManifestSchema
└── data-source-adsb-lol-military-declarative/
    └── manifest.json                       # DataSourceManifestSchema (pure JSON, no code)

Compose the Extension with defineExtension and list each source in provides.dataSources. extension build writes extension.json from it — one data-source entry per source — and each source's manifest.json:

import { defineExtension } from '@scrydon/sdk-authoring/extensions'
import { adsbLolMilitaryDeclarative } from './data-source-adsb-lol-military'

export default defineExtension({
  id: 'adsb-lol-military',
  name: 'ADS-B Lol Military Aircraft',
  version: '1.0.0',
  logo: './assets/icon.svg',
  provides: { dataSources: [adsbLolMilitaryDeclarative] },
})

A source that reads a credential or an endpoint from its connection also needs the extension to declare them — auth.credentials and configFields on the same defineExtension call. See Endpoints from a connection.

Build, inspect, upload

bunx @scrydon/sdk-authoring extension build src/extension.ts --outDir dist
# → dist/<id>-<version>.scrydon-extension.tar.gz

bunx @scrydon/sdk-authoring extension inspect dist/adsb-lol-military-1.0.0.scrydon-extension.tar.gz

curl -X POST "$AGENTIC_URL/api/extensions/import?organizationId=$ORG_ID" \
  -H "Cookie: $SESSION_COOKIE" \
  -F "file=@dist/adsb-lol-military-1.0.0.scrydon-extension.tar.gz"

New rules bind new builds. The build and the upload hold every source to the SDK's current rules. A source that is already installed keeps running on the definition it was installed with, even when a later rule would refuse that definition in a new build. If a later Scrydon cannot read an installed definition at all, that source alone stops and reads This source's stored definition no longer validates against this version of Scrydon; every other source keeps running. Install a newer version of the extension to fix it.

Data sources that need a credential

Some APIs require an account or API key. The extension declares the credential once, in auth.credentials on defineExtension — an extension that ships only data sources declares it there too, without any toolkit. The source names it with request.authRef. The manifest never carries a secret; authRef is a name, not a value. (The DataSourceManifestSchema rejects any attempt to embed an Authorization header or API key directly in the manifest.)

// defineExtension({ ... })
auth: {
  credentials: { exampleKey: { type: 'apiKey', label: 'Example API key' } },
  default: 'exampleKey',
},

// defineDataSource({ ... }).ingest
ingest: {
  mode: "poll",
  intervalSec: 300,
  request: {
    url: "https://api.example.com/v1/records",
    method: "GET",
    authRef: "exampleKey",   // a key of auth.credentials — NOT an inline token
  },
  // ...
}

The build refuses an authRef or credentialRef that names no credential in auth.credentials, and a source whose vendor is not the extension's id — the extension whose connections carry the credential. It also refuses an authRef that names a basicAuth credential: a data source that calls an API places one secret on its request — an API key (apiKey), a bot token (botToken) or an OAuth access token (oauth, oauthClientCredentials) — and a username and password is not one. A connection that signs in with a username and password is refused at run time too, with HTTP 412 data_source_connection_required, rather than having its secret sent as a key.

How the connection flows:

1. An organization administrator creates a connection for the extension in Settings → Platform → Extensions → <extension> → Connections, entering the credential the extension declared.
2. On each tick, the platform resolves that connection server-side and attaches the credential to the outbound request — your manifest never sees the secret.
3. If no enabled connection exists yet, the tick returns HTTP 412 data_source_connection_required — a clear signal to connect the account first. No data is fetched, and no error is silently swallowed.
4. Once a connection is enabled, subsequent ticks resolve it automatically and run.

If more than one enabled connection matches the source's environment and none is chosen, the platform does not choose between them: the source shows More than one connection in this environment can serve …, and its ticks return HTTP 412 data_source_connection_required without fetching, until an administrator chooses one with Change connection on the source's row, or adds an instance to read through both. Adding a connection never puts a source that was already running into this state: before the new connection is saved, the source is fixed to the connection it used (see Connecting accounts).

One source, several instances. An administrator can install the same source more than once in an environment, each copy an instance pinned to one of the extension's connections (for example a personal and a company Home Assistant). Every run of an instance — scheduled tick, Sync now, Fetch now, a push delivery, a stream, a query catalog — reads exactly its pinned connection. Every run, including a manual upload, writes the instance's own table: the source's table.name followed by the instance name (ha_states_company); the original instance keeps table.name unchanged. If an instance's pinned connection is deleted, disabled or moved to another environment, the instance returns HTTP 412 data_source_connection_required and fetches nothing. It never falls back to another connection. Your manifest does not change: one definition serves every instance.

No authRef? A source with no authRef (public API, no authentication needed) ticks immediately — no credential setup required.

Security

Resource limits

Enforced on every poll tick and every on-demand fetch:

Rows a filter drops do not count. The other modes have their own limits, and each also refuses a payload over them whole rather than keeping its first rows: a manual upload takes up to 50,000 rows and 16 MiB, a push delivery up to 1,000 rows and 1 MiB, and a stream message that maps to more than 1,000 rows is set aside (see Ingest modes).

What a declarative source cannot do yet

Four kinds of API a declarative source does not reach today, and what to do instead:

A search that can come back empty is not on this list: when the vendor omits the list for no matches (GDELT's {}, an RSS channel with no <item>), set itemsOptional: true on response and that answer is zero rows instead of a failed tick — see Anatomy.

Where to next