Ingest modes

A data source's ingest.mode says what drives the fetch. Everything else — the request block, response.itemsPath, mapping, filter, table.columns — works the same way in every mode, so moving between them is a change of driver, not a rewrite.

table.timestampColumn is required for modes that write rows, because storage partitions by it. On-demand sources may name the system timestamp __fetchedAt. Query sources obtain their mirror schema from governed remote driver metadata; see Query SQL sources. table.columns is required for the other declarative modes — it is the only row validator a declarative source has.

Install and configure a source

Extension installation, connection configuration and source operation are separate steps. Organisation administrators install the extension and manage its connections. Authorised workspace administrators activate its sources in the relevant workspace environment. Installing an extension does not establish a database connection or start a successful ingestion.

Open the installed source's Connections view and select Configure to open the extension's Connections tab. Create or select the applicable connection for the intended environment. Credentials are organisation-owned; connections can be scoped to Development, Production, or explicitly to all environments. Use separate environment connections when the credentials differ. Moving source configuration to Production does not copy Development credentials. Existing connection-selection rules and administrator permissions still apply. Database query sources require an enabled basic-auth connection scoped to the exact environment; an all-environments connection does not satisfy that requirement.

The Setup tab describes organisation installation. The Connections tab reports whether applicable credential metadata is configured; this does not verify remote connectivity. Check the source's latest operation and data freshness separately in Data Sources, and use Open table to inspect persisted records. Connection edits and credential rotation use the same Connections controls as other extensions; do not embed credentials in a source manifest.

If credential setup cannot load, use the displayed retry action or ask an organisation administrator to check the extension's declared credential provider. A failed load is not confirmation that the source needs no credentials.

manual — source-owned uploads

A manual source declares the format, selection, mapping and validation of an uploaded file. It owns its managed table, unlike a generic user-managed table upload. Declare ingest: { mode: 'manual', upload: { itemsPath: '$', format: 'csv' }, asOfColumn: 'asOf', mapping: { deviceId: { path: 'device' } }, writeMode: 'append' }. The table must declare an asOf timestamp column and a timestampColumn for partitioning. The asOfColumn must not also appear in mapping; the uploaded file cannot override it. A replace upload containing any invalid rows fails before replacing the existing dataset.

In Analytics, open the installed source detail, select Source file, enter As of (UTC) and press Upload to source. The source's mapping is applied to the stored blob, valid rows are written to this exact source/environment, and rejected-row counts are shown. Uploads are bounded to 16 MiB and 50,000 selected rows, and the form refuses a larger file before uploading it. The file reaches the ingest inside one request that the same 16 MiB bounds, so a file just under the limit whose quotes and line breaks have to be escaped (JSON especially) can still be refused as too large. A file over either limit is refused whole, never cut to its first rows: split it into several uploads, or have its producer send it as push deliveries of up to 1,000 rows each. A UTF-8 byte-order mark at the start of the file, which Excel's CSV UTF-8 format writes, is ignored. The stored provenance retains the source, uploaded storage key, descriptor version, received time and as-of time. A retry with the same blob and as-of time reuses the source-qualified receipt; changing the content or as-of time under that receipt is refused. Pressing Upload to source again for the same upload says This file was already received; nothing new to write. — its rows were written when it was first received, and nothing is written twice. The source's Last sync and row count stay those of that first upload.

A manual source is also the way in, for now, for two files a poll source cannot fetch:

• A file published under a URL that changes every day, such as GPSJam's https://gpsjam.org/data/2026-09-27-h3_4.csv. A poll request's URL is fixed in the manifest and has no date placeholder, so download the day's file and upload it with that day as its As of (UTC).
• A file larger than 10,000 rows. A poll or on-demand response that maps to more is refused whole (see Resource limits); a manual upload takes up to 50,000.

on-demand — fetch, append and return

An on-demand source answers a question the user just asked: "which hosts match this query?" It has no schedule. Each execution appends validated rows to its managed table and returns the rows exactly as they were stored, with a persistence outcome: a row your DLP policy set aside is not returned, and a row that could not be saved is not returned either.

defineDataSource({
  kind: 'table',
  id: 'shodan-host-search',
  vendor: 'shodan',
  displayName: 'Shodan — host search',
  scope: 'org',
  credentialRef: 'apiKey',
  table: {
    name: 'shodan_host',
    primaryKey: ['ip'],
    timestampColumn: '__fetchedAt',
    columns: [
      { name: 'ip', dataType: 'string', isPrimaryKey: true },
      { name: 'port', dataType: 'int', nullable: true },
      { name: 'org', dataType: 'string', nullable: true },
    ],
  },
  ingest: {
    mode: 'on-demand',
    request: {
      url: 'https://api.shodan.io/shodan/host/search',
      authRef: 'apiKey',
      auth: { placement: 'query', name: 'key' },
    },
    response: { itemsPath: '$.matches' },
    params: {
      query: { in: 'query', name: 'query', required: true, maxLength: 512 },
      page: { in: 'query', type: 'number' },
    },
    mapping: {
      ip: { path: 'ip_str' },
      port: { path: 'port' },
      org: { path: 'org', transform: 'trim_to_null' },
    },
  },
})

Writing intervalSec or writeMode on an on-demand source is a validation error, not a field that is quietly ignored: on-demand has no schedule and always appends.

The visible system columns are __fetchedAt (timestamp), __fetchedBy (verified initiating user, or verified service for a service-only call), and __args (canonical JSON text of validated caller arguments). Do not declare or map these reserved names, including case variants. They are classified at least internal, or at the source's stricter column level. Existing compatible tables gain missing columns; historical values remain null. Legacy installed snapshots without a timestamp use __fetchedAt without rewriting the snapshot.

A response that maps to more than 10,000 rows fails the fetch with exceeds maxRows (10000): nothing is returned or stored, rather than the first 10,000. Narrow the request with a parameter, such as a smaller page size.

persistence.status is committed, failed, or skipped. rowsWritten counts acknowledged new writes; receipt replay counts zero. Failed persistence retains the fetched rows and any confirmed earlier batches. A timeout may have stored rows even without confirmation. Batches are bounded to 1,000 rows and a 1 MiB envelope; an oversized individual row fails saving visibly. A fetch-only runtime without a composed sink reports skipped.

params — what a caller may send

In Analytics, open the source's detail panel and choose Fetch now. Enter a JSON object in Fetch arguments (JSON) using the keys declared in params, for example {"query":"port:443","page":1}. The panel lists each declared argument with whether it is required and, for an enum, its accepted values; a required argument whose enum has exactly one value is pre-filled. Sources without arguments use {}. A refused argument is named in the error, with its accepted values. A failed fetch says why in the words a scheduled run records for the same failure, for example that the request's host name could not be found, or the HTTP status and the provider's error message. The panel asks you to try again only when the cause could not be named. The panel shows returned rows as a grid under the columns the source declares in table.columns, and the rejected-row count, including an explicit empty result. Each click requests fresh data. The preview is discarded when the panel closes or another source opens; stored rows remain in the source table under its retention setting. The panel shows confirmed new writes, partial or unconfirmed saving, sync history, and Open table after table creation. A new fetch may append the same provider rows again. Responses use Cache-Control: no-store. A fetch that fails is recorded on the source as a scheduled run's failure is: its status, last error and Activity say why.

params is the closed list of arguments a caller may supply. An argument you did not declare is rejected, never forwarded. Each entry says where the value goes and what it may be:

A caller argument can only ever become a query parameter or a JSON body field. It can never reach the request URL's scheme, host or path, so it cannot move the call to a different server. Two rules are enforced when the manifest is parsed rather than at call time:

• a parameter whose wire name is the one request.auth uses for a query placement is rejected — otherwise a caller could choose the credential's value;
• a parameter whose wire name collides with a static request.query key is rejected — otherwise it would silently overwrite it.

in: 'body' requires request.method: 'POST' with a request.body of kind: 'json' whose value is an object; the argument is merged in as a top-level field.

There is no in: 'path', so an API that takes its input as part of the URL path cannot receive it from a caller today. TomTom's free-text geocoding, GET https://api.tomtom.com/search/2/geocode/{query}.json, is one: {query} in request.url is sent literally. Use the vendor's query-parameter variant where it has one — for TomTom, the structured geocode GET https://api.tomtom.com/search/2/structuredGeocode.json, whose countryCode, streetNumber, streetName, municipality and postalCode are all query parameters a params entry can declare.

What a caller gets back

Rows, validated against table.columns, plus the count of rows the source produced that failed that validation. Rows that fail are dropped, not returned.

If the source references a stored credential and the organization has no enabled connection for it, the fetch is refused with data_source_connection_required — the same refusal a scheduled poll run records as its sync state — rather than running without authentication.

stream — a subscription held open

ingest: {
  mode: 'stream',
  transport: 'websocket',
  endpoint: {
    url: 'wss://stream.example.com/v1/positions',
    authRef: 'apiKey',
    auth: { placement: 'bearer' },
  },
  subscribe: [{ kind: 'json', value: { BoundingBoxes: [[[-90, -180], [90, 180]]] } }],
  message: { itemsPath: '$' },
  filter: { requireNonNull: ['MetaData.time_utc'] },
  mapping: {
    mmsi: { path: 'MetaData.MMSI' },
    seenAt: { path: 'MetaData.time_utc', transform: 'iso_from_go_time' },
  },
}

• transport is websocket — the only transport with a governed host behind it.
• endpoint.url must be wss://. Cleartext ws:// is refused for the same reason request.url must be https://. Or read the host from the connection: endpoint: { endpointRef: 'baseUrl', path: '/api/websocket' } opens wss:// on the connection's https:// endpoint — see Endpoints from a connection.
• endpoint reuses the poll request block's credential vocabulary: authRef plus an auth placement of bearer, header or query — and, for a stream only, frame (below). An inline Authorization header is refused here too.
• endpoint.protocols offers WebSocket subprotocols in the handshake (at most 8). Each must be a single token (no spaces or commas) and appear once. If you offer any, the server must select one of them, or the connection is refused.
• The socket dials only the address the egress check approved, and verifies the server's certificate against the endpoint's host name. Redirects are not followed and compression is not negotiated. A message larger than 1 MiB closes the connection.
• subscribe is the list of frames sent once the socket opens — { kind: 'json', value } or { kind: 'text', value }. A WebSocket frame carries no content type, so unlike request.body the text frame needs no contentType.
• message is applied to every inbound message exactly as response is applied to a poll body: decode by format, then select the row array at itemsPath. A message that is one object rather than a list uses itemsPath: '$', and filter.requireNonNull drops a control frame that carries no data, such as a subscription confirmation.

Reconnect and backoff policy belongs to the runtime; the manifest has no tuning fields.

A stream follows its connection. When an admin saves a new key on the connection it uses, points the source at another connection, changes the connection's endpoint, or removes the connection, the running stream notices within about 10 seconds and reconnects with what the connection holds now. The rows it already received are committed first, so nothing is lost or written twice. Renaming a connection, or an OAuth token being refreshed, does not reconnect it. You never need to pause and resume a source for a connection change to apply.

A received row is kept durably within 250 ms, then committed to the table with the rows around it: a stream's table takes at most one commit a minute, so a row appears in it up to about a minute after it arrived. A commit holds up to 1,000 rows and just under 1 MiB of row data; a feed busy enough to fill one sooner commits it at once. One message that maps to more than 1,000 rows, or to more row data than one commit holds, is set aside whole.

A message or a row set aside is kept, as it arrived, for one hour — your Scrydon operator can read what was set aside within that hour. After it, what was set aside is deleted: its count stays in the source's statistics. A message larger than 1 MiB cannot be held whole: its count, reason and fingerprint are kept, and the record says its content was omitted.

A stream is delivered to its table at least once: after a reconnect, or when another replica takes the stream over, a batch can be committed again. Keep the default writeMode: 'upsert' (or changed-only) with a table.primaryKey that identifies one observation, so a replayed row overwrites itself. append would store the replay twice, so a stream declared append runs only on a deployment whose operator has enabled ingestBuffer.recoverableSinkEnabled; anywhere else it does not start, and the source reads as unavailable, naming that setting.

A key that goes in the first message

Some feeds (aisstream.io, for one) take their API key inside the subscribe frame, not in a header. JSON subscribe frames may reference stored apiKey or accessToken credentials, and auth: { placement: 'frame' } keeps the key there only — the WebSocket handshake carries it nowhere, neither as a header nor in the URL:

endpoint: {
  url: 'wss://stream.aisstream.io/v0/stream',
  authRef: 'aisstreamKey',          // the extension's credential (auth.credentials)
  auth: { placement: 'frame' },     // the key goes in the frame below, nowhere else
},
subscribe: [
  {
    kind: 'json',
    value: { APIKey: null, BoundingBoxes: [[[51.0, 1.5], [56.0, 9.0]]] },
    credentials: [{ path: ['APIKey'], credential: 'apiKey' }],   // fill APIKey with its apiKey
  },
],

• authRef names the credential; credential in each reference names the field of it that fills the path (apiKey, or accessToken for an OAuth connection).
• Every path must identify an existing null object field. Missing, overlapping, array and prototype paths are refused. Text frames never substitute credentials, and no secret value is shipped in the manifest.
• References require endpoint.authRef, which names the credential.
• A frame reference with no endpoint.auth is refused when you build or upload the extension: the default placement would also send the same key as Authorization: Bearer on the handshake. Say 'frame', or name the handshake placement you want as well. A stream installed before this rule keeps running exactly as it was installed — the key in the frame and on the handshake — until you install a version that says 'frame'.
• placement: 'frame' is refused on a stream whose frames reference no credential, and on any poll or on-demand request — an HTTP request has no frame.

When a stream is unhealthy

A stream records its sync state once per 30-second window rather than per message, and says what went wrong in the data source's status and last error:

A frame your filter drops (a subscription confirmation, a heartbeat) is neither: it is counted as a message received and nothing else. No row value appears in any of these.

Every source runs on Scrydon's ingestion service, which every deployment runs; stream and push sources also need its ingest buffer, and poll sources do not. A deployment without the buffer shows each stream and push source as unavailable, with the guidance for your Scrydon operator (it is enabled with ingestBuffer.enabled in the Helm values).

Worked example: AIS positions from aisstream.io

This source keeps live vessel positions for the North Sea from aisstream.io, which serves AIS over a WebSocket and takes its API key in the subscribe frame. The extension declares the key once:

import { defineExtension } from '@scrydon/sdk-authoring/extensions'
import { aisPositions } from './data-source-ais-positions'

export default defineExtension({
  id: 'aisstream',
  name: 'AIS Stream',
  version: '1.0.0',
  logo: './assets/icon.svg',
  auth: {
    credentials: { aisstreamKey: { type: 'apiKey', label: 'aisstream.io API key' } },
    default: 'aisstreamKey',
  },
  provides: { dataSources: [aisPositions] },
})

import { defineDataSource } from '@scrydon/sdk-authoring/extensions'

export const aisPositions = defineDataSource({
  kind: 'table',
  id: 'ais-positions-north-sea',
  vendor: 'aisstream',                  // the extension id: its connections carry the key
  displayName: 'AIS positions — North Sea',
  scope: 'org',
  table: {
    name: 'ais_position',
    primaryKey: ['mmsi', 'observedAt'],  // one row per vessel per report
    timestampColumn: 'observedAt',
    ttlSec: 7 * 86400,
    columns: [
      { name: 'mmsi', dataType: 'bigint', isPrimaryKey: true },
      { name: 'shipName', dataType: 'string', nullable: true },
      { name: 'latitude', dataType: 'double' },
      { name: 'longitude', dataType: 'double' },
      { name: 'speedKnots', dataType: 'double', nullable: true },
      { name: 'courseDeg', dataType: 'double', nullable: true },
      { name: 'headingDeg', dataType: 'int', nullable: true },
      { name: 'navStatus', dataType: 'int', nullable: true },
      { name: 'observedAt', dataType: 'timestamp', isPrimaryKey: true },
    ],
  },
  ingest: {
    mode: 'stream',
    transport: 'websocket',
    endpoint: {
      url: 'wss://stream.aisstream.io/v0/stream',
      authRef: 'aisstreamKey',
      auth: { placement: 'frame' },
    },
    subscribe: [
      {
        kind: 'json',
        value: {
          APIKey: null,
          BoundingBoxes: [[[51.0, 1.5], [56.0, 9.0]]],   // [[lat, lon], [lat, lon]]
          FilterMessageTypes: ['PositionReport'],
        },
        credentials: [{ path: ['APIKey'], credential: 'apiKey' }],
      },
    ],
    message: { itemsPath: '$', format: 'json' },
    filter: { requireNonNull: ['MetaData.MMSI', 'MetaData.time_utc'] },
    mapping: {
      mmsi: { path: 'MetaData.MMSI' },
      shipName: { path: 'MetaData.ShipName', transform: 'trim_to_null' },
      latitude: { path: 'MetaData.latitude' },
      longitude: { path: 'MetaData.longitude' },
      speedKnots: {
        path: 'Message.PositionReport.Sog',
        transform: 'value_map',
        args: { map: { '102.3': null }, passthrough: 'number' },
      },
      courseDeg: {
        path: 'Message.PositionReport.Cog',
        transform: 'value_map',
        args: { map: { '360': null }, passthrough: 'number' },
      },
      headingDeg: {
        path: 'Message.PositionReport.TrueHeading',
        transform: 'value_map',
        args: { map: { '511': null }, passthrough: 'number' },
      },
      navStatus: { path: 'Message.PositionReport.NavigationalStatus' },
      observedAt: { path: 'MetaData.time_utc', transform: 'iso_from_go_time' },
    },
    writeMode: 'upsert',
  },
})

Every message aisstream sends for this subscription is one envelope, not a list:

{
  "MetaData": {
    "MMSI": 246055000, "MMSI_String": 246055000, "ShipName": "TS10 HILLEGONDA     ",
    "latitude": 53.2949, "longitude": 5.4693,
    "time_utc": "2026-09-28 08:06:58.965887092 +0000 UTC"
  },
  "MessageType": "PositionReport",
  "Message": {
    "PositionReport": {
      "UserID": 246055000, "Latitude": 53.29490166666667, "Longitude": 5.469303333333333,
      "Sog": 3.2, "Cog": 70.2, "TrueHeading": 57, "NavigationalStatus": 0
    }
  }
}

(The position report carries more fields than shown here.) Why each part of the source is written the way it is:

• The key never reaches the handshake. placement: 'frame' puts it in APIKey in the subscribe frame and nowhere else. The manifest carries null; the organization's connection supplies the key when the stream connects.
• itemsPath: '$' selects the envelope itself as a one-row list, so each message is one row. aisstream sends its JSON in binary WebSocket frames; they are read as UTF-8, and format: 'json' says how to decode them, since a WebSocket message has no content type.
• The filter drops the subscription confirmation. The first message of every subscription is {"Message":{"CompressionEnabled":true},"MessageType":"SubscriptionConfirmation"}. It has no MetaData, so it is dropped. It is counted as a message received, never as a row set aside.
• The casing is aisstream's, not a convention. MetaData writes latitude and longitude in lowercase, beside MMSI, ShipName and time_utc. A path is case-sensitive, so MetaData.Latitude resolves to nothing and every row is set aside for a missing latitude. MMSI is a JSON number, hence bigint. The MetaData coordinates are the report's position rounded to 5 decimal places (about 1 m); map Message.PositionReport.Latitude and Longitude instead if you need every digit. ShipName arrives sometimes padded with spaces to 20 characters and sometimes trimmed; trim_to_null gives it one spelling and turns a blank name into null.
• time_utc is not RFC 3339. It is Go's time.Time text, and mapped straight through, every row is set aside with observedAt: Invalid ISO datetime. iso_from_go_time turns it into 2026-09-28T08:06:58.965887092Z and keeps every fraction digit.
• AIS sends "not available" as a number. A true heading of 511, a course of 360 and a speed of 102.3 mean the transponder did not report one. value_map with passthrough: 'number' turns exactly those into null and passes every other number through. They are common: in a sample of North Sea position reports, 66% had heading 511 and 30% had course 360. Without the map they would land as real bearings.
• primaryKey + upsert. The pair mmsi + observedAt keeps each vessel's track, one row per report, and makes a batch committed twice overwrite itself.

This source was checked against 4,491 position reports captured live from this subscription on 28 September 2026: every report produced a valid row, and the only message dropped was the subscription confirmation. The same box carried about 37 position reports a second that morning, peaking at 73, which batching absorbs. A larger box or more message types multiply that; narrow BoundingBoxes and FilterMessageTypes to what you use.

What you see in Analytics → Data sources for this source:

push — the vendor delivers to us

A push source has no request and no endpoint — we do not dial out. What it declares is how a delivery is authenticated, using the same vocabulary as Scrydon's inbound webhook verifier:

ingest: {
  mode: 'push',
  deliveryIdHeader: 'x-delivery-id',
  auth: {
    mode: 'hmac',
    headerName: 'x-hub-signature-256',
    algorithm: 'sha256',
    timestampHeader: 'x-signature-timestamp',
  },
  delivery: { itemsPath: '$.events' },
  mapping: { id: { path: 'id' }, seenAt: { path: 'timestamp' } },
}

The manifest declares the shape of the credential and never its value. There is no field to put a secret in: the secret is the organization's own stored connection credential, resolved server-side. Prefer hmac, then header; reach for url_token only when the producer cannot send headers at all, because a secret in a URL reaches proxy logs and referrers.

An hmac signature covers more than the body. The producer signs the string <timestamp>.<delivery id>.<raw body>, where <timestamp> is the Unix time in seconds it sends in timestampHeader and <delivery id> is the value of deliveryIdHeader. This binds each signature to one delivery at one time:

• A delivery whose timestamp is more than 300 seconds from the receiver's clock, in either direction, is refused with 401 push_signature_stale.
• A missing or non-numeric timestamp, or a signature over the body alone, is refused with 401 push_authentication_failed.
• A captured delivery re-sent under a new delivery id no longer verifies, and a re-send under the same delivery id is recognised as a duplicate of the original receipt.

timestampHeader must name its own header: it cannot be the signature header or the delivery-id header.

Only an authenticated delivery counts against the source's push rate limit, so a caller that knows the receiving URL but not the secret cannot exhaust it.

A delivery stops counting against the source's buffer capacity as soon as it is processed, but its id is remembered for one hour: within it a re-send is answered as a duplicate of the original receipt, and after it the id is forgotten. The hour is longer than twice the 600-second window in which one signed timestamp is accepted, so a signed delivery cannot be replayed after its id has been forgotten.

delivery is applied to each delivered body exactly as response is to a poll body.

Where a sender delivers

Every installed instance of a push source has its own address, POST <public push host>/api/platform/data-sources/push/<binding ID>. The binding ID is derived from the organization, the instance and its environment, so an added instance has its own. An organization administrator finds it in the extension's Connections tab: in the source's row, ⋯ → Push endpoint shows the path, the header that carries the credential or signature, and the delivery ID header, each with a copy button. The public push host is the deployment's — see apiIngestion for how each routing mode publishes it. Platform is not configured with that host, so the dialog shows the path only.

Worked example: signed sensor readings

A sender posts batches of sensor readings and signs each body with a secret it shares with the organization. The extension declares that secret, and the source names it with credentialRef, so the connection form asks for it:

import { defineExtension } from '@scrydon/sdk-authoring/extensions'
import { sensorReadings } from './data-source-sensor-readings'

export default defineExtension({
  id: 'acme-sensors',
  name: 'Acme Sensors',
  version: '1.0.0',
  logo: './assets/icon.svg',
  auth: {
    credentials: { signingSecret: { type: 'apiKey', label: 'Webhook signing secret' } },
    default: 'signingSecret',
  },
  provides: { dataSources: [sensorReadings] },
})

import { defineDataSource } from '@scrydon/sdk-authoring/extensions'

export const sensorReadings = defineDataSource({
  kind: 'table',
  id: 'acme-sensor-readings',
  vendor: 'acme-sensors',
  displayName: 'Acme — sensor readings',
  scope: 'org',
  credentialRef: 'signingSecret',       // what each delivery is verified against
  table: {
    name: 'sensor_reading',
    primaryKey: ['readingId', 'observedAt'],
    timestampColumn: 'observedAt',
    columns: [
      { name: 'readingId', dataType: 'string', isPrimaryKey: true },
      { name: 'deviceId', dataType: 'string' },
      { name: 'value', dataType: 'double', nullable: true },
      { name: 'observedAt', dataType: 'timestamp', isPrimaryKey: true },
    ],
  },
  ingest: {
    mode: 'push',
    deliveryIdHeader: 'x-delivery-id',
    auth: {
      mode: 'hmac',
      headerName: 'x-signature-256',
      algorithm: 'sha256',
      timestampHeader: 'x-signature-timestamp',
    },
    delivery: { itemsPath: '$.readings', format: 'json' },
    mapping: {
      readingId: { path: 'id' },
      deviceId: { path: 'device' },
      value: { path: 'value', transform: 'number_or_null' },
      observedAt: { path: 'ts', transform: 'iso_from_epoch', args: { unit: 'ms' } },
    },
    writeMode: 'upsert',
  },
})

The sender computes an HMAC-SHA256, keyed with the secret, of <timestamp>.<delivery ID>. followed by the exact bytes it sends, and sends it as lowercase or uppercase hex, with or without the sha256= prefix. Each delivery carries its own ID, and a retry of the same delivery reuses it with a fresh timestamp:

BODY='{"readings":[{"id":"r-1041","device":"pump-7","value":4.2,"ts":1790582820770}]}'
DELIVERY_ID='7d0f3c2e-5b1a-4a8e-9f61-2c4b8e1d0a93'
TS=$(date +%s)
SIG=$(printf '%s.%s.%s' "$TS" "$DELIVERY_ID" "$BODY" | openssl dgst -sha256 -hmac "$SIGNING_SECRET" | awk '{print $NF}')
curl -X POST "https://<push host>/api/platform/data-sources/push/<binding ID>" \
  -H 'Content-Type: application/json' \
  -H "x-delivery-id: $DELIVERY_ID" \
  -H "x-signature-timestamp: $TS" \
  -H "x-signature-256: sha256=$SIG" \
  --data-binary "$BODY"

The other verifiers differ only in auth and in what the sender adds to its request. Each tab is a complete ingest block for the same source:

ingest: {
  mode: 'push',
  deliveryIdHeader: 'x-delivery-id',
  auth: { mode: 'bearer' },
  delivery: { itemsPath: '$.readings', format: 'json' },
  mapping: {
    readingId: { path: 'id' },
    deviceId: { path: 'device' },
    value: { path: 'value', transform: 'number_or_null' },
    observedAt: { path: 'ts', transform: 'iso_from_epoch', args: { unit: 'ms' } },
  },
  writeMode: 'upsert',
},

ingest: {
  mode: 'push',
  deliveryIdHeader: 'x-delivery-id',
  auth: { mode: 'header', headerName: 'x-api-key' },
  delivery: { itemsPath: '$.readings', format: 'json' },
  mapping: {
    readingId: { path: 'id' },
    deviceId: { path: 'device' },
    value: { path: 'value', transform: 'number_or_null' },
    observedAt: { path: 'ts', transform: 'iso_from_epoch', args: { unit: 'ms' } },
  },
  writeMode: 'upsert',
},

ingest: {
  mode: 'push',
  deliveryIdHeader: 'x-delivery-id',
  auth: { mode: 'url_token', paramName: 'token' },
  delivery: { itemsPath: '$.readings', format: 'json' },
  mapping: {
    readingId: { path: 'id' },
    deviceId: { path: 'device' },
    value: { path: 'value', transform: 'number_or_null' },
    observedAt: { path: 'ts', transform: 'iso_from_epoch', args: { unit: 'ms' } },
  },
  writeMode: 'upsert',
},

A delivery carries at most 1 MiB and 1,000 rows, and one over either limit is refused whole. Split a larger batch into several deliveries, each with its own ID. What the sender gets back:

Order, retries and deliveries set aside

• Deliveries are written in the order they were accepted, one at a time per source. A delivery is written only once every delivery accepted before it has been written or set aside, so a later delivery never overtakes an earlier one.
• A delivery that cannot be written now is retried for as long as that lasts — an outage, a busy table — after 15 seconds, then twice as long each time, at most 5 minutes apart. While it waits, the deliveries behind it wait too. An outage never sets a delivery aside.
• A delivery the table refuses as itself 5 times is set aside, whole, so the deliveries behind it can be written: rows that can never fit the table, a payload that conflicts with one already written under the same position, a change to the source its rows no longer match. For one hour the delivery is kept with everything it carried and what happened to it, can be read, and can be put back at the end of the queue to be written once the cause is fixed — ask your Scrydon operator. After that hour it is deleted.
• Rows a delivery carried that break the table's columns are kept too, beside the rows that were written, for the same hour, never only counted.
• A retry with the same delivery ID within one hour answers 202 with duplicate: true, whether the delivery was written or set aside, and nothing is written again. After the hour the ID is forgotten, and a delivery sent with it again is accepted as a new delivery.

A source that verifies nothing: auth.mode: 'none'

A push address is derived from the organization, the source and the environment, so it is not a secret, and a source that checks no credential would take rows from anyone who can reach it. Such a source therefore refuses every delivery until an organization admin adds a connection for it — the connection is the opt-in:

• In the extension's Connections tab the source offers a connection that takes no credential, described as Accept unauthenticated pushes from this source. Creating it, enabled, for the source's environment accepts deliveries; disabling or deleting it refuses them again. Connection changes are recorded in the audit log with the admin who made them.

• Only a connection created with no authentication opts the source in. A connection the admin creates with a secret or a token for the extension's other sources never covers it, and a no-authentication connection never serves a source that verifies a credential. Declare the extension's credential of type none next to its others, so the connection form can offer it:

  typescript

  auth: {
    credentials: {
      signingSecret: { type: 'apiKey', label: 'Signing secret' },
      none: { type: 'none', label: 'No authentication (accept unsigned deliveries)' },
    },
    default: 'signingSecret',
  },

• Until then the source shows Needs setup, and deliveries answer 403 with push_unauthenticated_not_accepted. Nothing is queued and nothing is written.

• Each environment needs its own connection. Two enabled connections for the same environment refuse deliveries until one is removed or each instance is pinned to one; an extra instance of the source accepts only through the connection it is pinned to.

• A change takes effect on the next delivery.

Sources that verify a credential (bearer, header, hmac, url_token) are unchanged: their connection holds the secret, and a delivery without a usable connection answers 503 so the sender retries. Declare none only for a producer that cannot authenticate at all.

Moving between modes

poll and on-demand share the same request and response blocks verbatim, so turning a scheduled source into a fetch-on-request one is: drop intervalSec and writeMode, add params, keep everything else. The reverse holds too.

Connection readiness

Installed sources show connection readiness for the selected workspace environment in Analytics and Platform extension settings. Manual and static-row sources are built-in. Other modes derive credential needs from their declared auth reference; a shared credential is counted once for each source that uses it. “Ready” confirms applicable credential metadata, not a successful provider request.

Use Configure to open the existing extension Connections panel. After configuring a credential, returning to Analytics refreshes readiness. Missing setup can hide an installed source from the Data Sources list; an unavailable readiness service does not hide sources, and the Extensions page remains available for discovery. Query sources continue to require one enabled basic-auth connection in the exact environment.

Operational sync history

The source detail panel shows the latest 50 operational events and links to the existing table history when a table is available. Successful stream batches are summarized per UTC minute, table, operation and actor, with committed batch and row counts. Replaying a committed receipt does not increment these counts. Failures remain individual events; other ingest modes keep their individual operational events.

Hourly maintenance retains the latest 1,000 events per source in each organization and workspace environment, plus every error event younger than 30 days. Each pass removes at most 1,000 eligible older events, so a backlog can take several passes to clear. This operational history policy is separate from dataset retention and the individual compliance audit records.

Reading source status and history

Last sync succeeded describes the latest recorded acquisition result; it does not confirm that a stream is currently connected. Data stale is shown separately when the backing table has exceeded its freshness threshold. Paused and failed acquisition states remain visible independently of table freshness.

New successful stream batches appear as minute summaries with batch and committed-row counts. Older individual events and errors remain individual; history is not rewritten. An event without a measured duration does not display a duration.

Storage is full: free space or add capacity. appears (recorded as storage_full) when the deployment's storage has no disk left, so a write or a new table was refused. Nothing from that sync was written and nothing is lost: a stream batch or a push delivery is retried, and the next scheduled run tries again. It never walks a source toward auto-pause. Your Scrydon operator fixes it by freeing space or adding capacity; once they have, sources resume on their own.

Your DLP policy applies to every row

Before any row is written to a source's table — in every mode, including query mirrors — the platform scans it with your organization's DLP policy (Settings → Governance → DLP), the same policy that governs capability calls. Each row is decided on its own:

• A row the policy rewrites (mask, redact, hash) is stored rewritten. An on-demand caller's answer is the stored rows: the same rewrite, without the rows set aside, which its persistence outcome counts (persistence.dataPolicy). Fetch now in a source's details says how many: 2 rows set aside by the data policy.
• In Enforce mode, a row with a finding is set aside and never stored. The rows beside it are written as usual.
• A single row larger than one scan (1 MiB) cannot be checked, so it is set aside and never stored, in every mode. The rows beside it are scanned and written as usual.
• Nothing to act on: the row is stored as it arrived.

The source's status says what the policy did: data_policy_applied: The organization's data policy redacted 2 rows and set aside 1 rows it blocks (LLM_BLOCKED_BY_DLP) before they were written. A row set aside as too large is named there too: 1 rows were set aside unwritten, each larger than the organization's data policy scans at once (1 MiB). It is a warning beside a successful sync, never a failure, so it never walks a source toward auto-pause. Findings are recorded in the audit log under the data source, without the matched values.

If the policy cannot be checked, nothing is written and nothing is dropped: a stream batch or a push delivery is retried, a manual upload or on-demand request reports a failure you can retry, and a scheduled poll run is recorded as data_policy_unavailable and the next run is the retry. Retrying never writes a batch twice.