Data Sources

A Declarative Data Source is a poll-mode table source encoded entirely as JSON-serializable configuration — an HTTP request spec, an itemsPath selector, a field-mapping table, and a column list. You author it once with defineDataSource; the platform's generic poll runtime drives it on every tick without executing any customer-supplied code.

When to use this SDK

Use the Data Source Authoring SDK when:

• You want to pull rows from a public REST API on a schedule and make them available as a typed table inside Scrydon.
• You need the source definition to live in a Pack so re-uploading the pack also refreshes the source (same idempotency contract as ontologies and process flows).
• You are building a demo or starter Pack and want the data tables to come pre-wired with no manual configuration.

Do not use this SDK if:

• Your ingest logic is conditional, stateful, or requires code (pagination cursors, OAuth token refresh, custom payload signing). For those, author a code-shipped source in the platform monorepo — it uses the same defineDataSource call but with a produce() function.
• The destination is not a table. Non-tabular ingestion paths are out of scope for this SDK today.

Key constraint: packs carry pure JSON — never functions. The bundler strips any produce() functions during serialization. A packable source must therefore be fully declarative — every field in the source definition must survive JSON.parse(JSON.stringify(...)) round-trip intact.

Install

bun add -d @scrydon/sdk-authoring zod

npm install --save-dev @scrydon/sdk-authoring zod

import { defineDataSource } from '@scrydon/sdk-authoring/integrations'

Anatomy of a declarative data source

A declarative data source is made up of five blocks:

The mapping DSL

Each mapping entry either copies a path directly or applies a named transform. The transform set is bounded and auditable — adding a new transform requires a reviewed code change to the platform, not pack data. Arbitrary expressions, eval, and sandboxed code are intentionally not supported.

The iso_from_epoch_offset hard case

Some REST APIs — the ADS-B family is the canonical example — do not return per-row absolute timestamps. Instead, the envelope carries a single now field (epoch milliseconds) and each row carries a seen field (seconds ago). The iso_from_epoch_offset transform bridges the two:

seenAt = new Date(envelope.now - row.seen * 1000).toISOString()

Configure it as:

seenAt: {
  path: "seen",                              // per-row offset field
  transform: "iso_from_epoch_offset",
  args: {
    basePath: "$.now",                       // envelope field — resolved against the response root
    baseUnit: "ms",                          // $.now is epoch milliseconds
    offsetUnit: "s",                         // row.seen is seconds
    direction: "subtract",                   // now − seen → absolute time
  },
},

If row.seen is missing or non-numeric, the runtime defaults the offset to 0 (base time exactly), matching the code-source convention seenSecondsAgo = typeof raw.seen === "number" ? raw.seen : 0.

A complete example

The following is the real adsb-lol-military-declarative source shipped in @scrydon/sdk-authoring. It pulls military aircraft positions from https://api.adsb.lol/v2/mil and maps them to a typed aircraft_position table. The golden parity test in the monorepo asserts this produces byte-for-byte identical rows to the equivalent code source.

import { defineDataSource } from '@scrydon/sdk-authoring/integrations'

export const adsbLolMilitaryDeclarative = defineDataSource({
  kind: "table",
  id: "adsb-lol-military-declarative",
  vendor: "adsb-lol",
  displayName: "ADS-B Lol — Military Aircraft (declarative)",
  scope: "global",
  table: {
    name: "aircraft_position",
    primaryKey: ["icao24", "seenAt"],
    timestampColumn: "seenAt",
    columns: [
      { name: "icao24",           dataType: "string",  isPrimaryKey: true },
      { name: "callsign",         dataType: "string",  nullable: true },
      { name: "registration",     dataType: "string",  nullable: true },
      { name: "aircraftType",     dataType: "string",  nullable: true },
      { name: "category",         dataType: "string",  nullable: true },
      { name: "latitude",         dataType: "decimal" },
      { name: "longitude",        dataType: "decimal" },
      { name: "altitudeFeet",     dataType: "int",     nullable: true },
      { name: "groundSpeedKnots", dataType: "double",  nullable: true },
      { name: "heading",          dataType: "double",  nullable: true },
      { name: "seenAt",           dataType: "timestamp", isPrimaryKey: true },
    ],
  },
  ingest: {
    mode: "poll",
    intervalSec: 60,
    minIntervalSec: 30,
    request: {
      url: "https://api.adsb.lol/v2/mil",
      method: "GET",
      headers: { accept: "application/json" },
    },
    response: { itemsPath: "$.ac" },
    filter: {
      // Drop rows missing hex (ICAO), lat, or lon — mirrors the code source guard.
      requireNonNull: ["hex", "lat", "lon"],
    },
    mapping: {
      icao24:           { path: "hex" },
      // "RCH123 " → "RCH123", "   " → null. Mirrors raw.flight?.trim() || null
      callsign:         { path: "flight", transform: "trim_to_null" },
      // undefined → null. Mirrors raw.r ?? null
      registration:     { path: "r" },
      // undefined → null. Mirrors raw.t ?? null
      aircraftType:     { path: "t" },
      // undefined → null. Mirrors raw.category ?? null
      category:         { path: "category" },
      latitude:         { path: "lat" },
      longitude:        { path: "lon" },
      // "ground" → 0, number → passthrough, else → null. Mirrors parseAltitude()
      altitudeFeet: {
        path: "alt_baro",
        transform: "value_map",
        args: { map: { ground: 0 }, passthrough: "number", default: null },
      },
      // finite number → itself, string/NaN/undefined → null
      groundSpeedKnots: { path: "gs",    transform: "number_or_null" },
      heading:          { path: "track", transform: "number_or_null" },
      // new Date(envelope.now - row.seen * 1000).toISOString()
      seenAt: {
        path: "seen",
        transform: "iso_from_epoch_offset",
        args: {
          basePath: "$.now",
          baseUnit: "ms",
          offsetUnit: "s",
          direction: "subtract",
        },
      },
    },
  },
})

defineDataSource is an identity function at runtime — it validates the manifest via the DataSourceManifestSchema Zod schema and derives a row validator from table.columns. The emitted definition is pure data: no produce() function, no closures, serializable as-is into data-source-<slug>/manifest.json.

Write modes

Every tick produces a batch of rows; ingest.writeMode controls how that batch lands in the table. It is install-only — it provisions the StarRocks key model at table-create time and cannot be changed afterwards (changing it requires re-installing the pack with an updated manifest). The default is upsert.

ingest: {
  mode: "poll",
  intervalSec: 60,
  writeMode: "changed-only", // omit for the "upsert" default
  // ...request, response, mapping
}

Retention

For time-series tables that grow continuously, declare table.ttlSec alongside table.timestampColumn. The platform partitions the StarRocks table by day on the timestamp column and keeps only the most recent ceil(ttlSec / 86400) daily partitions — older partitions are dropped automatically.

table: {
  name: "aircraft_position",
  primaryKey: ["icao24", "seenAt"],
  timestampColumn: "seenAt",   // the column partitions are cut on
  ttlSec: 604800,              // keep ~7 days of partitions
  columns: [ /* ... */ ],
}

ttlSec is the authoring default. An organization admin can override the effective retention per source from Settings → Platform → Data Sources (open a source's detail panel and edit "Retention") — the override is applied live, with no table rebuild. The detail panel also shows the source's write mode (read-only) and its recent sync history, and links straight to the backing table.

The detail panel's Danger zone also has Remove data source: it drops the backing table and all of its rows, deletes the source, and — when the originating pack contributed nothing else — removes the pack too. If the pack ships other content, the source is removed but the pack is kept (uninstall it from Settings → Packs). Removal is irreversible; re-installing the pack brings the source back.

Bundle layout

A Pack that ships a data source adds one data-source-<slug>/ subdir per source, each containing a single manifest.json. Multiple data sources may coexist in the same Pack — data-source is a repeatable content kind, like workflow.

<bundle>.scrydon-pack.tar.gz
├── pack.json                               # PackBundleManifestSchema
└── data-source-adsb-lol-military/
    └── manifest.json                       # DataSourceManifestSchema (pure JSON, no code)

The top-level pack.json lists the data source as a contents[] entry and includes "data-source" in installOrder:

import { defineScrydonPack } from '@scrydon/sdk-authoring/packs'
import { adsbLolMilitaryDeclarative } from './data-source-adsb-lol-military'

export default defineScrydonPack({
  manifestVersion: 1,
  package: {
    id: 'adsb-lol-military',
    name: 'ADS-B Lol Military Aircraft',
    version: '1.0.0',
  },
  contents: [
    {
      kind: 'data-source',
      path: 'data-source-adsb-lol-military',
      version: '1.0.0',
      required: true,
    },
  ],
  installOrder: ['data-source'],
  metadata: { isSystemPack: false, isDemoPack: false, tags: ['adsb', 'aircraft'] },
})

Build, inspect, upload

bunx @scrydon/sdk-authoring pack build src/pack.ts --outDir dist
# → dist/<package.id>-<package.version>.scrydon-pack.tar.gz

bunx @scrydon/sdk-authoring pack inspect dist/adsb-lol-military-1.0.0.scrydon-pack.tar.gz

curl -X POST "$AGENTIC_URL/api/packs/import?organizationId=$ORG_ID" \
  -H "Cookie: $SESSION_COOKIE" \
  -F "file=@dist/adsb-lol-military-1.0.0.scrydon-pack.tar.gz"

Data sources that need a credential

Some APIs require an account or API key. For those, use the request.authRef field — a named reference to a credential connection configured in your org's settings. The manifest never carries an inline secret; authRef is a pointer, not a value. (The DataSourceManifestSchema rejects any attempt to embed an Authorization header or API key directly in the manifest.)

ingest: {
  mode: "poll",
  intervalSec: 300,
  request: {
    url: "https://api.example.com/v1/records",
    method: "GET",
    authRef: "example-api-key",   // named reference — NOT an inline token
  },
  // ...
}

How the connection flows:

1. You (or your org admin) connect the account in org settings, creating an enabled credential connection named "example-api-key".
2. On each tick, the platform resolves that connection server-side and attaches the credential to the outbound request — your manifest never sees the secret.
3. If no enabled connection exists yet, the tick returns HTTP 412 data_source_connection_required — a clear signal to connect the account first. No data is fetched, and no error is silently swallowed.
4. Once a connection is enabled, subsequent ticks resolve it automatically and run.

No authRef? A source with no authRef (public API, no authentication needed) ticks immediately — no credential setup required.

Security

Resource limits enforced per-tick:

Where to next