Scrydon
Authoring: Data SourcesExamples

ADS-B military aircraft — example data source

Install a declarative data source via a Scrydon Pack, sync it, and operate it from Analytics → Data Sources — sync history, live retention, the backing table, and removal. End-to-end ADS-B example with both UI and developer paths.

Downloads

Both files are emitted by the docs build script (apps/docs/scripts/build-pack-examples.ts) from a single source manifest at apps/docs/examples/scrydon-pack-adsb-aircraft/manifest.ts. They regenerate on every bun run dev / bun run build of the docs app — they're the same bytes, just two distribution shapes for the same data source.

FileUse it to …
📦 scrydon-pack-adsb-aircraft.scrydon-pack.tar.gzRecommended — install through the UI. Two stages: org admin admits the pack in Settings → Packs (catalog only — nothing is materialized yet); workspace user picks it from Analytics → Marketplace to instantiate the data source into their environment.
📥 adsb-lol-military-declarative.jsonApply the bare declarative manifest directly via curl to POST /api/table/data-sources/apply — the developer path that skips the pack wrapper. Useful for scripted testing.

Pack install is two-stage by design — see ADR 2026-05-30 for the rationale. Stage 1 catalogues the pack org-wide; Stage 2 instantiates the data source into one workspace environment.

Stage 1 — Org admin uploads the pack

  1. As an org admin, open Settings → Packs.
  2. Drag the downloaded scrydon-pack-adsb-aircraft.scrydon-pack.tar.gz onto the drop zone (or click to pick). The dialog shows the pack id scrydon-pack-adsb-aircraft, version 1.0.0, and a one-line contributes summary: 1 data source.
  3. Tick I understand this pack is unsigned… and click Upload. The pack is admitted to the org catalog. No data_source row is created yet — data-source-only packs defer materialization to Stage 2 (the upload completes silently and the pack appears in the marketplace).

Stage 2 — Workspace user instantiates the data source

  1. As a workspace user, open apps/analytics and confirm you have a workspace + environment selected (top-right switcher).
  2. Open Marketplace in the sidebar. The scrydon-pack-adsb-aircraft card appears with a single Install in this environment button (the pack ships exactly one data source and it is not yet present in this env).
  3. Click Install in this environment. The platform calls packs.install({ contentKinds: ["data-source"], workspaceEnvironmentId }) → api-table records a row in data_source scoped to your env.
  4. Open Data Sources in the sidebar — the new row appears with Source: adsb-lol-military-declarative, Vendor: adsb-lol, Interval: 60s, Status: Pending.
  5. Open the row's action menu and click Sync now. Expect a green toast with the rows-written count; Status flips to Healthy and Last sync populates after the auto-refetch.

The marketplace install path (Better Auth packs.install with contentKinds: ["data-source"]) reaches the same api-table apply endpoint via the installResolvedPack data-source delegate, so any subsequent installs of the same pack into different environments produce identical rows.

Path B — apply the JSON directly (developer)

Skip the pack-authoring tooling and apply the bare manifest to api-table's internal /data-sources/apply endpoint. This exercises the same code path as Path A's installDataSource hook — both converge on POST /api/table/data-sources/apply.

This is the fastest end-to-end exercise of:

  • the api-table data-source domain (row in data_source, sync-state columns updated on each tick),
  • the generic poll runtime in @scrydon/better-auth-integrations/host/declarative-poll/runtime.ts (HTTP fetch → itemsPath select → field mapping → row validation),
  • the Analytics → Data Sources view (status badge, last-sync time, rows-written counter, "Sync now" action).

The manifest below is the canonical adsb-lol-military-declarative source from @scrydon/sdk-authoring, serialized to plain JSON. The golden parity test in the monorepo asserts this produces byte-for-byte identical rows to the equivalent code source over the same fixture — so once "Sync now" succeeds you're exercising the production poll runtime against a real public API.

The endpoint is internal-auth gated (x-scrydon-internal-dev: 1 in dev, Dapr SPIFFE in prod) and accepts callers holding the data-source:apply capability — platform (reconciler/tick), agentic (pack-import), and api-platform (the Better Auth marketplace delegate). The curl call below uses the dev header; in prod the call lands through one of those three Dapr-attested callers.

Prerequisites

You have a local Scrydon stack running. The api-table service should answer:

curl -fsS http://localhost:7500/api/table/health

You need an organization id to scope the data source. Grab it from the organization table in your auth database, or from the URL of any organization-scoped page in the platform UI.

export ORG="<your-org-uuid>"

The manifest

This is the complete declarative manifest. It is pure JSON — no functions, no code. The pack bundler emits exactly this shape from defineDataSource({...}).

adsb-lol-military-declarative.json
{
  "kind": "table",
  "id": "adsb-lol-military-declarative",
  "vendor": "adsb-lol",
  "displayName": "ADS-B Lol — Military Aircraft (declarative)",
  "scope": "global",
  "table": {
    "name": "aircraft_position",
    "primaryKey": ["icao24", "seenAt"],
    "timestampColumn": "seenAt",
    "columns": [
      { "name": "icao24",           "dataType": "string",    "isPrimaryKey": true },
      { "name": "callsign",         "dataType": "string",    "nullable": true },
      { "name": "registration",     "dataType": "string",    "nullable": true },
      { "name": "aircraftType",     "dataType": "string",    "nullable": true },
      { "name": "category",         "dataType": "string",    "nullable": true },
      { "name": "latitude",         "dataType": "decimal" },
      { "name": "longitude",        "dataType": "decimal" },
      { "name": "altitudeFeet",     "dataType": "int",       "nullable": true },
      { "name": "groundSpeedKnots", "dataType": "double",    "nullable": true },
      { "name": "heading",          "dataType": "double",    "nullable": true },
      { "name": "seenAt",           "dataType": "timestamp", "isPrimaryKey": true }
    ]
  },
  "ingest": {
    "mode": "poll",
    "intervalSec": 60,
    "minIntervalSec": 30,
    "request": {
      "url": "https://api.adsb.lol/v2/mil",
      "method": "GET",
      "headers": { "accept": "application/json" }
    },
    "response": { "itemsPath": "$.ac" },
    "filter": {
      "requireNonNull": ["hex", "lat", "lon"]
    },
    "mapping": {
      "icao24":          { "path": "hex" },
      "callsign":        { "path": "flight",   "transform": "trim_to_null" },
      "registration":    { "path": "r" },
      "aircraftType":    { "path": "t" },
      "category":        { "path": "category" },
      "latitude":        { "path": "lat" },
      "longitude":       { "path": "lon" },
      "altitudeFeet": {
        "path": "alt_baro",
        "transform": "value_map",
        "args": { "map": { "ground": 0 }, "passthrough": "number", "default": null }
      },
      "groundSpeedKnots": { "path": "gs",    "transform": "number_or_null" },
      "heading":          { "path": "track", "transform": "number_or_null" },
      "seenAt": {
        "path": "seen",
        "transform": "iso_from_epoch_offset",
        "args": {
          "basePath": "$.now",
          "baseUnit": "ms",
          "offsetUnit": "s",
          "direction": "subtract"
        }
      }
    }
  }
}

What each block does

BlockPurpose
kind: "table"The only supported source kind today — declares a row-shaped destination.
idStable identity. api-table upserts on (organization_id, id) — re-applying the same manifest updates the row in place.
vendorFree-form vendor slug shown in the Data Sources list.
scope: "global"One source for the whole org. (Per-workspace scoping is a roadmap item.)
table.primaryKeyComposite PK used to dedup poll results against the silver table.
table.columns[]The exact column list — dataType + nullable + isPrimaryKey map straight to the silver-table DDL.
ingest.intervalSec / minIntervalSecTick cadence (60s here) and a floor enforced by the scheduler.
ingest.requestPlain HTTP request spec. No code, no signing — headers and query are static strings.
ingest.response.itemsPathJSONPath-like selector reaching the row array in the response envelope.
ingest.filter.requireNonNullRow-level filter: skips items whose listed paths are null/undefined.
ingest.mapping[<column>]One entry per table.columns[] name. path reads the source field; optional transform + args reshape it. The supported transforms are documented in data-sources/index.mdx#field-mappings.

Apply the manifest

Download the JSON file linked above, then assemble the apply body. The cleanest path uses jq so the JSON is composed safely (no shell escaping headaches):

# 1) Download the manifest once.
curl -fsS -o adsb-lol-military-declarative.json \
  http://localhost:4001/static/scrydon-pack-adsb-aircraft/adsb-lol-military-declarative.json

# 2) Compose the apply body and POST it.
jq -n \
  --arg org "$ORG" \
  --slurpfile manifest adsb-lol-military-declarative.json \
  '{ organizationId: $org, packId: "scrydon-pack-adsb-aircraft", packVersion: "1.0.0", manifest: $manifest[0] }' \
  | curl -fsS -X POST http://localhost:7500/api/table/data-sources/apply \
      -H 'content-type: application/json' \
      -H 'x-scrydon-internal-dev: 1' \
      -d @-

Expected response:

{ "sourceId": "adsb-lol-military-declarative" }

If you'd rather not depend on jq, save the full request body to disk once and POST it directly:

ORG="<your-org-uuid>"
jq -n --arg org "$ORG" \
  --slurpfile manifest adsb-lol-military-declarative.json \
  '{ organizationId: $org, packId: "scrydon-pack-adsb-aircraft", packVersion: "1.0.0", manifest: $manifest[0] }' \
  > apply-body.json

curl -fsS -X POST http://localhost:7500/api/table/data-sources/apply \
  -H 'content-type: application/json' \
  -H 'x-scrydon-internal-dev: 1' \
  -d @apply-body.json

Operate & observe it (Analytics → Data Sources)

Once the source is instantiated in your environment, everything below is point-and-click — no SDK, no API calls. This is the full operator experience for any data source, using the ADS-B example as a concrete subject.

As a workspace user, go to apps/analytics → Data Sources. Each installed source in the current environment is a row showing Source, Vendor, Interval, Status, Last sync, Rows, and Last error. A freshly installed source shows Status: Pending until its first sync. The view scopes to the workspace environment selected in the top-right switcher.

Open the row's action menu and choose Sync now. The badge switches to Syncing… while the poll runs — an external fetch can take a few seconds up to a minute — then the row refreshes on its own: Status → Healthy, Last sync updates, and Rows shows how many were written. You don't have to wait for the scheduled interval; this runs the same tick on demand.

Click the row to open its detail panel, which shows:

  • Write mode — how new rows land (upsert, changed-only, append, or replace). Read-only: it is fixed when the source is installed (see Write modes).
  • Retention — an editable control (covered next).
  • Sync history — the most recent ticks: when each ran, the event (e.g. rows upserted), how many rows it touched, and any error message.
  • Open table — a shortcut straight to the backing table, where you can browse and query the ingested rows. (Appears once the first sync has provisioned the table.)

In the detail panel, set Retention to the window you want to keep — for the ADS-B example, e.g. 7 days — and save. The platform keeps only the most recent daily partitions of the table and drops older ones automatically; the change applies live, with no table rebuild and no data re-load. Clear the field to revert to the source's authored default. See Retention for how the default is authored.

At the bottom of the detail panel, the Danger zone has Remove data source. Removing is destructive: after you confirm, the platform drops the backing table and all of its rows, deletes the source, and stops its schedule. If the pack that contributed this source contributed nothing else, the originating pack is removed too — so for a single-source pack like this ADS-B example, "Remove data source" cleans up both the source and the pack in one step.

If the pack also ships other content (an ontology, workflows, or more data sources), the source is still removed but the pack is kept — a toast tells you so — and you uninstall the pack itself from Settings → Packs when you're ready. This action cannot be undone; re-installing the pack is the way to bring the source back.

If a sync fails

If Status flips to Error, hover the Last error cell (or read it in the detail panel). The most common causes:

ErrorWhat it meansWhat to do
data_source_connection_requiredThe source needs a credential (its request references an authRef), but no enabled connection exists for your org.Connect the account in org settings under the name the pack expects, then sync again. This ADS-B example uses no credential, so you should not see this.
invalid_installed_manifestThe stored manifest failed full re-validation at tick time — e.g. a bad column type or transform name.Re-install the pack with a corrected manifest.
egress_deniedThe egress guard blocked the request URL (it must be https:// and resolve to a public host).Confirm the source's host is reachable and allowed in your environment.
upstream 5xxThe external API was momentarily unavailable or rate-limited (api.adsb.lol occasionally is).Click Sync now again — ticks are idempotent.

Developer aside — verify the row + sync state in the database

The operator UI above is the supported surface. If you're developing locally and want to confirm the underlying row directly:

SELECT source_id, status, last_sync_at, last_rows_written, last_error, consecutive_failures
FROM data_source
WHERE organization_id = :ORG;

After a successful Sync now:

       source_id            | status | last_sync_at        | last_rows_written | last_error | consecutive_failures
-----------------------------+--------+---------------------+-------------------+------------+----------------------
 adsb-lol-military-declarative | ok     | 2026-05-28 14:02:11 |               37  |            |                    0

Where this manifest comes from

The TypeScript source lives at packages/sdk-authoring/src/integrations/data-source/adsb-lol-military-declarative.ts — calling defineDataSource({...}) with the same shape. The pack bundler serializes the call's argument to JSON and emits it as a data-source/ subdir inside a .scrydon-pack.tar.gz. The /data-sources/apply endpoint accepts that JSON directly, which is what this page demonstrates: the same manifest, just without the pack wrapper.

If you want to ship the source inside a pack instead of applying it manually, follow the Author flow in sdks/authoring/data-sources.

On this page

On this page