ADS-B military aircraft — example data source
Install a declarative data source via a Scrydon Pack, sync it, and operate it from Analytics → Data Sources — sync history, live retention, the backing table, and removal. End-to-end ADS-B example with both UI and developer paths.
Downloads
Both files are emitted by the docs build script (apps/docs/scripts/build-pack-examples.ts) from a single source manifest at apps/docs/examples/scrydon-pack-adsb-aircraft/manifest.ts. They regenerate on every bun run dev / bun run build of the docs app — they're the same bytes, just two distribution shapes for the same data source.
| File | Use it to … |
|---|---|
| 📦 scrydon-pack-adsb-aircraft.scrydon-pack.tar.gz | Recommended — install through the UI. Two stages: org admin admits the pack in Settings → Packs (catalog only — nothing is materialized yet); workspace user picks it from Analytics → Marketplace to instantiate the data source into their environment. |
| 📥 adsb-lol-military-declarative.json | Apply the bare declarative manifest directly via curl to POST /api/table/data-sources/apply — the developer path that skips the pack wrapper. Useful for scripted testing. |
Path A — install through the UI (recommended)
Pack install is two-stage by design — see ADR 2026-05-30 for the rationale. Stage 1 catalogues the pack org-wide; Stage 2 instantiates the data source into one workspace environment.
Stage 1 — Org admin uploads the pack
- As an org admin, open Settings → Packs.
- Drag the downloaded
scrydon-pack-adsb-aircraft.scrydon-pack.tar.gzonto the drop zone (or click to pick). The dialog shows the pack idscrydon-pack-adsb-aircraft, version1.0.0, and a one-line contributes summary: 1 data source. - Tick I understand this pack is unsigned… and click Upload. The pack is admitted to the org catalog. No
data_sourcerow is created yet — data-source-only packs defer materialization to Stage 2 (the upload completes silently and the pack appears in the marketplace).
Stage 2 — Workspace user instantiates the data source
- As a workspace user, open
apps/analyticsand confirm you have a workspace + environment selected (top-right switcher). - Open Marketplace in the sidebar. The
scrydon-pack-adsb-aircraftcard appears with a single Install in this environment button (the pack ships exactly one data source and it is not yet present in this env). - Click Install in this environment. The platform calls
packs.install({ contentKinds: ["data-source"], workspaceEnvironmentId })→ api-table records a row indata_sourcescoped to your env. - Open Data Sources in the sidebar — the new row appears with Source:
adsb-lol-military-declarative, Vendor:adsb-lol, Interval: 60s, Status:Pending. - Open the row's action menu and click Sync now. Expect a green toast with the rows-written count; Status flips to
Healthyand Last sync populates after the auto-refetch.
The marketplace install path (Better Auth packs.install with contentKinds: ["data-source"]) reaches the same api-table apply endpoint via the installResolvedPack data-source delegate, so any subsequent installs of the same pack into different environments produce identical rows.
Path B — apply the JSON directly (developer)
Skip the pack-authoring tooling and apply the bare manifest to api-table's internal /data-sources/apply endpoint. This exercises the same code path as Path A's installDataSource hook — both converge on POST /api/table/data-sources/apply.
This is the fastest end-to-end exercise of:
- the api-table data-source domain (row in
data_source, sync-state columns updated on each tick), - the generic poll runtime in
@scrydon/better-auth-integrations/host/declarative-poll/runtime.ts(HTTP fetch →itemsPathselect → field mapping → row validation), - the Analytics → Data Sources view (status badge, last-sync time, rows-written counter, "Sync now" action).
The manifest below is the canonical adsb-lol-military-declarative source from @scrydon/sdk-authoring, serialized to plain JSON. The golden parity test in the monorepo asserts this produces byte-for-byte identical rows to the equivalent code source over the same fixture — so once "Sync now" succeeds you're exercising the production poll runtime against a real public API.
The endpoint is internal-auth gated (x-scrydon-internal-dev: 1 in dev, Dapr SPIFFE in prod) and accepts callers holding the data-source:apply capability — platform (reconciler/tick), agentic (pack-import), and api-platform (the Better Auth marketplace delegate). The curl call below uses the dev header; in prod the call lands through one of those three Dapr-attested callers.
Prerequisites
You have a local Scrydon stack running. The api-table service should answer:
curl -fsS http://localhost:7500/api/table/healthYou need an organization id to scope the data source. Grab it from the organization table in your auth database, or from the URL of any organization-scoped page in the platform UI.
export ORG="<your-org-uuid>"The manifest
This is the complete declarative manifest. It is pure JSON — no functions, no code. The pack bundler emits exactly this shape from defineDataSource({...}).
{
"kind": "table",
"id": "adsb-lol-military-declarative",
"vendor": "adsb-lol",
"displayName": "ADS-B Lol — Military Aircraft (declarative)",
"scope": "global",
"table": {
"name": "aircraft_position",
"primaryKey": ["icao24", "seenAt"],
"timestampColumn": "seenAt",
"columns": [
{ "name": "icao24", "dataType": "string", "isPrimaryKey": true },
{ "name": "callsign", "dataType": "string", "nullable": true },
{ "name": "registration", "dataType": "string", "nullable": true },
{ "name": "aircraftType", "dataType": "string", "nullable": true },
{ "name": "category", "dataType": "string", "nullable": true },
{ "name": "latitude", "dataType": "decimal" },
{ "name": "longitude", "dataType": "decimal" },
{ "name": "altitudeFeet", "dataType": "int", "nullable": true },
{ "name": "groundSpeedKnots", "dataType": "double", "nullable": true },
{ "name": "heading", "dataType": "double", "nullable": true },
{ "name": "seenAt", "dataType": "timestamp", "isPrimaryKey": true }
]
},
"ingest": {
"mode": "poll",
"intervalSec": 60,
"minIntervalSec": 30,
"request": {
"url": "https://api.adsb.lol/v2/mil",
"method": "GET",
"headers": { "accept": "application/json" }
},
"response": { "itemsPath": "$.ac" },
"filter": {
"requireNonNull": ["hex", "lat", "lon"]
},
"mapping": {
"icao24": { "path": "hex" },
"callsign": { "path": "flight", "transform": "trim_to_null" },
"registration": { "path": "r" },
"aircraftType": { "path": "t" },
"category": { "path": "category" },
"latitude": { "path": "lat" },
"longitude": { "path": "lon" },
"altitudeFeet": {
"path": "alt_baro",
"transform": "value_map",
"args": { "map": { "ground": 0 }, "passthrough": "number", "default": null }
},
"groundSpeedKnots": { "path": "gs", "transform": "number_or_null" },
"heading": { "path": "track", "transform": "number_or_null" },
"seenAt": {
"path": "seen",
"transform": "iso_from_epoch_offset",
"args": {
"basePath": "$.now",
"baseUnit": "ms",
"offsetUnit": "s",
"direction": "subtract"
}
}
}
}
}What each block does
| Block | Purpose |
|---|---|
kind: "table" | The only supported source kind today — declares a row-shaped destination. |
id | Stable identity. api-table upserts on (organization_id, id) — re-applying the same manifest updates the row in place. |
vendor | Free-form vendor slug shown in the Data Sources list. |
scope: "global" | One source for the whole org. (Per-workspace scoping is a roadmap item.) |
table.primaryKey | Composite PK used to dedup poll results against the silver table. |
table.columns[] | The exact column list — dataType + nullable + isPrimaryKey map straight to the silver-table DDL. |
ingest.intervalSec / minIntervalSec | Tick cadence (60s here) and a floor enforced by the scheduler. |
ingest.request | Plain HTTP request spec. No code, no signing — headers and query are static strings. |
ingest.response.itemsPath | JSONPath-like selector reaching the row array in the response envelope. |
ingest.filter.requireNonNull | Row-level filter: skips items whose listed paths are null/undefined. |
ingest.mapping[<column>] | One entry per table.columns[] name. path reads the source field; optional transform + args reshape it. The supported transforms are documented in data-sources/index.mdx#field-mappings. |
Apply the manifest
Download the JSON file linked above, then assemble the apply body. The cleanest path uses jq so the JSON is composed safely (no shell escaping headaches):
# 1) Download the manifest once.
curl -fsS -o adsb-lol-military-declarative.json \
http://localhost:4001/static/scrydon-pack-adsb-aircraft/adsb-lol-military-declarative.json
# 2) Compose the apply body and POST it.
jq -n \
--arg org "$ORG" \
--slurpfile manifest adsb-lol-military-declarative.json \
'{ organizationId: $org, packId: "scrydon-pack-adsb-aircraft", packVersion: "1.0.0", manifest: $manifest[0] }' \
| curl -fsS -X POST http://localhost:7500/api/table/data-sources/apply \
-H 'content-type: application/json' \
-H 'x-scrydon-internal-dev: 1' \
-d @-Expected response:
{ "sourceId": "adsb-lol-military-declarative" }If you'd rather not depend on jq, save the full request body to disk once and POST it directly:
ORG="<your-org-uuid>"
jq -n --arg org "$ORG" \
--slurpfile manifest adsb-lol-military-declarative.json \
'{ organizationId: $org, packId: "scrydon-pack-adsb-aircraft", packVersion: "1.0.0", manifest: $manifest[0] }' \
> apply-body.json
curl -fsS -X POST http://localhost:7500/api/table/data-sources/apply \
-H 'content-type: application/json' \
-H 'x-scrydon-internal-dev: 1' \
-d @apply-body.jsonOperate & observe it (Analytics → Data Sources)
Once the source is instantiated in your environment, everything below is point-and-click — no SDK, no API calls. This is the full operator experience for any data source, using the ADS-B example as a concrete subject.
As a workspace user, go to apps/analytics → Data Sources. Each installed source in the current environment is a row showing Source, Vendor, Interval, Status, Last sync, Rows, and Last error. A freshly installed source shows Status: Pending until its first sync. The view scopes to the workspace environment selected in the top-right switcher.
Open the row's ⋯ action menu and choose Sync now. The badge switches to Syncing… while the poll runs — an external fetch can take a few seconds up to a minute — then the row refreshes on its own: Status → Healthy, Last sync updates, and Rows shows how many were written. You don't have to wait for the scheduled interval; this runs the same tick on demand.
Click the row to open its detail panel, which shows:
- Write mode — how new rows land (
upsert,changed-only,append, orreplace). Read-only: it is fixed when the source is installed (see Write modes). - Retention — an editable control (covered next).
- Sync history — the most recent ticks: when each ran, the event (e.g. rows upserted), how many rows it touched, and any error message.
- Open table — a shortcut straight to the backing table, where you can browse and query the ingested rows. (Appears once the first sync has provisioned the table.)
In the detail panel, set Retention to the window you want to keep — for the ADS-B example, e.g. 7 days — and save. The platform keeps only the most recent daily partitions of the table and drops older ones automatically; the change applies live, with no table rebuild and no data re-load. Clear the field to revert to the source's authored default. See Retention for how the default is authored.
At the bottom of the detail panel, the Danger zone has Remove data source. Removing is destructive: after you confirm, the platform drops the backing table and all of its rows, deletes the source, and stops its schedule. If the pack that contributed this source contributed nothing else, the originating pack is removed too — so for a single-source pack like this ADS-B example, "Remove data source" cleans up both the source and the pack in one step.
If the pack also ships other content (an ontology, workflows, or more data sources), the source is still removed but the pack is kept — a toast tells you so — and you uninstall the pack itself from Settings → Packs when you're ready. This action cannot be undone; re-installing the pack is the way to bring the source back.
If a sync fails
If Status flips to Error, hover the Last error cell (or read it in the detail panel). The most common causes:
| Error | What it means | What to do |
|---|---|---|
data_source_connection_required | The source needs a credential (its request references an authRef), but no enabled connection exists for your org. | Connect the account in org settings under the name the pack expects, then sync again. This ADS-B example uses no credential, so you should not see this. |
invalid_installed_manifest | The stored manifest failed full re-validation at tick time — e.g. a bad column type or transform name. | Re-install the pack with a corrected manifest. |
egress_denied | The egress guard blocked the request URL (it must be https:// and resolve to a public host). | Confirm the source's host is reachable and allowed in your environment. |
upstream 5xx | The external API was momentarily unavailable or rate-limited (api.adsb.lol occasionally is). | Click Sync now again — ticks are idempotent. |
Developer aside — verify the row + sync state in the database
The operator UI above is the supported surface. If you're developing locally and want to confirm the underlying row directly:
SELECT source_id, status, last_sync_at, last_rows_written, last_error, consecutive_failures
FROM data_source
WHERE organization_id = :ORG;After a successful Sync now:
source_id | status | last_sync_at | last_rows_written | last_error | consecutive_failures
-----------------------------+--------+---------------------+-------------------+------------+----------------------
adsb-lol-military-declarative | ok | 2026-05-28 14:02:11 | 37 | | 0Where this manifest comes from
The TypeScript source lives at packages/sdk-authoring/src/integrations/data-source/adsb-lol-military-declarative.ts — calling defineDataSource({...}) with the same shape. The pack bundler serializes the call's argument to JSON and emits it as a data-source/ subdir inside a .scrydon-pack.tar.gz. The /data-sources/apply endpoint accepts that JSON directly, which is what this page demonstrates: the same manifest, just without the pack wrapper.
If you want to ship the source inside a pack instead of applying it manually, follow the Author flow in sdks/authoring/data-sources.