Data sources
Add a data source for each connection shared with you, name the table it writes, and keep it running — without being an organization administrator.
A data source brings rows from a system into a table in your workspace: a Home Assistant instance, a PostgreSQL database, an air-quality feed. Two people share the work:
- An organization administrator installs the extension and adds its connections in Platform → Extensions — the address and the credential — and decides who may use each connection. Connections come first: they can be added before any data source that reads them exists. Using a connection never shows its credential.
- A data engineer adds data sources in Analytics, through the connections shared with them, and keeps them running.
Who can add and change a data source
You can add a data source, and change its connection, its schedule, the tables it mirrors and what happens when they gain columns — or remove it — when both are true:
- you may write in the workspace (you are a member, not a viewer), and
- the connection it reads has been shared with you (Use), or you administer the organization.
An organization administrator can do all of it everywhere. You can only change or remove a data source whose connection is shared with you.
Whether you may install a source in your environment is the same decision the Marketplace install makes, so you are told before anything is installed whether adding it installs it, or whether to ask an administrator to install it.
Ask for access
If no connection of a source is shared with you, ask for access. Every organization administrator gets a notification — and an email, when they receive notification emails — that opens the source's connections in Platform with you offered under Who can use it, so sharing a connection with you is one click on Share. The request is recorded in the Audit Log under your name. You can ask once a day for each source and environment; asking again sooner tells you that you already asked.
Add a data source
Each connection you pick becomes its own data source, with its own table, schedule and status: two connections, two data sources. A connection that already feeds a data source of this source can be picked again: the new data source is named apart from the existing one, and you choose its table in Table mappings as for any other. Everything happens in one sheet on the Data Sources page; nothing is created until you press Add.
In Analytics → Data Sources, select Add data source and search for the source. Sources are grouped by how their data arrives — a database Scrydon copies on a schedule is grouped apart from one it reads where it is, and in each group the sources you can add come first. A source that is not installed in your environment yet says installs on Add when you may install it; when you may not, the sheet says so and an organization administrator installs it. A source already added in another environment of your organization is listed but cannot be added: a data source lives in one environment at a time, and the sheet says so before you press Add. A source that reads from no connection is one data source, so once it is added in your environment it is listed as Already added: choosing it says so and offers Open data source instead of Add.
The dropdown lists the connections shared with you in this environment, under the environment's name, each with whether it is ready. One that is not ready stays in the list, says why, and cannot be picked — including a connection saved for All environments, which a database source never reads: an organization administrator gets Edit connection, which opens it in Platform to choose its environment, and anyone else Ask for access. If none is shared, the dropdown says No connections shared with you, the sheet names the administrators and offers Ask for access (see above), and there is nothing to add yet; an organization administrator, who can use every connection, is offered Add a connection instead. When the source already running here could use several connections, choose the one it keeps.
One line says where each data source's rows land, each table marked (new) or (reuse). Edit mappings opens Table mappings beside the sheet: one naming rule names every new table, and each table is a field where you type another name to create that table instead, or pick an existing table you can write to (see Reusing an existing table). To leave tables out, select their rows and choose Leave out; a table left out says Left out until you give it somewhere to land. When the name the rule gives is already an existing table that fits, the data source reuses that table instead of creating a second one of the same name. A name that is taken by a table that does not fit, not a valid table name or used twice is listed under Needs a decision. Done keeps what you decided; Back returns to the sheet as it was. When the source is installed by this add, the first connection's data source writes the source's own table. A source that reads from no connection is one data source with one table: the sheet offers its Name and its Table, each starting at the source's own. A table name you change is checked as you type like any new table's — allowed characters, not taken — and the line says where its rows land.
One line holds the defaults: how often the data sources run (see Schedule), how long their tables keep rows (an organization administrator sets this), and their classification. You can raise the classification when the data is more sensitive; lowering it below your organization's floor for unlabelled data is for an administrator, so only an administrator is offered a label below the floor. The classification is the one the table is created with, and is recorded in the Audit Log under your name. Left unset, a new table is created with your organization's floor — never below it. The line ends alerts on: the data sources alert when their state changes and are in the daily digest (see Alerts and the daily digest). Turn Alerts off to add them without either; the change is recorded in the Audit Log under your name.
Beside Add, the sheet says what it creates — for example "Creates 2 sources, 1 table", where a reused table is not counted — or, while Add is off, why: no connection picked yet, no connection that can be picked, no table chosen, or what is still to decide, named with what to do — for example "1 name already taken — choose New or Reuse". Add stays off for as long as the line says why. Every batch is scanned for sensitive data before it is written, and the run is recorded under your name. The sheet then shows what each data source came to — one that is refused says why and never stops the others, and you can try the refused ones again — and its first run: when the data sources started and how often they run, then one card per data source naming the table it writes and whether it is Live, still on its First sync, or failed. A push data source says before Add that it receives data at its own push endpoint; until its first delivery arrives, its card says Waiting for its first delivery, shows the path its sender posts to, and Push endpoint shows the rest of what the sender needs (never the secret). Each card says, in sentences, how many rows the run read and from which connection (for example "Read 120 rows from Home", when you may use that connection), what the scan for sensitive data masked or set aside, by kind and count but never the value (for example "Scanned: 1 email address masked", or rows set aside, or "nothing masked or set aside"; a value a rule only audits is not counted there, and the Audit Log records it) and how many rows it wrote — never before they have landed. A card always shows its newest run as that run went, never another run's counts: one that found nothing to change says No changes to write, and one that failed says why (for example "The run failed: …", or why it paused) beside Failed or Paused, rather than that it waits for its first run. A database data source shows one line per table you chose instead, with the table it lands in, when its last run ended (for example "· 14:03"), and where its run is: reading, then what the scan did to the rows written so far ("Scanned for sensitive data: nothing masked or set aside"), writing its rows ("Writing 64 rows…") or written, and Open table once the table exists. Sync this table on a line syncs that table alone, now: it does not change when the data source next runs on its schedule, and a failure there does not count toward the failures that pause the data source — the line itself says how its run went. Once rows have landed, Preview rows shows the first rows the data source wrote, Open table opens its table, and Sync now runs a scheduled data source again and follows that run. Below the cards, one Activity list covers all of them, newest first, and filters by Info, Warning or Error. It names who added the data sources ("You added Home sensors and Office sensors"; a data source added before people were recorded, or by an automatic install, names nobody) and, for each run, what it read before what it wrote. While their alerts are on, the sheet says so under the list. The same view is a data source's Activity tab, headed Latest run once the data source has finished a run.
Naming a destination table
A destination table name:
- uses lowercase letters, digits and single underscores, and starts with a letter;
- is at most 30 characters long;
- does not start with
agent_, which is kept for tables agents create, ormirror_, which the platform gives a database table nobody named.
A new table is written by one data source. A name another data source in the environment already writes — or a table of that name that already exists — is refused, and you choose another, or you reuse that table on purpose (below). The table a data source writes is set when it is created; changing its connection later does not move it.
Reusing an existing table
Several connections of a source can write into one table — for example every site's weather station into one weather_all table. In Table mappings, open a table's dropdown: Existing tables you can write to lists the tables of your environment your own write would be allowed into, each saying whether it fits. Pick one for each connection that should share it, or select rows and use Reuse… to point them all at one table.
A table fits when the source's rows land in it exactly as they would in a table of its own:
- it is a data source's own table (not one you uploaded, a database mirror or an on-demand fetch), and this source is fetched on a schedule, streamed, pushed or uploaded (a database's tables reuse other database copies instead — see Choosing a database source's tables);
- nobody replaces it: a source that replaces its whole table on every run cannot share one, and no source can share a table such a source writes;
- it has the same key — and, for a table that updates rows by key, the connection is part of that key, so two connections' rows never overwrite each other (a keyed table made before connections could share tables does not fit);
- each column the source writes holds in the table's column as it is or wider (a whole number fits a larger whole number or a decimal), and the table needs no value the source does not write. A column the table lacks is added to it, empty for the rows it already has; one the source requires cannot be;
- it is labelled at least as high as the source's data. A table labelled below the source's data never takes it, and you cannot later raise the source's classification above the table it reuses.
Each table that fits says what it gains — Same columns, or how many optional columns it adds — and how many data sources already write there, for example "Same columns · 1 source writes here". A table that does not fit stays in the list, marked Doesn't fit, cannot be picked, and says why. On the Tables page, grouped by data source, a table several data sources write is listed under each of them, marked Shared, and each group counts it; the page's table count counts it once, and a line above the groups says that the group counts can therefore add up to more than the total. Each data source's Activity shows its own runs into a table it reuses, streamed ones included. Removing a data source never deletes a table: it stops writing, and the table and every row in it stay — whether the data source made the table or reused it, and while another data source still writes it.
Row tags
Every table a data source writes carries four extra columns, filled in by the platform on every row it writes:
| Column | Holds |
|---|---|
_source_id | The data source that wrote the row |
_connection_id | The connection it was read through (empty for a data source pinned to no connection) |
_run_id | The run that wrote it: a scheduled run, an on-demand fetch, a mirror run or a delivery |
_loaded_at | When the platform received the row |
They tell the rows of data sources sharing one table apart, and let you filter, audit or delete one connection's rows. A data source's Preview rows uses them to show only the rows it wrote, even in a table it shares. A source cannot set them: whatever a row carries under those names is replaced. A table created before row tags gains the four columns the next time its data source writes; the rows it already has leave them empty, and its Activity never lists them as new columns. The table preview hides them until you show them with its column picker.
Every table a data source writes also carries a markings column: the markings a reader must hold to see the row. A data source leaves it empty, so its rows stay visible to everyone who can read the table; a projection fills it from the rows it reads. It is what lets an ontology read a data source's table at all — a table without it, or with a markings column that is not an array, is reported as not provisioned. Like the row tags, a table created before it gains it the next time its data source writes, its Activity never lists it as a new column, and the table preview hides it until you show it.
In a table with a key (an upsert), a run writes only the rows whose values changed. A row a run sends again with the same values is left as it is, so its tags still name the run that last changed it — _loaded_at says when the row's current values arrived, not when the source last sent them — and a run that changes nothing writes nothing. What a run reports as written is what it changed: a database table's line then says No changes to write, and its activity keeps one line for every run in a row that changed nothing — for example "No changes to crm_orders since 14:03", at the newest such run — instead of a "Wrote" line for each.
Choosing a database source's tables
Some database sources let you choose their tables. You choose them in the Add data source sheet, before Add: as soon as you pick a connection, the sheet reads that database and lists its schemas, tables and columns as one tree, and Add creates the data source with the tables you chose, where each lands, the columns it keeps and its label, in one step. Nothing is created until you press Add; Cancel leaves nothing behind. Later, open the data source and its Tables tab to change them — it starts from the tables it mirrors now and shows how each one's mirror is doing:
- Open a schema to list its tables, and tick the tables to mirror; ticking a schema ticks every table it lists. The database's own schemas (such as
information_schema,sysorpg_catalog) are not listed. A table's key is in its row, beside its name: the key the database reports, or, when it reports none, a key column you choose there from the table's columns that the database says always have a value — until you do, the table counts as to decide. Saving a key the table does not have, or one that can be empty, is refused beside that table; a sync that still cannot use it as a key fails and says why. A table whose columns look like personal data says so as soon as its schema lists it, so a table you tick without opening its columns is still classified for what it holds (below). - Into · new or existing, in the same row, is where each chosen table lands: the new table the naming rule (
{schema}_{table}by default) names, one you type in the same field, or an existing table — another database source's copy you can write to (below). A name already used here, or used twice, is flagged in the row and must be changed before the tables start. Unticking the table takes it out again. - A ticked table opens on its columns: it reads every column until you leave some out, and only the columns kept are ever read. A column whose name looks like personal data (an email address, a phone number, a name, an address or a date of birth — a date of birth counts in a data source's columns, not in an uploaded file's) is flagged Personal data — judged from its name and type, never from a row — and a column you leave out says Left out. A table that keeps such a column is classified Confidential, and one sentence per table under the tree says so — for example "2 columns in customers look like personal data. email will be classified Confidential; phone is left out." A key column always stays.
- How rows land, folded under a table's columns, holds how its rows land — update by key (step 1), the default, or append only the new rows, by a timestamp column, the only way left when no column can be the table's key — optionally how many days of rows to keep, and the table's Classification, which you can raise but never set below the one its personal data calls for or your organization's minimum. It opens on its own when the table needs one of these from you.
- New columns, one line for every chosen table, says what happens when a table gains columns (see When the source adds columns).
- Select Add — it says how many tables, for example Add 2 tables (in the Tables tab, Save). The tables are mirrored from the next sync. If a table is refused after the data source was created, the sheet says why beside its name; change it from the Tables tab.
With many tables, Edit mappings opens Table mappings over the same database: one naming rule — {table}, {schema}_{table} or {source}_{table} — names every table it lists, schema by schema and page by page, and Needs a decision shows only the tables whose name is taken, invalid or used twice, or that look like personal data. Select rows, or a whole schema, to create them or leave them out in one go; Reuse… points the selected tables at an existing table every one of them fits.
Landing in an existing table. A database table can write into another database source's copy instead of a table of its own — for example the customers table of two order databases into one crm_customers. The dropdown lists the copies of your environment you can write to once the table's columns were read, each saying whether it fits: it is another database source's copy (a table of yours, a fetched or streamed table, or a table this data source already writes never is), it has the same key and the connection is part of it (an append lands only in a table without a key), its rows are timed by the same column, each column the table keeps has the same type there, and it is labelled at least as high as this table's data. A column it lacks is added, empty for the rows it already has; a column the table has that yours does not keep stays empty in your rows. Your rows are tagged with your data source and connection (see Row tags), so they never overwrite another's. The table keeps its own name and history. A table that does not fit stays in the list, marked Doesn't fit, cannot be picked, and says why. Removing either data source keeps the table and every row in it.
Once a table has written rows into its destination, where it lands (a new table or the existing one), how its rows land, its timestamp column, its history and its classification stay as they are (a table that appends without a timestamp column may still be given one): saving a change to any of them is refused and names the table. Only what happens when it gains columns can still change. To land a table differently, add another data source with a new destination table — the existing table and its history are kept.
The database is read through the connection you picked, with that connection's own login, so choosing its tables needs the same access as adding the data source: write in the workspace, and the connection shared with you. In the Add sheet the database is connected as soon as you pick the connection; while that takes a few seconds the sheet says Connecting to the database, and a database that cannot be reached says why. That connection to the database is yours while you choose: picking the connection off, cancelling or closing the sheet ends it, and it ends on its own after an hour. After Add, the data source connects with its own; the Tables tab reads through it, and says Connecting to the database while it does.
How rows land
How rows land decides the history the table keeps:
| How rows land | History |
|---|---|
| Snapshot — each sync replaces the table | Latest sync only |
| Append — each sync adds its new rows | Every version |
| Upsert by key — a row whose key exists is updated | Latest only, one row per key |
For an API source (a feed, a device, a service), how rows land is set by the source's definition — it shapes the table when it is created — and the source's details show it; a data source cannot choose another. For a database table you choose append or upsert when you choose its tables. Snapshot is not offered for a database table: replacing it every sync would delete the rows it has kept, and a mirror never deletes rows.
A database table upserts by key unless you choose otherwise. Append copies only new rows: each sync reads the rows whose timestamp column — a date or date-time column whose value only grows, such as when a row was created or last changed — is newer than the newest one the last sync copied, and adds just those; a row with no value there is not copied. So append is offered only once the table's columns show such a column, and it asks you to choose one before it is saved. Appending the whole table again on every sync is not offered: it would add every row each time.
An upsert may name a timestamp column too, as the row's own time. Without one, a row whose values did not change is not written again, and the table keeps no history: its rows keep the time they were first read, so removing rows older than some days would remove rows the database still has. Keep history for is turned off for such a table, and saving one with a history is refused.
A table with no column that can be its key and no date or timestamp column cannot be mirrored: every sync would copy all of its rows again, and the sheet says so beside it.
When the source adds columns
An API source lands the columns its definition names. A field the provider adds is not landed until a new release of the definition declares it, and nothing needs to be decided.
A database table can gain columns at any time. For each mirrored table you choose what happens then:
| Setting | What happens |
|---|---|
| Allow all | The new columns land, and the destination table gains them. Existing columns are never dropped or renamed. |
| Ask me first (the default) | The columns the table already has keep landing. The new ones wait: the data source shows schema changed and appears in Needs attention until someone who may change it selects Accept and add to the table in its overview. They land, and the table gains them, at the next sync. |
| Block | The new columns are left out. Every sync still lands the columns the table has, and says which it left out. |
A column the remote table removes or retypes is not a new column: the mirror stops and says the destination changed, whatever the setting.
Schedule
A source that runs on a schedule — an API source that polls, or a database source that mirrors its tables — runs on its definition's interval unless you set one for the data source. Set it when you add the data source, or later in its Schedule & retention tab; Use the definition's schedule returns it.
The interval can be as short as the definition allows — the field says what that is, for example every 1 min — and as long as one week. A data source never runs more often than its definition allows, even when its definition later raises that limit. A stream, a push endpoint, a manual upload or a source you fetch on demand does not run on a schedule, so there is nothing to set.
Status
Every data source shows one status:
| Status | What it means |
|---|---|
| Healthy | Its last sync succeeded. A warning, such as a stream that went quiet or a run that waited for a busy table, shows beside it. A data source that has run before and was resumed or changed says Waiting for its next run until its schedule runs it again; the list shows that run, and its Last sync, as soon as it lands — even a run that found nothing new — without a reload. One whose first rows someone else brings says what it waits for until they arrive: a push data source Waiting for its first delivery, a manual one Waiting for its first upload, and one you fetch on demand Waiting for its first fetch. |
| Failed | Its last sync failed, or its connection cannot serve it. The status says why and who fixes it — your organization administrator for a connection, your Scrydon operator for a deployment setting. |
| Paused | Scheduled syncs are skipped until someone resumes it. |
| Running | A sync is running, or it is waiting for the very first run the platform starts itself. |
A database source that mirrors its tables is Failed while its database cannot be read live, and otherwise shows its last mirror run: a failed run stays Failed until a run succeeds, however often the database answers in between.
When a database source's mirror fails as many runs in a row as your organization's auto-pause threshold (10 unless an administrator changed it), its copy is paused: it says Copy paused after 10 failed runs in a row, with the last error, and every organization administrator gets an in-app notification. Only the copy stops — live queries against the database keep working. Fix the connection, then choose Resume copying: the copy runs again, its failure count starts from zero, and live queries are not interrupted. A run that could not start because the table was busy with another write does not count toward the threshold; it is simply tried again within minutes. Pausing the source yourself stops both the copy and live queries: the row menu offers Pause copying for a database copied on a schedule, Pause live access for one read where it is, and Pause syncing for any other source, and asks you to confirm, saying what stops and that its tables and rows are kept.
Runs as the person who added it
A data source's scheduled runs, and the runs Sync now and Sync this table start, act as the person who added it. Before each run — for a stream, every 30 seconds while it is connected — the platform checks that this person is still a member of the organization, can still change data sources in the workspace (an organization administrator always can), and can still use the connection the data source is pinned to. The run's rows, and its entries in the Audit Log and the table's history, are recorded under their name; the data source's table itself belongs to the platform, whoever added the source.
When that person no longer has this access — the connection is no longer shared with them, they lost access to the workspace, or they left the organization — the data source stops instead of running without it. Nothing that run read is written. It reads Stopped: the person it runs as lost access and says who no longer has access to which connection (a connection not shared with you is not named). An organization administrator fixes it by sharing the connection with that person again or giving back their access; then choose Resume syncing. A data source keeps running as the person who added it: to have it run as someone else, that person removes it and adds it again. A database source stopped this way stops its live queries too. If the platform cannot check the person's access at that moment, nothing stops: the run is tried again.
A data source added before people were recorded runs as the platform, as it always did.
Needs attention lists the failed data sources, those paused automatically after failing again and again, those stopped because the person they run as lost access, and database sources whose tables gained columns that wait for someone to accept them. The Connection column names the connection each reads; a connection not shared with you is not named.
Alerts and the daily digest
A data source alerts on a change of its state, never on every failed run:
| Alert | When |
|---|---|
| Started failing | Its first failed sync after running well. A failure that is not the source's own — the provider asking for fewer requests, the data policy that could not be checked, full storage, rows Scrydon could not write for a reason on its side — is not one, and goes into the daily digest instead. Neither is a run that waited for a table busy with another write, or for a database table another run of the same data source was copying: those rows are written moments later and nothing is recorded. A data source that was already failing when alerts were turned on for it — or before your deployment had alerts at all — does not alert until it has run well again and then fails. |
| Running late | A source that runs on a schedule has not synced for its interval, plus twice as long as its last run took, plus five minutes. |
| Paused | It failed as many runs in a row as your organization's auto-pause threshold and was paused — for a database source the copy is paused and live queries keep working, and that alert is the Copy paused one described under Status — or it stopped because the person it runs as lost access to its connection or its workspace, or left the organization. |
| Recovered | Its next good sync after an alert about it. |
Every alert names the data source, the reason, and who fixes it — your organization administrator for a connection, your Scrydon operator for a deployment setting or a late schedule, otherwise whoever maintains the source — and opens Data sources. A failure clears itself after the next good run.
A data source that keeps failing and recovering does not keep alerting: after a Started failing or Running late alert, the next one waits six hours, and what happens in between — including the recovery that answers it — goes into the digest instead. A late source that then fails, and a failing source that is then paused, alert at once: they say something new.
Once a day (07:00 UTC unless your operator changed it) the daily digest lists, per data source, what changed in the last day that did not need an alert: failed runs, warnings such as rows the data policy masked or set aside, schema changes, and rows written. Each line opens with the data source's health as the list shows it, followed by what it is waiting for or what went wrong when the list names that — a push source nothing was sent to yet reads Healthy · Waiting for its first delivery. A data source with nothing but routine writes is left out, and when nothing changed for any of them there is no digest. You get at most one digest a day for each organization.
Alerts and the digest go to your organization's administrators and to the person who added the data source, while they are still a member, in the notifications inbox and by email unless they turned email off (see Notifications). An administrator's digest lists every data source; yours lists the ones you added. A data source added with alerts off sends neither.
Change or remove a data source
Open a data source to see its tabs:
- Connection — point it at another connection shared with you. Its table and history stay; new rows come from the connection you choose.
- Overview — for a database source, each mirrored table's setting for new columns (which you can change when you chose its tables), the columns waiting to be accepted, and the ones its setting left out.
- Schedule & retention — how often it runs, and how long rows are kept. Anyone who may change the data source changes its schedule. Only an organization administrator changes retention, because a shorter window deletes older rows; everyone else sees it.
- Push endpoint — for a data source that receives pushed deliveries: the path the sending system posts to, the header naming each delivery, and how each delivery proves itself (a signature or credential header, keyed with the connection's secret). The secret itself is never shown, and the host is your deployment's public push host — ask your Scrydon operator for it. When the deployment does not run push sources, the tab says so. It needs the same access as changing the data source.
- Remove — it stops syncing. Its tables and its sync history are kept, and its connection is not changed. The tab and the confirmation say how many tables it keeps.
The data source an extension installs with itself is removed by an organization administrator, with Remove data source in the list. Removing any data source keeps its tables and every row in them: grouped by data source, the Tables page lists them under Removed source · its name — the name the list showed it by; a data source removed before names were kept shows its key there instead — which says they are no longer synced, and you delete a table there when you no longer need it. A data source you add later never takes them over, even under the same name: it is a new data source with its own tables, and its first run and activity start when you add it. To write into a kept table again, choose it under Into · existing table when you add the data source.