Endpoints from a connection
Define a source once and let each connection supply its own URL, database and credential
A data source can take where it connects from the connection an administrator creates, instead of from its manifest. You define "Home Assistant states" once; a personal and a company Home Assistant are two connections, each with its own URL and its own token. Nothing about either system is baked into the extension you publish.
Two things make this work, and they live in different places:
| What | Declared in | Holds |
|---|---|---|
| The connection's fields | The extension — defineExtension({ configFields, auth }) | A non-secret endpoint URL (or host, port and database), and the credential |
| The source's references | The data source — request.endpointRef, request.authRef, … | The names of those fields. Never a value. |
An administrator fills in the fields once, on the connection. The source reads them by name on every run.
Declare the connection on the extension
configFields are the non-secret values a connection holds. A field the platform will dial is declared with type: 'url' and purpose: 'endpoint'. Credentials go in auth.credentials, exactly as for any other extension.
An extension that ships only data sources declares both directly on defineExtension — it needs no toolkit:
import { defineExtension } from '@scrydon/sdk-authoring/extensions'
import { homeAssistantStates } from './data-source-states'
export default defineExtension({
id: 'home-assistant',
name: 'Home Assistant',
version: '1.0.0',
logo: './assets/icon.svg',
configFields: [
{
key: 'baseUrl',
label: 'Home Assistant URL',
type: 'url',
purpose: 'endpoint',
required: true,
placeholder: 'https://xxxx.ui.nabu.casa',
},
],
auth: {
credentials: { token: { type: 'apiKey', label: 'Long-lived access token' } },
default: 'token',
},
provides: { dataSources: [homeAssistantStates] },
})A field can also carry validations — { pattern, message, whenHostSuffix? } — that the connection form and the platform check before the value is used (for a url field, the pattern reads the URL's path). Patterns run on the platform for every value checked, so extension build and install refuse one that can backtrack catastrophically: nested repetition, a repeated group with alternatives such as (a|aa)+, a backreference, more than three variable-length repetitions, a repeat count over 25, or a pattern over 512 characters. A value longer than 256 characters is refused before any pattern runs.
Read the endpoint in the source
Where a source would write url, it writes endpointRef — the key of the endpoint field — and path, the part that follows the connection's URL. authRef names the credential:
import { defineDataSource } from '@scrydon/sdk-authoring/extensions'
export const homeAssistantStates = defineDataSource({
kind: 'table',
id: 'home-assistant-states',
vendor: 'home-assistant', // the extension id — its connections carry the endpoint and token
displayName: 'Home Assistant states',
scope: 'org',
table: {
name: 'ha_states',
primaryKey: ['entity_id'],
timestampColumn: 'last_changed',
columns: [
{ name: 'entity_id', dataType: 'string', isPrimaryKey: true },
{ name: 'state', dataType: 'string', nullable: true },
{ name: 'last_changed', dataType: 'timestamp' },
],
},
ingest: {
mode: 'poll',
intervalSec: 60,
request: {
endpointRef: 'baseUrl',
path: '/api/states',
authRef: 'token',
auth: { placement: 'bearer' },
},
response: { itemsPath: '$' },
mapping: {
entity_id: { path: 'entity_id' },
state: { path: 'state' },
last_changed: { path: 'last_changed' },
},
},
})The request goes to the connection's URL with path appended. A path already on the URL is kept, so a Home Assistant behind a reverse proxy works unchanged:
| Connection URL | path | Request |
|---|---|---|
https://xxxx.ui.nabu.casa | /api/states | https://xxxx.ui.nabu.casa/api/states |
https://proxy.example.com/ha | /api/states | https://proxy.example.com/ha/api/states |
Everything else in request — method, headers, query, auth, body — works exactly as it does with url. Static query parameters belong in request.query, never in path. On-demand sources use the same request block.
A request names its endpoint exactly once: either url, or endpointRef and path. Setting both, or neither, is a build error that says which key to remove or add.
A public endpoint, with no credential
When the service needs no authentication, the connection carries only the URL. Declare the extension's credential as none and leave authRef out of the source:
// defineExtension({ ... })
configFields: [{ key: 'baseUrl', label: 'Feed URL', type: 'url', purpose: 'endpoint', required: true }],
auth: { credentials: { none: { type: 'none' } }, default: 'none' },
// defineDataSource({ ... }).ingest
request: { endpointRef: 'baseUrl', path: '/v1/readings' },The administrator still creates a connection — it is where the URL lives — with no secret to enter. A source whose url is fixed in the manifest and needs no credential keeps working without any connection, as before.
The path rule
path is appended to a URL an administrator chose and the organization approved, so nothing in it may lead the request elsewhere. It must:
- start with
/, and not with//; - contain only URL path characters — no spaces, backslashes or control characters;
- contain no
.or..segment — also not with a;parameter (/..;/admin), which Tomcat, Jetty and Spring resolve like..; - contain no escaped
.,/or\(%2e,%2f,%5c), no escaped%(%25, which a proxy that decodes twice reads as another escape) and no NUL (%00); - carry no
?query string and no#fragment.
Each is a build error naming the rule. The same rule is enforced again when the request is built.
Streams
A stream endpoint takes the same two forms. With endpointRef and path, the socket opens on the connection's host as wss://:
ingest: {
mode: 'stream',
transport: 'websocket',
endpoint: { endpointRef: 'baseUrl', path: '/api/websocket', authRef: 'token' },
message: { itemsPath: '$' },
mapping: { /* ... */ },
}The endpoint field itself holds the https:// address; https://xxxx.ui.nabu.casa becomes wss://xxxx.ui.nabu.casa/api/websocket.
Query sources
A query source can read its host, port and database from the connection. Name the connection's type: 'text' fields in endpoint.endpointRefs:
// defineExtension({ ... })
configFields: [
{ key: 'dbHost', label: 'Database host', type: 'text', required: true },
{ key: 'dbPort', label: 'Port', type: 'text', required: true, placeholder: '5432' },
{ key: 'dbName', label: 'Database', type: 'text', required: true },
],
auth: {
credentials: { warehouse: { type: 'basicAuth', label: 'Database login' } },
default: 'warehouse',
},
// defineDataSource({ ... })
credentialRef: 'warehouse',
ingest: {
mode: 'query',
engine: 'postgres',
endpoint: {
endpointRefs: { host: 'dbHost', port: 'dbPort', database: 'dbName' },
trustedCaRef: 'warehouse-ca',
},
tables: [{ schema: 'public', name: 'readings' }],
},- Each value is held to the same rule as a value written in the manifest: a hostname (no scheme or port), a port from 1 to 65535, a database name of letters, digits,
_and-. - The trust anchor is named once — literally as
endpoint.trustedCaRef, or from a field asendpointRefs.trustedCaRef. TLS always verifies the server's identity; there is no opt-out. - The credential is a
basicAuth(username and password) credential. - For MySQL, a table's
schemais its database, so each declared table must name the database the connection supplies. That is checked when the connection's values are read.
The connection's values are checked at every catalog registration, before anything connects. Nothing connects until they pass; the catalog stays pending and says why:
| The connection's values… | The catalog shows | Fix |
|---|---|---|
| are missing (a value of only spaces counts as missing) | pending · credential unavailable | Fill in the field on the connection. |
| name an invalid host, a port that is not digits from 1 to 65535, or an invalid database — or a MySQL table names another database | pending · credential unavailable | Correct the value on the connection. |
| contain the connection's password | pending · credential unavailable | Remove the password from the value. (The login name may equal the database name — that is common and allowed.) |
| name a host that is not on the organization egress allowlist | pending · egress blocked | An administrator approves the host. |
| use a port no database network of your deployment opens | pending · egress blocked | Ask your Scrydon operator to add the database's network, with that port, to the deployment's private networks. |
A catalog registered against the connection's previous values stops querying them as soon as a check fails.
Changing the host, port or database re-registers the catalog against the new server at the next reconciliation; from then on, queries and mirror runs go only to the new server.
Every time the connection opens — and while it stays open — each address the host resolves to is checked again. A database is reachable only inside a private network your Scrydon operator opened for your organization, on a port that network opens; a network in a private range also needs your organization's Private networks switch on (Settings → Governance → Egress). The organization's own switches never open a database address by themselves. The connection's trust anchor must be one that network lists. These addresses are always refused, whatever is configured:
- loopback (
127.0.0.0/8,::1) and the deployment's own cluster ranges; - link-local and cloud-metadata addresses (
169.254.0.0/16,fe80::/10,fd00:ec2::/64) and Azure's WireServer (168.63.129.16); - shared address space (CGNAT,
100.64.0.0/10), benchmarking (198.18.0.0/15), the documentation ranges (192.0.2.0/24,198.51.100.0/24,203.0.113.0/24,2001:db8::/32),0.0.0.0/8, multicast and reserved space (224.0.0.0/4,240.0.0.0/4).
A query host is a DNS name. An IP address is refused as an invalid host — use the name the database's certificate carries.
What the administrator does
An organization administrator adds the endpoint's hostname to the organization egress allowlist (Settings → Governance → Egress). Only an explicit entry approves it: platform default domains and hosts an extension declares for its tools do not. The entry must name at least two labels — example.com approves ha.example.com, a bare com approves nothing. An IP address is approved only by an entry that is exactly that address, and only when it lies inside one of the deployment's private networks.
In Settings → Platform → Extensions → Home Assistant → Connections, add a connection. One form holds everything: a name ("Home Assistant · personal"), the environment, the URL, and the token.
The source reads that connection on its next run. A second system is a second connection, with its own URL and token.
The URL is checked before anything connects to it, and a refusal says what to fix:
| The URL… | Result |
|---|---|
| is missing, or only spaces | The source reports that it needs a connection. Nothing is fetched. |
is not https://, or contains a username, password, ? query or # fragment | Refused as an invalid endpoint. Nothing is fetched. |
contains the connection's token or password — also split by a tab or line break, or percent-encoded. A token of 8 characters or more counts wherever it appears; a shorter one only as a whole part of the address, such as a host label or a path segment, so a one-letter test token does not refuse https://example.org | Refused. Enter the token in the credential field, not in the address. |
| uses a port other than 443, or reaches an address, that no private network of your deployment opens | Refused. Ask your Scrydon operator to add the network to the deployment's private networks. |
| reaches a private network your deployment opens while your organization's Private networks switch is off | Refused. Turn on Private networks in Settings → Governance → Egress. |
| reaches a reserved address, or one of Scrydon's own | Refused, always — the addresses listed above for query sources. |
| names a host that is not on the egress allowlist | Refused until an administrator approves the host. |
| names an IP address the allowlist does not list exactly | Refused until an administrator adds that exact address. |
Save only refuses what is wrong with the value itself; whether a network or the allowlist admits it can change later, so the connection is saved and the source's readiness and sync status say what is missing. The port and every address are checked again each time the source connects — a name that later resolves somewhere else is refused then.
The same message tells the administrator what to fix, whether it appears under the field in the connection form or in the source's sync status. Readiness judges the connection's fields only for an organization owner or admin: a workspace member sees whether the source's connection, account and credential are in place, and learns about the connection's address from the source's sync status once it runs.
A connection is chosen for the environment the source runs in; a Production source never falls back to a Development connection.
Log lines and sync status never show the connection's URL or host. A source that reads its endpoint from a connection is named by the field and the path — "the connection's Home Assistant URL at /api/states" — and the host, or any address it resolves to, is replaced by the field's label wherever an error repeats it. A source with a fixed url is shown with its URL, any credential in it masked. Either way, a source that holds a credential shows none of what the server or the network said: a failed request reads only its status line (HTTP 401 from …) or "request failed for …", because a server can echo the credential in a form no mask recognises.
Local addresses need your operator. A stock Home Assistant at http://homeassistant.local:8123 is plain HTTP, which is never used. Serve it over HTTPS, and then either give it a public hostname (Home Assistant Cloud, or a reverse proxy on port 443), or ask your Scrydon operator to add its network and port to the deployment's private networks for your organization — with your certificate authority if it signs the certificate — and turn on Private networks.
What the build checks
extension build, extension validate and the platform's install gate refuse an extension whose data sources name something it does not declare. Each message names the source and the missing piece:
| Refused | Fix |
|---|---|
endpointRef (or an endpointRefs entry) names no config field | Add the field to configFields — type: 'url', purpose: 'endpoint' for a request or stream, type: 'text' for a query endpoint part |
endpointRef names a field that is not an endpoint field | Declare it with type: 'url' and purpose: 'endpoint' |
authRef or credentialRef names no credential | Add it to auth.credentials |
A query source's credential is not basicAuth | Declare it with type: 'basicAuth' |
The source's vendor is not the extension's id | Set vendor to the extension's id — the extension whose connections carry the endpoint and credential |
The code manifest's extension.id differs from extension.json | An extension has one id; set both to the same value |
Why a named field, not a URL template. A placeholder such as {{connection.baseUrl}} could appear anywhere in a URL, including the host, so "the host comes only from an administrator-approved endpoint" would be a hope. endpointRef plus path makes it the shape of the manifest: the host can come from nowhere else.