Incremental Polls

By default every run of a declarative poll source sends the same request, and upsert on the item's id absorbs the overlap. Many vendor APIs can do better: they return a marker with each response (X's meta.newest_id) and accept it on the next call (since_id), so each run fetches only what is new.

The cursor block on a poll's ingest declares that marker:

• param — the query parameter the stored value is sent as.
• from — where the next value is read in the response.
• fromLink — instead of from, for a vendor that answers with a link to its next request: where that link is in the response. The next value is the link's param query parameter.
• morePath — optional: where the vendor says more results are waiting.

The platform stores the cursor per installed source and moves it only after the run's rows were written.

Declare a cursor

ingest: {
  mode: 'poll',
  intervalSec: 300,
  request: {
    url: 'https://api.x.com/2/tweets/search/recent',
    query: { query: 'from:scrydon', max_results: '100' },
    authRef: 'token',
  },
  response: { itemsPath: '$.data', format: 'json', itemsOptional: true },
  mapping: {
    id: { path: 'id' },
    text: { path: 'text' },
    createdAt: { path: 'created_at' },
  },
  writeMode: 'upsert',
  cursor: { param: 'since_id', from: '$.meta.newest_id', morePath: '$.meta.next_token' },
}

A cursor declares exactly one of from and fromLink. cursor is accepted on a poll source only. Every other ingest mode rejects the field.

A cursor in a link: fromLink

Some vendors hand back the next request as a whole link rather than a value. Microsoft Graph's delta queries end their answer with @odata.deltaLink, and the token to send next time is that link's $deltatoken parameter:

{
  "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users(displayName,mail)",
  "@odata.deltaLink": "https://graph.microsoft.com/v1.0/users/delta?$deltatoken=oEcOySpF_hWYmTIUZBOIfPzcwisr…",
  "value": [{ "id": "87d349ed-…", "displayName": "Adele Vance", "mail": "adelev@contoso.com" }]
}

fromLink points at the link; param names the query parameter read from it, and the same parameter is sent on the next run:

request: {
  url: 'https://graph.microsoft.com/v1.0/users/delta',
  query: { $select: 'displayName,mail' },
  authRef: 'token',
},
response: { itemsPath: '$.value', format: 'json', itemsOptional: true },
writeMode: 'upsert',
cursor: {
  param: '$deltatoken',
  fromLink: "$['@odata.deltaLink']",
  morePath: "$['@odata.nextLink']",
},

• Graph's keys hold a ., so the path quotes them inside brackets: $['@odata.deltaLink']. A dotted path ($.@odata.deltaLink) would look for deltaLink inside @odata and find nothing.
• The link must be an absolute URL that carries param. The value is stored decoded and sent encoded, so a token with %3d in the link is sent back as the same token.
• Only param is taken from the link. The rest of the next request is the one you declared — its url, request.query and credential — so declare the request the link points back to.
• A delta round that does not fit in one response has no @odata.deltaLink on its first page, only @odata.nextLink. When that next-page link carries param too, the cursor moves to it, so each run reads the round's next page. When it carries another parameter, the cursor does not move and the next run starts the round again. See More pages.
• A vendor can expire a token. Graph answers a request whose $deltatoken has expired with HTTP 410 Gone. That run fails and the cursor stays, so every later run fails the same way until an organization admin resets the cursor; the next run then starts a new round.

How a run uses it

1. The first run has no stored cursor and sends the request exactly as declared — no param.
2. A later run adds param=<stored value> to the query string, as long as the stored value was taken under the same connection and request (see What resets a cursor).
3. After the response is decoded, the value at from becomes the next cursor if it is a non-empty string or a finite number. A number is sent as text. An id above 2^53 loses precision as a JSON number, so point from at the vendor's string form when it has one (X's ids are strings).
4. The next cursor is stored only after the run's rows were written, in the same update that records the run as successful.

Anything else leaves the cursor where it was:

• from resolves to nothing, null, an empty string, an object or a list. X omits meta.newest_id when nothing is new, so the next run asks from the same point.
• fromLink resolves to anything but an absolute URL, or to one without param or with it empty.
• The value is longer than 2,048 bytes (UTF-8). The run logs cursor_too_long and the cursor does not move. For fromLink the bound is on the parameter's decoded value, not the whole link.
• The fetch failed, the response could not be decoded, or the rows could not be written. A cursor is never stored past rows a failed run lost.

A run that succeeds still moves the cursor past rows it did not write: rows that fail the source's validation, and rows the organization's data policy blocks. Fetching them again would end the same way. If an upgrade fixes the mapping or validation that rejected them, the cursor is kept (see What resets a cursor): reset it so the next run starts as a first run.

Write mode

A cursor needs writeMode: 'upsert' (the default) or 'changed-only'. The other two are rejected at build time:

The parameter must be its own

param may not collide with anything else that sets a query parameter: a key in request.query, a parameter already written into request.url, or the credential's query name when auth.placement is query. Either value would decide, or be decided by, the cursor the vendor sees. Rename one of them.

More pages: morePath

A run sends one request and reads one response. When morePath resolves to anything but nothing, null, an empty string or false, the vendor holds pages this run did not read. The run logs cursor_more_pages, writes the rows it read, and is not counted as complete: the source's sync state says more pages wait, and its health is not certified by that run. A run is complete only once it reads the last page.

What the cursor does depends on where it is read from:

• from. The cursor does not move. The value at from names the newest item, and the pages behind it are older; moving to it would skip them for good. The next run asks from the same point again.
• fromLink. The cursor moves to the param query parameter of the next-page link at morePath, so the next run reads the round's next page. A drive's delta (?token=) pages this way. When the next-page link carries a different parameter (Graph's users delta pages with $skiptoken but resumes with $deltatoken), the cursor does not move, and each run starts the round again.

Raise the page size where the vendor allows it (Graph's Prefer: odata.maxpagesize=… header, in request.headers), so a round or a burst fits in one response.

If the log shows cursor_more_pages on most runs, the vendor produces more than one response holds per interval: shorten the interval, or raise the vendor's page size within the resource limits. With a from cursor this is required, not just faster: until a response holds the whole burst, the cursor cannot move.

What resets a cursor

A cursor belongs to the request it was taken under. The platform stores it with a scope: a hash of the connection the run resolved, that connection's admitted endpoints, the declared request, the definition's intervalSec and minIntervalSec, and the cursor block itself. A run whose scope differs does not send the stored value; it starts as a first run and logs cursor_reset. If that run cannot advance the cursor, the stored one is cleared.

Reset a cursor by hand

An organization admin can clear a source's cursor in Analytics → Data sources: open the source, go to the Schedule & retention tab and choose Reset cursor, then confirm. The action is shown only for a source that declares cursor, and only to organization admins. A reset that clears a stored cursor is recorded in the Audit Log as data_source.cursor.reset, naming the admin; a source with no stored cursor has nothing to clear: Analytics says Nothing to reset, and nothing is recorded.

Use it when the table has lost rows the vendor still holds — after deleting rows by mistake, for example. The next run sends no param and fetches what the vendor returns for a first request; upsert absorbs the rows already in the table.

Validation summary

These are build-time errors, caught by extension build before an archive is ever uploaded: