Refresh copied records by reading only what changed
A pipeline that copies ERP records into the application can read the whole source on every pass, or read only what changed since the last one. Reading only the changes keeps a large copy current at a fraction of the calls, which matters when every call is a round trip to someone else’s production system.
The two kinds of pass answer different questions. A full pass reads every record and is the only pass that may conclude a record was deleted, since a record absent from a complete read is gone. An incremental pass reads what changed since a watermark taken from the ERP’s own change stamps, never from the application’s clock, and loads it. A full pass therefore stays scheduled beside it: the declaration requires one.
There are three ways to ask the ERP what changed: Odoo’s last-updated stamp, an OData property holding each record’s modification time, and OData change tracking, which also reports what the ERP deleted. The watermark moves only after the records that reached it are loaded, so a crash re-reads and never skips. Changing what the pipeline extracts or how it maps it resets the watermark, so the next incremental pass reads the whole source once and every record gets the new mapping.
Large populations are read page by page. Each page is loaded and checkpointed before the next is read, a pass may span several runs, and what a pass has already read is stored one row per record rather than kept in memory. A run therefore holds one page, whatever the size of the source.
Example — Keep a large account list current every ten minutes
An application keeps a copy of the accounts in the company’s Microsoft Dataverse. Every ten minutes it follows Dataverse’s change tracking and receives the accounts changed and those deleted since the last run. A nightly full pass reads every account and catches anything the tracking could not report.
For engineers
incremental sits beside scheduleCron, which runs the incremental passes, and declares fullPassCron, which runs the full ones. The following are design examples, checked against the declaration schema; ExternalDataIncrementalExtractionMode comes from @wildo-ai/saas-models/external-data.
// Odoo: read what changed by `write_date`, re-reading the last five minutes at the start of each pass.
scheduleCron: '*/10 * * * *',
incremental: {
mode: ExternalDataIncrementalExtractionMode.ODOO_WRITE_DATE,
fullPassCron: '0 3 * * *',
lookbackSeconds: 300,
},
// OData v4: a modification-time property this entity carries, named by its remote name.
incremental: {
mode: ExternalDataIncrementalExtractionMode.ODATA_MODIFIED_AT,
fullPassCron: '0 3 * * *',
modifiedAtRemoteField: 'modifiedon',
},
// OData v4 change tracking: the service's delta link, which also reports deletions.
incremental: {
mode: ExternalDataIncrementalExtractionMode.ODATA_DELTA_LINK,
fullPassCron: '0 3 * * *',
},
scheduleCron is required with any of them, since nothing else would run an incremental pass. A full pass that needs several runs continues on scheduleCron ticks, interleaved with incremental passes; neither track waits for the other, and a record an incremental pass loads while a long full pass is in flight is never mistaken for a deletion when that full pass completes.
What each mode reads
| Mode | Reads | Watermark | Deletions |
|---|---|---|---|
ODOO_WRITE_DATE | records by (second, id) from the watermark, less lookbackSeconds (default 300) | the last row loaded | full pass only |
ODATA_MODIFIED_AT | $filter=<field> ge <watermark − lookback>, in key order | the largest timestamp read, moved only when the pass completes | full pass only |
ODATA_DELTA_LINK | the whole set once with Prefer: odata.track-changes, then the service’s @odata.deltaLink | the delta link | reported by the service, and handed to the orphan policy |
Odoo stores write_date with microseconds and serves it to the second, so the Odoo mode finishes each second by id before moving on: every record is read once, however many share a second. The lookback re-reads recent changes because a long transaction can commit a record whose stamp is older than a watermark another pass already advanced past; a re-read costs nothing, since the load compares content and writes nothing for an unchanged record.
What startup refuses
The mode must be spoken over the binding’s dialect, and the vendor facet must declare it:
| Vendor | Modes |
|---|---|
| Odoo | ODOO_WRITE_DATE |
| Microsoft Dataverse, generic OData v4 | ODATA_MODIFIED_AT, ODATA_DELTA_LINK |
| Business Central, SAP S/4HANA (OData v4 roots) | ODATA_MODIFIED_AT |
| Salesforce, NetSuite, any OData v2 root | none: full passes only |
ODATA_DELTA_LINK is also refused with a sourceFilter, because change tracking refuses $filter, and with a composite key, because a deleted entity is reported by one id.
Expired positions and paging
A pass spanning runs keeps the service’s next link as its cursor. If the service has forgotten it (410 or 404), or it points at a host the instance no longer uses, the same pass starts again from its first page, and nothing it had read is orphaned; this happens once per pass, and a second restart is refused. An expired delta link, 410, 404 or a code the vendor facet declares (Dataverse 0x80044352), resets the track, and the same run falls back to a full pass. Both are named in the run’s history.
maxRowsPerRun (default 10,000) bounds a run’s work, not its memory, and a run ends on a page boundary. An OData v4 source is asked for pages of at most 1,000 rows with Prefer: odata.maxpagesize, never $top, because $top limits the whole collection and would make a pass look complete after its first page. maxRunDurationMs (default fifteen minutes) bounds how long a run waits on a busy source before it stops as a partial pass, which the next run continues.