Bring external records into your application
An application sometimes needs more than a view of another system’s data. People may want to attach files, add an internal note, connect records to their work or query them locally.
Wildo’s copy-in pipelines bring selected external values into an ordinary resource in the application’s database. The pipeline describes where the records come from, how fields map and how later runs reconcile changes.
This makes ownership clear. The remote system supplies its mapped fields; the application can keep its own fields beside them. A schedule controls how often the local copy is refreshed, and a pipeline can also read only what changed or re-read a record when its source announces a change.
Example — Keep customer accounts and local notes together
Wonder Todos synchronizes client accounts from the company’s Odoo into its external-customers resource. Odoo supplies the client’s name, shown with its finance reference, and its email; a team member can add an internal note locally. A later synchronization refreshes the mapped values without replacing the note.
For engineers
Map external values to local fields
This pipeline declaration is from Wonder Todos’ external-customers.resources-config.ts. The surrounding resource is locally stored and offers READ, LIST and UPDATE; source comments are omitted:
externalDataPipeline: {
binding: {
dialect: HttpApiTransportDialect.ODOO_JSONRPC,
system: 'odoo',
entityRef: 'res.partner',
keyFields: [{ localField: 'id', remoteField: 'id', codec: HttpApiKeyComponentCodec.INTEGER }],
subjectRights: { posture: SystemOfRecordSubjectRightsPosture.OWNED_COPY },
purposeRefs: ['task-planning-service'],
},
remoteKeyLocalField: 'odooPartnerRef',
partitionUnitLocalField: 'organizationUnitId',
partitionKeyLocalField: 'odooCompanyRef',
mapping: [
{
kind: ExternalDataMappingEntryKind.CONCATENATION,
localField: 'name',
parts: [
{ kind: ExternalDataConcatenationPartKind.REMOTE_FIELD, remoteField: 'name' },
{ kind: ExternalDataConcatenationPartKind.REMOTE_FIELD, remoteField: 'ref' },
],
separator: ' · ',
},
{ kind: ExternalDataMappingEntryKind.REMOTE_FIELD, remoteField: 'email', localField: 'email' },
],
sourceFilter: { is_company: true },
population: ExternalDataDestinationPopulation.CLOSED,
orphanPolicy: ExternalDataOrphanPolicy.IMPERSONALIZE,
scheduleCron: '0 * * * *',
},
The binding names the declared Odoo instance and its res.partner entity. remoteKeyLocalField identifies the local business-key field used to reconcile a source row. The two partition fields are written by the engine, never mapped: one holds the organization unit the record’s Odoo company is assigned to, the other Odoo’s own company id, so a row still says which company it came from after that company’s assignment changes. Rows of a company left unassigned are read on every pass and counted as present without being loaded, so withdrawing an assignment never removes or scrubs a client; until the company is assigned again, nobody can read those rows.
The mapping names the values the pipeline owns. The CONCATENATION entry is a declared expression: the displayed name is the client’s name followed by its finance reference, and a missing reference is dropped with its separator. sourceFilter chooses client accounts rather than the contacts filed under them, and the hourly schedule expresses the application’s refresh cadence.
The subject-rights posture follows from the data. A client recorded as a company in Odoo can still be a sole trader, so the copied name and email describe people and NO_SUBJECT_DATA would be refused. The copy also keeps internalNote, the team’s own note about a client, which Odoo does not hold. A copy that keeps something its source does not is kept beyond the source, so it declares OWNED_COPY: the application erases the copy itself and keeps the next pass from loading the person back, while Odoo answers requests about the records it came from, which the erasure receipt states. A copy that only mirrored Odoo would follow it instead, and remove what Odoo removes.
Declare which local fields the source owns
The destination schema in external-customers.schemas.ts pairs that mapping with an explicit reconciliation key, engine-written company fields and read-only imported fields. Comments and unrelated fields are omitted:
odooPartnerRef: z.string().min(1).isBusinessKey().isUnique().isDBIndexed().isSummaryField()
.excludeFromCreate().excludeFromUpdate(),
organizationUnitId: z.string().optional().isDBIndexed().isForeignKey().excludeFromCreate().excludeFromUpdate(),
odooCompanyRef: z.string().optional().excludeFromCreate().excludeFromUpdate(),
name: z.string().optional().isSummaryField().excludeFromCreate().excludeFromUpdate()
.dataCategories(PersonalDataCategory.IDENTITY),
email: z.string().optional().excludeFromCreate().excludeFromUpdate()
.dataCategories(PersonalDataCategory.CONTACT_DETAILS),
internalNote: z.string().optional().dataCategories(PersonalDataCategory.USER_CONTENT),
The remote-key field must be a string marked both isBusinessKey() and isUnique(). Mapped fields must be excluded from caller CREATE and UPDATE. Optional source values need destination fields that admit absence: an Odoo client with no company is shared by every caller, so both company fields accept absence. The data categories on name and email are what a NO_SUBJECT_DATA posture would be checked against. internalNote is not mapped, so a refresh preserves the application’s own annotation.
Decide which records belong to the pipeline
population: CLOSED means every row comes from the source, so the resource does not offer caller creation. Local UPDATE remains useful for fields the pipeline does not own. Mapped destination fields are declared excluded from caller CREATE and UPDATE in the schema, keeping that ownership visible through the generated input and form.
orphanPolicy: REPORT records unmatched records without deleting them; a copy that follows its source and copies personal data may not declare it, because it would keep a record the source deleted. Disappearance reconciliation requires a completed pass with an accepted, complete source-membership history. An open population, where people also create rows, runs its declared policy too: only rows the pipeline loaded are ever compared with the source, so a row a person created is never removed.
This pipeline declares IMPERSONALIZE instead. A client absent from a complete pass has its name, email and note scrubbed in place, and the row stays because the todos done for the client point at it. The scrub writes no erasure suppression, so a record Odoo lists again loads as a new row. IMPERSONALIZE needs a destination that erases in place (RETAIN_AND_IMPERSONALIZE); the factory refuses it otherwise.
Extract, transform and reconcile through shared services
Extraction uses the same remote read client as virtual resources. A pure transformation maps each page, and the load step uses the existing data-seeding reconciler. Managed fields preserve local values outside the pipeline’s ownership; content comparison avoids rewriting rows whose mapped values have not changed.
Cursor and membership checkpoints let a bounded run continue without forgetting identities seen earlier. The final comparison uses that accepted history together with the current run’s observations, rather than treating earlier pages as vanished rows.
| Pass outcome | Reconciliation behavior |
|---|---|
| More pages remain | Load the current rows; defer orphan comparison. |
| Complete pass with identifiable source rows | Compare against the accumulated membership set. |
| A rejected row has no usable source identity | Load valid rows, but skip orphan inference, including REPORT. |
| Loading fails | Do not advance the accepted cursor/checkpoint; replay from the accepted position. |
The run summary separates extraction completion, load status and reconciliation status. A complete extraction is not by itself a successful import. A missing accepted membership chain is refused instead of silently replacing it with an empty set.
The resulting records support local resource behavior. Their source values reflect the last successful refresh; a virtual resource is the choice when each read should consult the remote system directly.