Keep searchable knowledge up to date
A useful knowledge base must follow the information it comes from. Wildo derives passages from declared resource fields and maintains the derived corpus in background work.
Embedding work and reconciliation have separate jobs: one produces vectors, while the other checks that stored knowledge still matches the source records and declarations.
Example — Update a document after it is indexed
Changing a knowledge document updates its derived passages. Background embedding produces vectors for the new content, while reconciliation can recover missed changes and remove passages whose source no longer exists.
For engineers
This declaration selects markdown chunking and a deliberate overlap. The same ingestion pipeline is used by the live write path and reconciliation, so a later chunking change can rebuild existing material as well as future writes.
const knowledgeText = z.object({
title: z.string().min(1).max(200).ragSource(),
body: z.string().max(200_000).ragSource({
chunkingStrategy: RAGChunkingStrategy.MARKDOWN,
chunkSize: 1_200,
chunkOverlap: 150,
}).optional(),
});
The embedding drain runs on its five-minute schedule and processes a bounded backlog, including embeddings stamped with an old model binding. The hourly sweep checks content and chunking stamps, removes orphan sets, repairs scope stamps and reports embedding backlog; it also visits missing sets. These are converging background mechanisms, not a promise that every change becomes semantically searchable synchronously. No application-owned chunk cleanup or embedding scheduler is needed for this path.
Run the workers and inspect convergence
The engine registers both batch factories during application startup. The deployment must run its batch/cron execution infrastructure, with the RAG chunk store and ingestion services available. Vector work additionally needs an enabled embeddings provider and its credentials in the worker runtime. The drain reports an absent provider rather than manufacturing vectors; it leaves the text corpus available for lexical retrieval.
Inspect the completed batch results and their structured summary logs. A declaration alone does not establish that a worker ran or that the backlog cleared:
| Result | What it tells you |
|---|---|
Drain providerConfigured, embeddingModel | Whether vector work had a resolved model binding |
Drain chunksEmbedded, chunksReembeddedForMigration | New vectors versus migration to another model binding |
Drain failedEmbedCalls | Provider calls needing investigation; their chunks remain for retry |
Sweep hashMismatchReingested, configMismatchReingested | Content updates versus changed chunking declarations |
Sweep orphanedChunkSetsDeleted, missingSetsIngested | Removed sources versus sources being indexed for the first time |
Sweep embeddingBacklogChunks | Text chunks still waiting for vectors; the sweep counts, the drain produces them |
Either reachedRunCeiling | Work remains for subsequent bounded runs |
After a source or model change, expect migration and backlog counters to converge. Repeated failures or a backlog that does not drain need inspection of the recorded provider/ingestion errors, rather than another application-owned scheduler.