Recover security deliveries after an outage
Customer security collectors can become unavailable without stopping the application. Wildo queues delivery, retries failures and retains exhausted deliveries for an administrator to inspect and replay.
Example — Recover deliveries when a receiver fails
A customer repairs an expired collector certificate, reviews the retained failures and replays their events through the normal delivery path.
For engineers
The organization’s SIEM configuration contains delivery.retryAttempts, retryBackoffSeconds and timeoutSeconds. An illustrative selected configuration is:
delivery: {
retryAttempts: 3,
retryBackoffSeconds: 30,
timeoutSeconds: 10,
},
The dispatcher builds and signs the delivery job; the worker checks its integrity before attempting the configured destination. The worker makes an initial attempt plus the configured retries: retryAttempts: 3 permits up to four HTTP attempts. Failed attempts use the configured backoff. The shared circuit-breaker contract pauses a destination after five consecutive failures for five minutes, protecting delivery capacity for other receivers.
Use the retained failure as the recovery unit
The dead-letter record retains the original event envelope, failure reason and attempt information. It does not freeze the formatted HTTP request. The persisted failure counter starts at one and the last-attempt timestamp records that retained failure; these are not a complete HTTP attempt history. The organization’s retrySingle operation accepts an entryId, loads the current scoped delivery configuration and queues the event before deleting the dead-letter entry. Replay therefore uses the current destination, format and authentication settings. retryAll performs the corresponding scoped sweep.
The implementation’s successful path is explicit:
await enqueueEnvelope(dispatcher, scope, siemConfig, envelope);
await deleteRequeuedRows(repo, executionContext, scope, [entryId], utils.logger);
These selected lines are from siem-delivery-dlq.custom-impl.backend.service.ts. A failed enqueue leaves the record available. Cleanup follows successful queueing; a later delivery failure creates its own retained failure record. Receivers should tolerate replay rather than treating the pipeline as exactly-once delivery.
The original business operation does not wait for the collector. Inspect delivery outcomes, not only the business response, when confirming a customer’s export is operating.