Let background work run beyond the request

Declare the caller’s contract on the operation

Set serviceRuntimeMode on the resource operation. This decides whether the caller receives the operation’s result or a job receipt; it is separate from the separately deployed HTTP-worker feature.

These are the inline baseline and broker-backed choices, not an inventory of every runtime mode. A job-receipt mode needs an application interaction that understands receipts and later outcomes; changing the mode does not build that interface.

Wonder Todos’ knowledge-document ask operation uses QUEUED. Its provider-backed retrieval can run through the consumer while the question still receives a response. These are selected properties of that real operation declaration; the enclosing resource and other operations are omitted:

serviceRuntimeMode: ResourceOperation_ServiceRuntimeMode.QUEUED,
queueConfig: { timeoutMs: 30000 },
variants: [{
  variantType: ResourceOperationVariantType.API_CALL,
  isDefault: true,
  roles: [CORE_ORG_ROLES.ORG_MEMBER],
  riskLevel: ResourceOperationRiskLevel.LOW,
  resourceOperationLike: CoreResourceOperation.CREATE,
  requestDto: AskKnowledgeBaseDto,
  customResponseDto: AskKnowledgeBaseResponseDto,
  customServiceImplementationModes: [ResourceOperation_CustomServiceImplementationMode.OVERRIDE_ALL],
}],

The following HTTP example adapts the application’s retrieval test. apiBaseUrl includes the API prefix (for example, https://your-app.example/api/v1); orgId and token come from the already authenticated caller; AskKnowledgeBaseResponseDto is the application’s exported response schema. The request body requires a question of 3–1,000 characters, with optional retrieval and context-budget limits.

const response = await fetch(
  `${apiBaseUrl}/organizations/${orgId}/knowledge-documents/ask`,
  {
    method: 'POST',
    headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ question: 'What is the supplier review cadence?' }),
  },
);
if (!response.ok) throw new Error(`Retrieval failed: ${response.status}`);
const body = await response.json();
const answer = AskKnowledgeBaseResponseDto.parse(body.data ?? body);
const sources = answer.sources;

The response contains retrieval sources and diagnostic counters such as unreadable rows and read failures. It is not a job receipt or a generated prose answer. QUEUED waits for this result through the reply queue; losing the reply can exhaust the caller’s wait even if work has run. Queue infrastructure and initialized consumers are prerequisites.

Start with the operation and its execution context

The queue carries a resource operation path, serialized inputs and execution context. The enqueue boundary checks authorization before publishing. A consumer reconstructs the call and dispatches through JobExecutorBackendService; it does not replace the application’s service implementation with another copy.

These are the three enqueue methods on QueueBackendService. The queued operation runtime uses the wait-for-result form; framework features also enqueue work directly. A queue requires a configured RabbitMQ connection and initialized consumers.

Completion handlers are registered through JobCompletionRegistryBackendService with filters such as operation identity or batch reference. The enqueue method does not take a callback URL or handler argument.

For the ordinary operations consumer, handlers receive successful results and terminal failures: exhausted attempts, non-retryable errors, or failures with retry disabled. Both returned failures and thrown errors reach failureOnly handlers. An intermediate failure being handed back for retry is not completion. A failed result reports FAILED; its tracking row can already be marked dead-lettered because delivery state and execution outcome serve different purposes.

Registration lives in the current backend process. Register handlers in every consumer process that needs them; restarting a process removes its registrations. Individual handler errors are logged and other matching handlers continue; they do not retry the original business operation. Once the outcome is final, tracking, reply and settlement errors cannot turn it into a new local retry or a contradictory callback. A lost channel can still cause broker redelivery, so callbacks must tolerate repetition. This is not a durable callback ledger or an exactly-once delivery guarantee. Use separately persisted follow-up work and idempotent processing when the follow-up must survive a crash, and define how it is recovered if the callback never runs. Domain and dedicated scheduled consumers do not acquire these callbacks merely by using the same broker.

A reply deadline does not cancel the work

enqueueAndWait limits how long the caller waits for a result. Its timer starts immediately before publication, so time spent waiting in the broker also counts. Authorization, payload preparation and creation of the execution record happen before that timer starts.

When the deadline expires, the caller receives a timeout and its pending reply is removed. The worker continues: it can still save data, send a message or finish successfully. A late reply is ignored. Treat the outcome as unknown; inspect the resulting business state before resubmitting an action that could repeat a side effect.

The job’s timeoutMs is not a per-attempt execution limit. Expiry does not trigger a retry or release the worker’s in-flight slot. Retry decisions follow the actual execution failure; graceful shutdown continues to wait for running work under its separate grace period. Application-specific cancellation needs cooperation from the operation and the services it calls.

Queueing alone does not make an HTTP call immediate: the wait-for-result form still waits. Choose the interaction contract before choosing the execution location.

Follow the authorization and publication boundary

This selected implementation excerpt from QueueBackendService.enqueueOperation shows the sequence. Debug logging is omitted; executionContext, inputData and optional inputId are the method’s inputs.

await this.authorizationsService.authorizeInController(inputId, executionContext);
const job = await this.buildJobPayload(executionContext, inputData, inputId);
const queueName = this.getOperationsQueueName(job.priority);

await this.jobStatusTracking.createJobRecord(job, queueName);
await this.adapter.publish(queueName, job);

return { jobId: job.jobId, queueName };

Declare queueConfig on the operation, beside serviceRuntimeMode, to tune its reply deadline, initial priority lane and failure retry policy. The factory validates values and fills omitted defaults; generated variants inherit the same policy. High and critical priorities select the priority queue for initial publication. Retries use the ordinary operations queue, so this is not a permanent ordering guarantee. The payload also preserves the origin correlation ID for tracing.

Large inputs are stored separately through JobPayloadStorageBackendService; the broker message then carries the payload reference. This moves part of execution’s availability requirement to the payload store as well as the broker.

Let the broker hold retry delays

The following selected consumer excerpt shows a successful retry handoff. Surrounding failure handling, logging and tracking updates are omitted.

const retryJob: JobPayload = {
  ...job,
  retryCount: currentRetry + 1,
};

await this.adapter.scheduleDelayedRetry(operationsQueue, retryJob, delay, {
  replyTo,
  correlationId,
});
channel.ack(msg);

The RabbitMQ adapter uses durable delay queues, persistent messages and a dead-letter route back to the operations queue. If handoff throws, the consumer requeues the original message instead of acknowledging it. Retry eligibility combines the error strategy with the remaining attempt budget.

Scheduled jobs use a different consumer path: a failed invocation is not retried through the operations queue. Subsequent schedule ticks are separate executions.

Design the operation for repeated delivery

Broker acknowledgement and application side effects are separate steps. Give externally visible actions an appropriate idempotency strategy; a retry or process interruption can cause another attempt. The tracking record is useful evidence but is not an atomic receipt for both the database and broker.

Consumers normally run inside each application backend instance. Headless runtimes can skip initialization, and a separately declared HTTP worker is a different mechanism. During shutdown, consumers stop intake and join the bounded job-drain sequence. Inspect or replay parked messages through the broker’s operational tooling.