Keep a record of how your experts corrected the agents
When an agent proposes an action and a qualified person approves it, amends it or rejects it, that decision is recorded: who decided, on which agent’s proposal, for which case, and how far the final version is from what the agent proposed.
For a firm whose agents carry out its method, these corrections are the expertise itself. The record shows how often each agent, tool or case needs an expert to step in, and it is the material for improving the agents. The content of a correction is kept only when the client organization chooses to keep it, and reusing it as evaluation data needs a recorded legal basis.
Example — A reviewer amends an agent's proposed adjustment
An agent proposes a ledger adjustment for a client’s month-end close. The accountant reviewing it changes the account and approves, choosing the reason “wrong target” from the firm’s list. The decision is recorded against the agent, the tool and the case. At the end of the quarter the firm sees that this agent’s adjustments are amended far more often than its reconciliations, and looks at that tool first.
For engineers
The decision record, always
Every decision a person takes on something an agent proposed is recorded once, in agentWorkDecisions: a tool call approved, edited or denied, a reviewable output from an actor flow, or an edit to a row an agent drafted. The record holds the outcome, the decider, the agent, the tool, the case, the timestamps, the changed paths, an edit distance and two digests keyed per organization. It holds no free text and no content.
Only the settle that wins the decision writes the record, so a retried review or a double settle produces one row. If the process stops between the decision and the write, a repair pass writes it from the decision’s own row. The record is bound to the audit retention class: it is the evidence of how agent work was supervised, so it outlives the content it describes.
Reasons the reviewer chooses
The application declares the reasons a reviewer can give for a correction. The reviewer’s picker offers them on every decision about agent work, and a code the application did not declare is refused. A reason is recorded only with an override outcome. This excerpt is from Wonder Todos’ backend configuration:
agentWorkReview: {
reasonCodes: ['wrong_target', 'wrong_wording', 'not_requested'],
privilegedCaseResourceTypes: [],
// A todo's subject is the client it names: an organization may then keep that client out of every evaluation export.
caseSubjects: [{ caseResourceType: 'todos', subjectField: 'externalCustomerId' }],
},
The content, only on the organization’s choice
By default an organization keeps metadata only. It can choose to keep the content too: the proposal, the final version, the difference between them and a written reason. The content is bound to the ordinary data retention class, read only by the role the organization names as its content reader, and scrubbed when the case is erased.
The override rate
From the metadata alone, the organization’s settings show the override rate over a period, grouped by agent, tool, case or decider. Lapsed and withdrawn decisions are counted apart, so they neither raise nor lower the rate.
Exporting corrections as evaluation data
An organization can authorise exporting its decisions as an evaluation dataset, on one of two recorded bases: the firm acting as processor on its client’s documented instruction, or as controller after assessing that the further use is compatible. Without an active authorisation the export is refused. Kept content leaves only after every identifier its declared shape names is replaced by a pseudonym; content whose shape cannot be resolved is withheld. An organization can exclude clients: no case of an excluded client leaves, including cases opened after the exclusion, and adding a client marks the datasets that already carried its cases. Every export is recorded.