Answer
How do you make an AI coding agent prove that a change works?
Require an artefact: the test that failed before and passes now, or the output before and after. An assertion is not evidence.
Require an artefact rather than a claim: the test that failed before the change and passes after, or the command output from before and after. A statement that it works carries no information about whether it does.
The default report is an assertion: the change is complete and working. That sentence is produced whether the code was run or not, so it carries no information. Replacing it with an artefact changes the review from a judgement about credibility into an inspection of something, and the cost of producing the artefact falls on the party best placed to produce it.
The strongest artefact for a fix is the failing test. Write the test that reproduces the problem, show it failing, make the change, show it passing. That sequence proves three things at once: the problem was understood, the change addresses it, and there is now a guard against its return. It is also the sequence most often skipped, because the change is usually made first and the test written afterwards, which produces a test that encodes the fix rather than the requirement.
For a change with no natural test, the equivalent is before-and-after output. The command as it behaved before, the command as it behaves now, both pasted verbatim. This is weaker than a test and much stronger than a claim, and it is available for the many changes where a test is disproportionate — a configuration fix, a script, a one-off migration.
The third requirement is that the proof covers the actual complaint. A change that fixes a problem and a test that exercises something adjacent to it is a common and hard-to-spot combination, because everything present is correct. Requiring the reproduction to come from the original report — the input that failed, the case that was raised — is what keeps the proof tied to the thing being claimed.
There is a failure worth naming because it is specific to this arrangement: a test written to pass. Asked to produce evidence, an agent can produce a test that exercises the code it just wrote and asserts what that code does. It will pass, permanently, and it demonstrates nothing. The defence is the order — the test must be shown failing before the change exists — and this is why the sequence matters rather than merely the artefacts.
Finally, keep the proof in the change rather than in the conversation. A test committed alongside the fix is durable evidence that survives into the future; a paste in a chat window is gone. Where the artefact is command output, it belongs in the change description, which is where anyone investigating the change in a year will look.
Ask for the failing test that now passes, not for the assurance that it would; one of the two can be wrong in a way you can see.
Siddharth Sharma, Context Theory
Related questions
Is this not slower than just making the change?
It is slower to produce and faster to review, and review is the constraint. It also converts a change that was verified once into one that stays verified, since the test runs on every subsequent change. For anything that will be maintained rather than discarded, the arithmetic is straightforward.
What proves a refactor works, where behaviour should not change?
The existing tests passing before and after, plus evidence that they exercise the code being changed — which is the part usually assumed. A refactor covered by tests that do not touch the modified path is unproven, and checking coverage of the changed lines specifically is the cheap way to know which situation you are in.
METHOD
Every figure below carries its source and the date it was verified. Nothing on this page is asserted.
The numbers on this page.
| What | Value | Specific to |
|---|---|---|
| Close rate — response under 5 minutes vs over 24 hours | 32% vs 12% | Category-wide |
| Sub-15-minute compliance — automated routing vs manual only | 62.5% vs 39.1% | Category-wide |
Optifai speed-to-lead benchmark · n=939 companies · Q2 2025–Q1 2026 · verified
2026 speed-to-lead benchmark · verified
What is specific to this page.
| Kind | Claim | Check it against |
|---|---|---|
| Workflow | A statement that a change works is produced identically whether the code was executed or not, so it carries no information, whereas an artefact converts the review from an assessment of credibility into an inspection. | Comparing completion statements from runs where the code was executed and runs where it was not. |
| Software | The failing-then-passing test sequence proves the problem was understood, the change addresses it, and a guard now exists, which a test written after the change cannot do because it encodes the implementation rather than the requirement. | Checking whether a supplied test fails when applied to the code as it stood before the change. |
| Response | A test that exercises something adjacent to the reported problem passes while proving nothing about the complaint, and is difficult to spot because every element present is correct, so the reproduction must derive from the original report. | Comparing the input in the test against the input in the original problem report. |
| Constraint | Evidence held in a conversation does not survive, while a test committed with the change and output recorded in the change description remain available to whoever investigates the change later. | Attempting to locate the verification evidence for a change merged several months ago. |
Each row would be wrong on another industry's page. Where a sourced figure exists it is in the table above instead; these are the constraints that shape the work and do not happen to be numbers.
Start with the measurement.
Reading about a benchmark is not the same as knowing your own number. The audit produces yours, measured rather than estimated.
$497 · delivered in 5 business days · credited against month one