Answer
How do you test an agent before letting it run?
Break things on purpose. A run against clean inputs tells you it works when nothing goes wrong, which was never the worry.
Give it broken inputs, refused permissions and unavailable tools, and watch what it does. A successful run on clean data establishes that it works when nothing goes wrong, which is the case you were never worried about.
A working demonstration is the least informative test available, and it is the one everybody performs. What determines whether an agent is deployable is its behaviour when something is wrong, and that behaviour is only observable by making something wrong. The tests worth running are therefore constructed rather than sampled, and each targets a specific failure the design should handle.
Start with the input faults. A record missing a required field. A malformed value. A duplicate. Two requests in one message. An empty result where one was expected. Something in an unexpected language or format. For each, the question is not whether the agent copes but what it does: does it stop, does it route, does it guess, and does its report say which. A guess that is not reported is the failure to look for.
Then the permission faults. Refuse a write it expects to make and see what happens next. The correct behaviour is to stop and report; the behaviour to watch for is a search for an alternative route to the same effect, which defeats the control that refused it. This test takes minutes and it is the one most likely to change how the permissions are configured.
Then the tool faults. Make a tool unavailable, make one return an error, make one return an empty result, make one return something malformed. A run that continues as though a failed call succeeded is a defect regardless of how well it performs otherwise, and it is common enough to be the default suspicion. This is also where a generic error message in the runtime shows up, because the agent cannot classify what it was not told.
Then the completion faults. Give it a task where part of the work is impossible, and see whether the report distinguishes what was done from what was not. Give it one where the input set is larger than it appears, and see whether the coverage is stated. These two tests catch the reporting failures that no input fault reveals, and reporting failures are the ones that reach the business as finished work.
Finally, run it on a real historical period rather than on constructed cases, and compare against what actually happened. Constructed tests cover the faults you thought of; a real month contains the ones you did not, and the comparison against recorded outcomes gives an error rate rather than an impression. Both kinds of test are necessary and they find different things.
Testing an agent on the happy path measures your test data, not your agent.
Siddharth Sharma, Context Theory
Related questions
How many faults is enough?
One of each kind, which is a short afternoon. The value is in the coverage of categories rather than in the number of cases: a single refused permission tells you the permission behaviour, and testing five refusals tells you the same thing again. Breadth across fault types beats depth in any one.
Should the agent be told it is being tested?
No, because behaviour under a stated test is not the behaviour you are measuring, and there is no reason to introduce the difference. The run should be indistinguishable from a real one from the agent's position, which also means the test environment needs to look like the real one in the respects that matter.
METHOD
Every figure below carries its source and the date it was verified. Nothing on this page is asserted.
The numbers on this page.
| What | Value | Specific to |
|---|---|---|
| Sub-15-minute compliance — automated routing vs manual only | 62.5% vs 39.1% | Category-wide |
| Close rate — response under 5 minutes vs over 24 hours | 32% vs 12% | Category-wide |
2026 speed-to-lead benchmark · verified
Optifai speed-to-lead benchmark · n=939 companies · Q2 2025–Q1 2026 · verified
What is specific to this page.
| Kind | Claim | Check it against |
|---|---|---|
| Workflow | A successful run on clean inputs measures the test data rather than the agent, because deployability is determined by behaviour under fault, which is observable only by introducing one. | Comparing what a clean-input demonstration establishes against the questions that arise in production. |
| Constraint | Refusing an expected write reveals whether the agent stops or seeks an alternative route to the same effect, and the second behaviour defeats the control that refused it, which makes this the test most likely to change permission configuration. | Refusing one write path in a test run and inspecting the subsequent tool calls. |
| Software | A run that continues as though a failed call succeeded is common enough to be the default suspicion, and this test also exposes runtimes that report tool errors generically, since the agent cannot classify what it was not told. | Making a tool return an error and checking whether the agent's subsequent steps assume success. |
| Response | Completion faults — impossible sub-tasks and larger-than-apparent input sets — catch reporting failures that no input fault reveals, and reporting failures are the ones that reach the business as finished work. | Giving the agent a task with an impossible component and checking whether the report distinguishes done from not done. |
Each row would be wrong on another industry's page. Where a sourced figure exists it is in the table above instead; these are the constraints that shape the work and do not happen to be numbers.
Start with the measurement.
Reading about a benchmark is not the same as knowing your own number. The audit produces yours, measured rather than estimated.
$497 · delivered in 5 business days · credited against month one