Answer
How should a small business evaluate an AI vendor?
On what happens when it is wrong, what happens to your data, and what you take with you if you leave.
Three questions: what happens when it gets something wrong, what the terms say about your data, and what you take with you if you leave. Capability is the one thing a demonstration reliably establishes and the least useful to test.
A demonstration is designed to establish capability and it does so successfully, which is why capability is the wrong thing to spend the meeting on. The three things that determine whether the arrangement works over a year are all about the unhappy cases, and none of them appears unless asked for. They are also all answerable in a short conversation, which makes this a cheap evaluation rather than a thorough one.
The first is error handling. What happens when it is wrong: how you find out, whether you can see which items were affected, whether you can correct it, and whether the correction persists. A vendor with clear answers has thought about operating the thing rather than selling it, and one without them is offering a capability that will be operated on trust. Ask to be shown an error rather than to be told the rate.
The second is what happens to your data. Where it is processed, whether it is retained and for how long, whether it is used to improve their systems, who else can access it, and what contractual terms apply — including whether the terms differ between the plan being demonstrated and the one being sold. For businesses with regulatory obligations this is a threshold question rather than a preference, and the answer decides whether the rest of the evaluation is worth having.
The third is exit. What you take with you: the data, the configuration, the history, the accumulated corrections. A tool that has learned from a year of your feedback and will not export it has produced a switching cost that was invisible at purchase. This is the question vendors are least prepared for and the one that most changes the character of the relationship.
Two smaller checks are worth adding. Bring your own difficult cases to the demonstration rather than watching theirs, because the interesting information is in what fails and their examples were chosen not to. And ask what the product does not do, since a vendor who can answer that clearly is describing a bounded product and one who cannot is selling something whose boundaries you will discover.
Finally, be sceptical of specificity in the wrong place. A vendor quoting an accuracy figure without stating the population it was measured on, or promising an outcome that depends on your business rather than on their product, is doing the thing that this whole subject makes easy. A useful figure comes with what was measured, on what, and when, and a vendor who supplies that is unusual enough to be worth noticing.
Every vendor can show you it working; ask instead to be shown it failing.
Siddharth Sharma, Context Theory
Related questions
Does it matter if a product is built on someone else's model?
Less than the question implies. Most useful products are, and what matters is whether the work they have done around it — the integration, the constraints, the interface, the error handling — is worth what they charge. The question to ask is what they have built rather than what they have wrapped, and the answer is usually visible in the error handling.
How long should an evaluation take?
The three questions take an afternoon. A trial on your own real inputs, including the awkward ones, takes a fortnight and tells you considerably more than any demonstration. Where a vendor will not permit a trial on real data, that is itself an answer about the second question.
METHOD
Every figure below carries its source and the date it was verified. Nothing on this page is asserted.
The numbers on this page.
| What | Value | Specific to |
|---|---|---|
| US SMB retainer, focused one-to-two-service engagement | $1,500–$4,000 | Category-wide |
| Buyers who eliminate vendors publishing no pricing, before contact | 60% | Category-wide |
2026 agency pricing survey · per month · verified
2026 B2B buyer surveys · supersedes the 43% figure carried in blueprint v2 · verified
What is specific to this page.
| Kind | Claim | Check it against |
|---|---|---|
| Procurement | A demonstration is constructed to establish capability and does so reliably, which makes capability the least informative axis to evaluate and error handling, data terms and exit the three that determine whether the arrangement lasts. | Comparing what a vendor demonstration covers against the questions that arise during the first year of operation. |
| Constraint | Contractual data terms frequently differ between the plan demonstrated and the plan sold, so the terms question must be asked about the specific plan rather than about the product. | Comparing the published terms for the vendor's consumer and business tiers. |
| Buying behaviour | A product that accumulates value from a business's corrections and does not export them creates a switching cost invisible at purchase, which makes the exit question the one that most changes the relationship. | Asking what can be exported at the end of a contract, and in what form. |
| Response | An accuracy figure without a stated population, method and date is unusable, and a vendor supplying all three is uncommon enough that the disclosure itself is a signal. | Asking what population a quoted accuracy figure was measured on and when. |
Each row would be wrong on another industry's page. Where a sourced figure exists it is in the table above instead; these are the constraints that shape the work and do not happen to be numbers.
Start with the measurement.
Reading about a benchmark is not the same as knowing your own number. The audit produces yours, measured rather than estimated.
$497 · delivered in 5 business days · credited against month one