How do you know whether an AI workflow is reliable enough to run unattended?

The question is usually asked as though reliability were a property of the system. It is a relationship between three things: how often it is wrong, how bad a wrong answer is, and whether anything catches it. Change any one and the answer changes, which is why the same workflow can be obviously ready in one business and obviously not in another.

Start with the error rate, and measure it against real inputs rather than examples. Take a completed period, run the workflow across it, and compare its output to what actually happened. This produces two things a demonstration cannot: a rate, and a set of the inputs it gets wrong. The second is more useful than the first, because errors are almost never uniformly distributed. A workflow that is accurate overall but fails on a specific recognisable class of input is a workflow with a routing rule waiting to be written.

Then price the error. Not the average error, the undetected one, since detected errors are a cost you already pay in review. The relevant question is what happens if a wrong output goes all the way through: a customer receives something incorrect, a record is wrong until someone notices, an invoice is misfiled, a decision is made on a bad number. Where that cost is bounded and recoverable, a modest error rate is fine. Where it is a regulator, a contract or a reputation, the tolerance is different and no error rate that a language model produces will be low enough on its own.

That is the point where the answer stops being about the model. If the consequence is severe, the correct design is not a better model but a narrower job with a deterministic check around it. Constrain the output to a fixed set of values and validate it. Require a field to match a record that exists. Reject anything outside a range. Route the uncertain cases to a person instead of guessing. Each of these converts an open-ended failure into a rejected run.

Then add the observation you will need afterwards, because reliability is not a static property. Log the input, the output and the decision for every run. Sample a handful each week and check them by hand — not to catch individual errors, which is not worth anyone's time, but to notice drift, because the inputs will change and the workflow will not. The first sign of trouble in a mature automation is nearly always a change in what arrives rather than a change in what the system does.

A useful last test before going unattended: can you say, without looking, what proportion of runs need human correction and which kinds? If nobody in the business can answer that, the workflow has not been measured, whatever it has been doing. That is a statement about the evidence, not about the quality of the work, and it is the honest place to start.