How should a law firm use AI for document review?

Review has two properties that shape everything. Volume, which is why the work is expensive and why automation is attractive. And consequence, since a document that should have been produced and was not, or that should have been withheld and was not, is a problem of a different order from an ordinary error. Those two together mean the useful measurement is recall — what proportion of the relevant material the system surfaced — and not accuracy in general.

So the arrangement that holds up is measured narrowing. The system ranks or filters the set; a person reviews a random sample of what it excluded; the proportion of relevant material found in that sample is the estimate of what the narrowing is missing. This is the only construction that supports a statement about completeness, and it is the statement that matters if the process is ever examined. A tool used without it produces a smaller set and no basis for saying it was the right one.

Extraction is the second and less contested application. Dates, parties, defined terms, obligations, termination provisions, governing law, amounts: pulling these into a schedule from a stack of agreements is a well-defined task whose output sits next to its source and can be spot-checked. It is also the application with the clearest saving, because the alternative is a person reading every document for the same handful of fields.

Privilege deserves separate treatment and more caution than responsiveness. A privilege determination depends on who was involved, in what capacity and for what purpose, and those facts are frequently not in the document. A system can surface candidates by finding communications involving counsel; it cannot make the determination, and a workflow that treats its output as a decision has delegated a judgement with consequences that cannot be undone once a document has been produced.

Confidentiality applies to the whole exercise and is the first question rather than the last. Review material is client information, and the obligation to understand how a tool handles it and to obtain informed consent applies here as much as anywhere. In practice this narrows the tool choice considerably and it is better established at the outset than discovered when a client asks what was used.

Finally, the record. A defensible review is one where the method can be described afterwards: what the system did, what was sampled, what the measured recall was, who made the decisions. This is ordinary process documentation and it is the difference between a narrowing that can be explained and one that has to be defended by assertion.