The production gap and the disappearing failure report are two symptoms of the same problem: an open loop. The tool that improves fastest is the one that closes it — tightly, deliberately, as a first-class part of how the product is built.
Not all failures are equally useful. A failure on a document the tool has never seen before is the most valuable feedback it can produce — but only if you capture it before it disappears.
A document tool's performance on your evaluation set and its performance on your users' actual documents are two different numbers. The gap between them is structural, not a bug — and closing it requires a different kind of work than improving the eval.
No tool handles the entire long tail. The behavior that separates a trustworthy tool from a dangerous one is what it does on the document it can't handle: decline honestly, or guess and hope.
The easy documents are all easy in the same way, and a tool handles them on day one. The value — and the difficulty — lives in the long tail of documents that are each weird in their own particular way.
If the tail is the product and honest declines mark its edge, then the work is a slow walk down the tail — turning each declined document into a handled one. That walk is what compounds into a tool nobody can catch.
Users don't carefully audit every field a tool extracts. They skim. A tool that assumes a thorough review gets one that doesn't happen — so the output has to be built for the glance, not the audit.
A user reviewing a tool's output has a small, fixed amount of attention to spend. The tool's real job at the review stage is to spend that budget where it changes outcomes — not to hope there's more of it than there is.
The dangerous extraction error isn't the one that looks broken — the user catches that. It's the one that looks exactly like a right answer and sails straight through the quick review.
If the last mile is getting output into the user's workflow, the first mile is getting the document in. The friction at the start of the task quietly decides whether the tool gets used at all.