Most of our LLM failures were predictable

A client pipeline broke last month. The LLM read the wrong invoice total from a PDF, the next job trusted it, and nobody noticed for three days. In the review someone called it a black box problem. It was not. The PDFs from that one vendor put the total in a footer instead of a table row, and we never tested that layout.

That is the pattern. Every LLM failure we had to debug in the last year turned out to be something we could have written down in advance. Field extraction breaks when the document looks different from the layouts you tested. Ask a model to recall a specific part number and it will make one up that looks right. Reword a prompt and the output changes, because it is a sampling process and you never limited what could come in.

You do not need interpretability research to see any of this coming. You need to sit down and list where the thing breaks for your data. That is boring work, and it makes the project look worse on a slide.

What we do now is dull and it works. Before anything ships, we write a one-pager: the few failure modes most likely in this workflow, a rough hit rate from a sample of real inputs, and what happens to the next system in line when each one fires. For the invoice job that is a Pydantic schema with a confidence field, and anything under the threshold goes to a human queue instead of straight to booking. Not glamorous. We caught the footer case the second time around because it was on the list.

The real reason teams ask for “more explainability” is that it sounds like progress while putting off the uncomfortable part: once you name the failure modes, someone has to own them. Naming them makes approval harder and blame easier.

The models mostly do what the benchmarks said they would do. Ask where it breaks before you ask how it works.