April 2027 · 8 min
Trust Is Denominated in Errors
Users don't ask how often a model is right. They ask what it cost them the last time it was wrong.
We shipped a model that was right ninety-four percent of the time, and the field stopped using it in a month. Later we shipped one that was measurably worse, and it became infrastructure. It took me an embarrassingly long time to understand why, because the answer was not on any evaluation dashboard.
The first model’s errors were expensive and random. The second’s were cheap and predictable.
Accuracy is an average, and nobody experiences an average. A superintendent experiences the one morning the system told him a crew was safe when it wasn’t. From that moment his question is never “how accurate is it” — it is “can I afford to be wrong the way this thing is wrong.” Trust is not denominated in percentage points. It is denominated in the cost, the direction, and the predictability of the errors.
Predictability matters most. People work comfortably beside a tool that fails in a known shape — always conservative, always flags borderline cases, always misses the same rare pattern — because they can build their own judgment around it, the way you drive a car with a known blind spot. A tool that fails in a new shape each time cannot be worked around, only supervised. And supervising a tool is more work than not having it.
So the engineering target shifts. Past a threshold, marginal accuracy is worth less than error shaping: making the failures land on the cheap side, in the visible place, in the same shape every time. This is a design discipline with almost no tooling and no leaderboard, which is why demo-driven teams skip it and why their ninety-four percent models die in the field.
Doctrine: users price a model by its worst plausible morning. Shape the errors before you chase the average.