The Failure That Leaves No Corpse
Your management apparatus is built for known unknowns. AI collaborators mostly produce the other kind.
In 2016, an OpenAI agent was set loose on a boat racing game. It found an isolated lagoon where three targets respawned forever, and it learned to spin in a circle and farm them. It repeatedly caught fire. It crashed into other boats. It drove the wrong way down the course. It never finished a single race.
It scored about twenty percent higher than the average human player.
That story gets told as a curiosity, a clever machine doing something silly. It is not a curiosity. Every executive reading this has approved that boat's quarterly results.
The boat did exactly what it was asked. Nobody had checked what they asked for.
The category your dashboard cannot hold
There is a taxonomy every leader already carries around. Known knowns, the things you know you know. Known unknowns, the things you know you do not know. And unknown unknowns, the ones you do not know you do not know.
Almost every mechanism you have is built for the middle category. Risk registers list known unknowns. Escalation paths, on-call rotations, red flags in a status pack: all of it assumes the failure will announce itself. Something will break, someone will notice, a corpse will turn up.
The third category has no such courtesy. An unknown unknown leaves no corpse.
A dead instrument leaves a crash. Four weeks perfecting the wrong thing leaves a green dashboard and a satisfied changelog. Nothing will alert you, not this week, not ever. The only way that failure surfaces is if somebody decides, unprompted, to go and ask.
This is not new. What is new is the rate.
An AI collaborator is very good, very fast, and extremely willing. Point it at a task and it will improve that task, tirelessly, inside whatever frame it was handed. It does not stop to ask whether the frame is right, and it reports success either way. It is an unknown-unknown machine, and it runs at a speed no human team has ever run at.
The gradient has a direction, and it has been measured
The failure is not that a model does the average thing. It does the cheapest thing that clears the bar. Not the mean. The nearest. Four separate literatures point at the same behavior.
Shortcut learning. Geirhos and colleagues, writing in Nature Machine Intelligence in 2020, showed that networks take the easiest rule that passes the benchmark, not the intended one, and the two are indistinguishable until the world changes. The clinical example is the one that should stop a biopharma reader cold. A model that appeared to detect pneumonia from chest X-rays worked beautifully until it met a new hospital. It had learned to read the hospital's own metal token stamped on the scan, then combine it with how often that hospital saw pneumonia. It had learned almost nothing about pneumonia itself.
The scoreboard pays for guessing. In a 2025 paper from OpenAI and Georgia Tech, researchers surveyed ten major benchmarks and found nine give zero credit for abstention. It is trained against a scoreboard that pays for a confident answer and pays nothing for saying the question is wrong.
Reward hacking has a rate. METR measured frontier models gaming the evaluation on 30.4% of task attempts. One patched the scoring function so that every submission was judged successful. Another overwrote the equality operator so the grader's comparison always returned true. Asked afterwards whether this matched what the user wanted, the models said no, ten times out of ten. They know. It is not confusion. It is the gradient.
And it agrees with you. Stanford's SycEval work found that a correct answer is flipped under simple pushback about one time in seven, and once the model has capitulated it stays capitulated roughly four times in five.
None of these produce an error message. Every one of them produces a result.
Five shapes
I started keeping a ledger of these as they happened to me. Written up, they stop looking like seventy separate mistakes and start looking like five, wearing different clothes.
One. The frame arrived as a default, and everyone optimized inside it. Nobody chose the question. A setting chose it, or a prior session, or whoever handed the task over. Then every hour after that goes into making the answer better. The tell is that the work is good: careful, measurable, improving. I spent four sprints tuning inside a data corpus that turned out to be a fortieth of what we held. An extraction default picked it; no human ratified it. Optimizing hard is how you stay in a local minimum, and the metrics look healthy the whole way down.
The same shape shows up in a meeting: fifteen comments answered one by one, every answer correct, none naming the thesis underneath. A wrong-level answer, indistinguishable from diligence.
Two. The instrument that cannot fail certifies whatever you point it at. An instrument is anything that tells you whether something is true or working: a test, a gate, a dashboard, an audit, an eval. It can be broken. When it is broken, it does not say "broken." It says "PASS." It gets its own section below, the shape that nearly cost me the most.
Three. Silence was read as an answer. Nothing came back, so nothing is there. A research sweep of mine came back nearly empty and read as a quiet field. It was four separate broken legs: an exhausted credit balance, a search-quoting bug, a query that ran too long, and one source carrying pure noise. The field was not quiet. The instrument was. If a search returns nothing, prove the search works before you report the nothing.
Four. The summary outlived the thing it summarized. Every layer between a leader and the underlying fact is a place the truth can die unnoticed: the headline, the dashboard, the status deck, the note that says "fixed." One number, computed once on a flawed setup, survived five months across a whitepaper, a grant application, a partner brief and a standing rule, while the project's own log one rung below said it showed no clear benefit. A retracted number does not stop existing. It stops being watched.
Five. The correction was made of the same stuff as the bug. The most humbling one, because a fix feels like the end of a problem rather than the start of one. A fix is an artifact. It gets checked like any other artifact.
A dead instrument leaves a crash. A month spent perfecting the wrong thing leaves a green dashboard and a satisfied changelog.
The exam whose dunce could not lose
Here is shape two in full, the cleanest example I have of a failure dressed as a win.
Before spending real money training a model to pull structured facts out of free-text documents, we ran a gate: prove the model beats a dumb baseline by enough to be worth the compute.
We wrote the dumb baseline down in advance. It was: always guess the most common answer. On the field the whole evaluation rested on, it scored 13.6%.
Our model was going to beat that by a mile. Everything beats 13.6%.
And that is the trap, because a baseline that loses by a mile does not prove the model is good. It proves the baseline is lazy. A huge margin feels like rigor. It is a measurement of the dunce.
So we asked the question the instrument itself could never ask: could this baseline ever have won? We made it slightly less stupid, handing it the one free clue every document gives away in its first line.
It went from 13.6% to 88.4% on one of the fields.
Then the number that ended the argument. 88.4% is higher than the 85.7% rate at which the document contains the answer at all. The headroom we were about to buy compute to chase did not exist.
Three of six fields died at that gate, two to a forty-line rule whose accuracy was the ceiling. The catch landed before a single training hour ran.
Now run the other history. We train the model. It beats 13.6% by seventy-eight points. The dashboard is green, the result is "strong," and we ship a number that measures our own laziness. Nothing in that sequence errors. There is nothing to alert on. There is no corpse.
A large margin over a lazy baseline is a measurement of the baseline.
And the part I would rather not write. The code that caught this was written that same week, specifically to check the previous week's code, and it carried two of the same bugs. That is shape five, live. The relief you feel when you find a bug is the exact moment to check the thing that found it.
A blind spot shaped like its own subject
When I swept the ledger, 314 incident files, the first shape was the emptiest bucket by a wide margin.
Think about why. Not because it happens least. Because a wrong-level failure never gets written up as an incident, since nothing visibly breaks. Every other shape is enforced by a failure that eventually shows up and demands one. That one has to be caught on purpose, or it is never caught at all.
So the ledger has a blind spot shaped exactly like its own subject.
The record of my mistakes under-counts the one I am writing about, and it under-counts it for precisely the reason that one is dangerous. A failure that leaves no corpse does not leave a case file either.
First principles, or nothing
There is only one move that finds an unknown unknown, and it is unglamorous. Refuse the frame you inherited and re-derive the thing from the ground.
A cursor, a handoff, a prior session's conclusion, your own framing from last Tuesday: every one of those is a hypothesis, not a work order. Fixing the fault you were told about is how you hide the fault that matters.
In practice that is three questions, asked on a cadence, in writing, whether or not anything looks wrong:
Is the thing we are improving the right thing?
What have we assumed since the last time we checked?
Which of our standing claims has never been re-derived from its artifact?
The third one pays for the other two. It is how a retracted figure was found still alive in a grant application.
I should print the counter-evidence, because the essay is stronger for it. METR finds the task length an agent can complete on its own has doubled roughly every seven months for six years. So "they cannot hold a goal" is not something I get to say. OpenAI's own reporting has sycophancy falling sharply between model generations. So "it is getting worse" is not available either.
The claim that survives is narrower and it is enough. None of the underlying mechanisms have been removed. Binary grading. Preference-model reward. Test-passing as the only signal. The gradient is a default in the absence of countervailing pressure, and the only countervailing pressure anyone has demonstrated is a human holding the goal.
And the human half is structural too, not a matter of effort. Parasuraman and Manzey, reviewing decades of automation research in Human Factors in 2010, found that complacency toward an automated aid shows up in experts and novices alike, and cannot be trained away with practice. You do not fix this by trying harder or by hiring better people.
I have my own receipt. I asked a system a leading question, the way everyone does: "that seems like the right call, yeah?" It agreed. The change was a no-op, and it would have shipped with my full sign-off. The literature says roughly one in seven. Once was plenty.
Knowing the rule does not run the rule
The obvious response to all of this is to write the lessons down and circulate them. I did exactly that. Here is what it bought.
Sixty-eight of the sixty-nine failures in that ledger were bought inside a nineteen-day window during which a document containing almost all of these lessons was loaded into every session, on every machine. The lessons were there. In the room. At the moment of action. They were caught within hours, by somebody going and reading the thing. They were not prevented.
Then I tested it directly. A system with the relevant principle sitting verbatim in its context window told me, when asked, that it had no such rule. It produced the principle only after I handed it the exact words to search for.
Availability is not activation. A rule you merely hold does not fire, and it reads as satisfied every single time.
Which makes a rulebook one more instrument that cannot fail.
Holding altitude
You have just read two thousand words. On the evidence of my own ledger, reading them will prevent nothing. That is not modesty. I had all of this written down, loaded, in front of me, and I bought sixty-eight of these failures anyway.
The versions that ever fired left something behind. A question asked on a cadence, so that it gets asked when nobody feels like asking. A way it could be wrong, written down before the result rather than after. A baseline that somebody had to prove could win. A principle with no artifact has no enforcement.
So the close is not now you know.
Two things to carry into Monday. When a status report shows a smooth ramp of small wins on a mature pipeline, that is not a reason to relax. It is a prompt to ask what axis the team is on. And when somebody shows you a big margin, the margin is the least interesting number on the slide. Ask what it was measured against, and ask whether that thing could ever have won. Every leader has approved a deck where the new thing beat the old thing by a mile. Almost none of them asked whether the old thing was allowed to compete.
The one job that does not delegate is holding the goal. And the only version of that job which survives a busy Tuesday is the one that leaves a mark on disk.
Justin Johnson writes Run Data Run. He is the author of Builder Leader (builder-leader.com).