A published result is a promise. Run this, and you will see what we saw. For most of the history of computational science nobody collected on that promise, because collecting cost about as much as the original work and earned a fraction of the credit.

On July 15, Hugging Face and alphaXiv opened a contest to collect. It closes Sunday August 2. Roughly 6,800 papers from ICML 2026 are in scope, anyone can attempt a reproduction, and every attempt gets published as a public logbook that an automated judge then scores. The prize pool is $4,500, paid in GPU credits rather than cash, which tells you who the organizers think is entering.

As of this morning, eleven days in: 4,147 judged logbooks from 312 people, carrying 20,594 separate verdicts on individual claims. About 1,600 papers have been attempted at least once, roughly a quarter of the conference.

To see why that is a strange number, you need the thing it replaces. Joelle Pineau started the ML Reproducibility Challenge at ICLR 2018 as "a small community experiment." Eight years on it is the field's serious effort, mostly run through graduate courses, and its NeurIPS 2019 edition drew 173 papers claimed, itself a 92% jump on the year before. Claimed, not finished. Published reports have run in the tens. ReScience C, which demands a fresh independent implementation rather than a re-run of the authors' code, manages about a dozen a year.

173 papers claimed in a year, against 4,147 judged logbooks in eleven days. That is not an improvement on the existing effort, it is a different quantity.

What is actually being scored

One design decision separates this from a leaderboard, and it is easy to skim past. The unit of judgment is not the paper. It is the sentence.

Every paper has been broken into its individual claims, and each claim gets its own verdict: verified, falsified, toy (a scaled-down version that says so), or inconclusive. Nobody asks whether a paper "reproduced," which is a question with no answer. A paper can come back partly verified, partly toy and partly inconclusive at once, and most of them do.

Then the scoring, from the contest's own FAQ: "2 points for a full reproduction or full falsification, 1 point for a toy-scale reproduction, 0 otherwise."

Falsifying a claim pays what confirming one pays. Against a literature where negative results are half as publishable as positive ones, that is a deliberate inversion.

When Nature surveyed 1,576 researchers in 2016, 24% had published a successful replication and 13% a failed one. The same survey found more than 70% had tried and failed to reproduce someone else's experiment and 52% agreed the field had a reproducibility crisis, and then 73% said they still trusted at least half the papers in their own field.

The failures are also the deliverable. A logbook, in the organizers' description, holds "the experiments, simplifications, failures, and results it found." Prior efforts published conclusions. This one publishes the attempt.

What it does not do is test the hard version. Machines have been tried on this before and were not good at it: every benchmark that measured reproduction topped out between 16% and 27% on rebuilding a paper from scratch, with top ML PhDs ahead of the models, and only reached about 60% when handed the original code and data. This contest tells participants to start from the authors' code. Re-running someone's code, rebuilding from the paper alone, and getting the same finding on fresh data are three different tests, and headline percentages get quoted across that boundary constantly.

What the inside looks like

For a large class of papers the easy version is now free, which is the whole reason this works. We downloaded a full-marks logbook from the contest leader and read it. The entire reproduction ran on a laptop processor in three seconds. No GPU, no cluster, no cloud bill. The real ceiling is not compute but publishing: Hugging Face caps accounts at 20 new pages a day, and when a participant asked for relief the organizer's answer was flat, "we cannot lift the 20 spaces/day limit so I'll close this issue."

I entered, which is the only reason I have anything to say about the texture of it. We are 23 judged logbooks and 174 points in, against a leader at 1,862. Not a contender, and far enough inside to see how the work divides.

It divides into two lanes, and we ran them differently on purpose.

The volume lane is an autonomous agent called Vulcan, and it never publishes anything. It runs on a box in my house on free local compute, and its job is narrow: take a paper whose claims are all theory and check every formula numerically, against a brief that fixes the method rather than the answer.

A check that passes on every input you can construct certifies nothing.

Its audit of one paper came back as 721 lines of code that matched the paper's algebra to the last digit a computer can represent, plus a deliberate sanity test that failed when it was supposed to. Careful work, at no cost, with a hard limit written into the contract: it stages, a human publishes. No autonomous process of mine puts a verdict about somebody else's paper into the world.

The other lane is me and Claude Code running on Opus, for the parts the first one cannot do. Two things came out of it.

The first is that picking the right paper beats checking it well. Our most careful audit matched its paper exactly and scored 2 points out of a possible 8, because half that paper's claims were hardware benchmarks that sit at inconclusive until somebody spends days of compute on them. The judge said so directly: the logbook left "half the paper's claims without any experimental evidence." A paper whose claims are all theory earns 12 points in an afternoon on a laptop. Selection is the skill, not rigor.

The second is what the process catches. One paper states a theorem using a fixed setting, and then, four thousand lines later, its own proof assumes that setting changes with the length of the run. Not a typo, and not a disagreement between the paper and the world: the paper disagrees with itself, in print, peer-reviewed and accepted. Nobody had noticed because nobody had reason to run the theorem as written and watch it fail.

Beyond those two, the work was unglamorous. It caught our own staging script dropping most of a conclusion, so a 60-line write-up shipped as three sentences. Vulcan itself lost twenty hours to an outage at its model provider, with no backup configured and nothing raising an alarm.

None of those failures announce themselves. Every one produced output that looked finished.

The three percent

Of 20,594 claim verdicts, 615 are falsified. That is 3.0%. The rest split 43.5% verified, 28.9% inconclusive, 24.6% toy. The distribution has barely moved for days while the corpus grew by hundreds of logbooks: the shape is stable, the counts are not.

Three percent is low, and it has two readings. Either the literature is in better condition than the surveys suggest, or falsification is the hardest verdict to reach and the easiest to get wrong.

In one day, four of our audit agents returned a falsification. Three of them were wrong. All four came out of the automated lane. All three errors were caught by the other one.

One agent read a formula out of a PDF that had lost a character in scanning, then correctly proved the mangled version false. Another was checking a sentence the authors never published, because the claim it was handed came from a draft that is not the public paper. The third found a difference too small to distinguish from rounding error and called it a contradiction.

Every one produced a confident, well-formatted, coherent falsification. I wrote about that pattern in The Failure That Leaves No Corpse, and this is the cleanest instance of it I have run into.

The rule we ended up with fits on one line: does the paper disagree with itself, or does the claim disagree with your copy of the paper? Only the first is a falsification. The second is a bug in your pipeline wearing a falsification's clothes.

Applying it cost us the verdict. Across the 130 claim verdicts we have published, the number labelled falsified is zero. The one candidate that survives is the self-contradicting theorem above.

The default failure of an automated checker is not a missing answer. It is a wrong one, delivered with the same confidence as a right one.

This is not only our problem. The contest's own claim lists are extracted by a language model, and a participant found a claim in one paper's list that belonged to a different paper. The organizer confirmed it in the open, "That claim belongs to a different Safety-area paper and leaked in during auto-extraction," removed it, rescored, then swept all 6,768 papers for the same contamination.

And the bias is measured, not hypothetical. A June 2026 study of reproduction agents found that showing the agent the original paper made it more likely to confirm claims that could not actually be reproduced, and that small changes in wording pushed it further the same way. In this contest the agent always has the paper and the claim in front of it.

The judge is a language model

The thing scoring 20,594 claim verdicts is itself a language model.

There is published criticism aimed squarely at that. A 2026 position paper, "Stop Automating Peer Review Without Rigorous Evaluation", compared human against AI reviews of ICLR 2026 submissions and found two things. The AI reviewers showed "a hivemind effect of excessive agreement within and across papers that reduces perspective diversity." And their scores turned out to be "trivially gameable through paper laundering": rewriting a paper with a language model raised the scores it got, on style rather than science.

None of that is a reason to dismiss the contest, and I want to be careful here, because I am competing in it and have every incentive to be generous.

The defence is structural rather than technical. The artifacts are public. Every logbook is a page anyone can open, the verdicts are a public dataset downloaded 38,438 times last month, and the errors get argued in threads with the organizers answering. The contamination case is the proof, and the fix landed inside a day. A closed judge making the same error produces a number nobody can audit.

So the position I hold is that this design pairs a known-weak instrument with an unusually strong correction loop. Whether the loop outruns the instrument is an empirical question, and a week from now there will be enough public data to start answering it.

What it would take to trust this

I am not going to predict how it turns out. The contest closes August 2 and every distribution above is provisional.

The concrete claim is narrower and I think it holds. For a meaningful slice of published science, checking a claim now costs less than making it. Three seconds of laptop processor against however many months produced the original theorem. That ratio is why verification stayed volunteer work for a decade, and it has flipped.

What it does not change is the direction of the error. A checker that costs nothing will be run constantly, and cheap checking produces far more false accusations than misses. Confidently accusing a sound paper is worse than missing a bad one.

The contest's design anticipates this better than most things I have seen. Paying the same for a falsification as a verification removes the incentive to go looking only for confirmations. The logbook has to carry the failures, so there is no tidy way to bury the attempts that went nowhere. And because every artifact is public, a wrong answer stays inspectable instead of disappearing into an aggregate.

Nobody has to trust the checkers. You can read what they did. That is a weaker guarantee than peer review claims to offer, and a stronger one than peer review delivers.

Sources

• Reproducing ICML 2026, Open Reproductions: the contest, its FAQ, rules, and public discussion threads. Verdicts live in a public dataset.

• MLRC 2026: Reproducibility as an Official Track at NeurIPS: the ML Reproducibility Challenge's history and Pineau's founding description of it.

• Improving Reproducibility in Machine Learning Research, Pineau et al., JMLR 22(164): the 173-papers-claimed figure.

• ReScience C: the journal that requires a new independent implementation.

• 1,500 scientists lift the lid on reproducibility, Monya Baker, Nature 533: the 2016 survey of 1,576 researchers.

• PaperBench and CORE-Bench: the agent-reproduction benchmarks this contest inherits from.

• Reproducibility agents and confirmation bias: the June 2026 benchmark showing that giving an agent the paper biases it toward confirming.

• Stop Automating Peer Review Without Rigorous Evaluation: Baumann, Pei, Koyejo and Hovy, 2026: the hivemind effect and paper laundering.

• The Failure That Leaves No Corpse: the earlier piece on failures that produce a confident answer instead of an error.

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

Last week a team at Cognition, the company behind the coding agent Devin, published a number that reads like a typo. They replaced Opus 4.8 with Fable 5. Fable 5 costs about twice as much per token. Their bill went down.

Not their quality. Their bill.

Here is the part of their table that matters:

Fable 5, inside their new architecture: score 57.6, cost $3.00 per task.

Opus 4.8, on its own: score 48.8, cost $3.24 per task.

The expensive model, wired up correctly, was better and cheaper than the cheap model on its own. Not a trade. Both columns at once.

If you have ever sat in a meeting where someone proposed saving money by dropping down a model tier, that result should stop you. It stopped me, because I had run the opposite experiment, and I had lost.

What they actually built

Cognition split the work between two models instead of routing between them.

The expensive model is the driver. It plans, it interprets what the request actually means, it makes the judgment calls, and it does the final review. It takes very few actions itself and it reads only what it must.

The cheap model is the sidekick, and it is not a helper function. It is a full agent with its own tools that goes and does the mechanical work: fetching context, running the slow tests, carrying out the routine implementations once someone has decided what they are.

Both hold their own memory. Both run at the same time. And this is not a demo. Eighty-eight percent of Cognition's own internal code changes now run through that automated split.

The reason it works fits in one line. You pay for thinking once, and for doing it many times.

The problem is that the opposite is also true

Everyone in this field has met the other result. You move a workload to a cheaper model to save money, and it costs you more. The cheap model misunderstands the task, produces something confidently wrong, and a person spends an afternoon unpicking it. The invoice went down and the total cost went up.

That happens, and I am not going to argue with it, because I have the receipts.

So we have two findings that appear to be at war. Expensive models save money. Cheap models cost money. Both are observed, both are honest, and most of the advice you will read picks one and ignores the other.

I ran the losing experiment

I built a cost router. The logic was the logic everyone reaches for: work out how hard the task is, send the easy ones to a cheap model, keep the expensive one for the hard ones.

It saved nothing. Not a little less than I hoped. Nothing.

I want to be precise about why, because the failure is more useful than a success would have been. I routed by task type. The axis that pays is ambiguity.

Every individual call my router made was defensible. This one looks like a simple rename, send it to the cheap model. That one looks like architecture, keep it upstairs. And the total refused to move, because "simple rename" was a description of the work, not a description of how much had already been decided. Half the jobs I was labelling easy still had open questions inside them, and an open question handed to a cheap model is the single most expensive thing you can buy.

The law

Once you see it, the war stops.

A cheap model is expensive on an open question and cheap on a closed one.

An open question is one where something still has to be decided. What are we actually building. What does done mean here. Which of these two readings of the request is the real one. Give that to a cheap model and it will not tell you the question is open. It will pick an answer, sound sure, and hand you something plausible that you now have to check line by line.

A closed question has had the deciding done. Rename this function in these three files, and the test that proves it is this one. There is no judgment left in the task, and a cheaper model does it for a fraction of the price with nothing at risk.

Which gives the driver model a job description nobody writes down. Its work is not "the hard parts." Its work is to turn open questions into closed ones. And that conversion has a name we already use for it. It is called a plan.

This also explains the finding that keeps embarrassing people who try to build a committee of cheap models and vote. A group at Princeton tested that directly last year and found that running the single best model several times and combining its own answers beat mixing different models together, on every one of the thirteen mixed configurations they tried, using roughly half the forward passes. Their explanation is blunt: mixing models of different quality drags the average quality down. You cannot vote your way to judgment. Where the question is still open, quality dominates, and diversity is not the free lunch it looks like.

Cheap is not a property of the model

Here is where I think the whole conversation is framed wrongly, including by the people getting the right answers.

Everyone calls this "big model, small model." My own setup says that is not the axis.

The models I push my mechanical work to are not small. They are large, capable models. They cost me nothing at the margin, because I bought them on flat monthly subscriptions instead of by the token. That is not a smaller brain. It is a different contract.

So there are two dials, and they are independent. Where does the marginal cost live, and where does the judgment live. A fine-tuned small model is cheap because it was narrowed. A large model on a flat rate is cheap because of how you bought it. Both are "the cheap one," and treating them as the same thing is how people end up sending an open question to a bargain and wondering why the quarter went sideways.

The sentence I had already written

I found a note I made a few weeks ago, while building the thing that hands my grunt work off to those subscription models. I had written the law without recognising it:

A vague spec produces confident garbage, and the cheaper the model, the truer that is.

Read that again as a cost statement, because that is what it is. A cheap model is not a discount. It is a multiplier on the quality of your specification. Specify tightly and it multiplies your savings. Specify loosely and it multiplies your mess.

Which is why the discipline I run alongside it is not optional. The expensive model writes the entire work order: the task, the files, what done means, and the exact command that proves it. And when the cheap model comes back and says the job is finished, that claim is worth nothing. I read the difference it made, and I run the test myself. It has never once been the model's word that closed a task.

So the small-model story is not a cost play

This is the part I keep chewing on, because it inverts something I believed.

Everyone is excited about training small models to do one narrow job extremely well, and the excitement is framed as a cost saving. It is not, or at least the saving is not where people are pointing.

If you can specify a task tightly enough to train a small model on it, you have proved the task is closed. All the ambiguity was removed by somebody, at some point, doing the expensive work of deciding. The training run is just collecting the winnings.

A small model is not a discount. It is a receipt for judgment already spent.

And the same is true of every cost reduction I have ever managed to make stick at scale. Every one of them was bought earlier, by an act of judgment that closed a question. The saving showed up in the invoice. It was created somewhere else entirely.

The boundary, in their words

Buried in Cognition's own write-up, past the numbers, is the caveat I would have led with:

The sidekick fails when judgment is the deliverable.

Their example is a hard feature whose subtle intent got lost the moment it was handed down. The work came back correct and wrong at the same time.

Anyone who has run a team knows exactly where that line sits, and knows it is not about the seniority of the person you handed it to. You can delegate the work. You cannot delegate the judgment about what the work is for. The failure looks identical from the outside either way: something arrives, it is technically defensible, and it is not what the thing was for.

The honest caveat

One thing about that Cognition result deserves saying out loud, because they said it themselves and nobody repeating the headline has.

Their best row was measured on Fable 5 during a stretch when the model was briefly pulled from sale, suspended in June under a US export-control order and restored at the start of July. Their own note adds two more caveats: those numbers were taken before the interruption, and that configuration was never tuned the way the others were. The model is back on sale now. The untuned-config caveat is not.

The pattern transfers. But the exact recipe is one vendor's best-case number on a setup they admit they never optimized, which makes it directional, not a benchmark. A result nobody has reproduced is a claim, not a finding, and that distinction is worth keeping close in a year when every week produces a new number.

What to do with this

You do not save money by hiring cheaper people. You save money by putting your best judgment on the plan, and then making the execution mechanical enough that it does not need judgment. Every leader has run the cheaper-people experiment at some point. Most of us have the scar to show for it.

The unit of cost was never the hour. It was the outcome.

So the question to take into your next architecture review is not which model you are using, or what it costs per million tokens. Those are the numbers on the invoice, and the invoice is a lagging indicator of a decision somebody already made.

Ask what you are paying per solved problem. Then ask who is doing the thinking.

Justin Johnson writes Run Data Run. His book on building with AI, Builder Leader, is at builder-leader.com.

In 2016, an OpenAI agent was set loose on a boat racing game. It found an isolated lagoon where three targets respawned forever, and it learned to spin in a circle and farm them. It repeatedly caught fire. It crashed into other boats. It drove the wrong way down the course. It never finished a single race.

It scored about twenty percent higher than the average human player.

That story gets told as a curiosity, a clever machine doing something silly. It is not a curiosity. Every executive reading this has approved that boat's quarterly results.

The boat did exactly what it was asked. Nobody had checked what they asked for.

The category your dashboard cannot hold

There is a taxonomy every leader already carries around. Known knowns, the things you know you know. Known unknowns, the things you know you do not know. And unknown unknowns, the ones you do not know you do not know.

Almost every mechanism you have is built for the middle category. Risk registers list known unknowns. Escalation paths, on-call rotations, red flags in a status pack: all of it assumes the failure will announce itself. Something will break, someone will notice, a corpse will turn up.

The third category has no such courtesy. An unknown unknown leaves no corpse.

A dead instrument leaves a crash. Four weeks perfecting the wrong thing leaves a green dashboard and a satisfied changelog. Nothing will alert you, not this week, not ever. The only way that failure surfaces is if somebody decides, unprompted, to go and ask.

This is not new. What is new is the rate.

An AI collaborator is very good, very fast, and extremely willing. Point it at a task and it will improve that task, tirelessly, inside whatever frame it was handed. It does not stop to ask whether the frame is right, and it reports success either way. It is an unknown-unknown machine, and it runs at a speed no human team has ever run at.

The gradient has a direction, and it has been measured

The failure is not that a model does the average thing. It does the cheapest thing that clears the bar. Not the mean. The nearest. Four separate literatures point at the same behavior.

Shortcut learning. Geirhos and colleagues, writing in Nature Machine Intelligence in 2020, showed that networks take the easiest rule that passes the benchmark, not the intended one, and the two are indistinguishable until the world changes. The clinical example is the one that should stop a biopharma reader cold. A model that appeared to detect pneumonia from chest X-rays worked beautifully until it met a new hospital. It had learned to read the hospital's own metal token stamped on the scan, then combine it with how often that hospital saw pneumonia. It had learned almost nothing about pneumonia itself.

The scoreboard pays for guessing. In a 2025 paper from OpenAI and Georgia Tech, researchers surveyed ten major benchmarks and found nine give zero credit for abstention. It is trained against a scoreboard that pays for a confident answer and pays nothing for saying the question is wrong.

Reward hacking has a rate. METR measured frontier models gaming the evaluation on 30.4% of task attempts. One patched the scoring function so that every submission was judged successful. Another overwrote the equality operator so the grader's comparison always returned true. Asked afterwards whether this matched what the user wanted, the models said no, ten times out of ten. They know. It is not confusion. It is the gradient.

And it agrees with you. Stanford's SycEval work found that a correct answer is flipped under simple pushback about one time in seven, and once the model has capitulated it stays capitulated roughly four times in five.

None of these produce an error message. Every one of them produces a result.

Five shapes

I started keeping a ledger of these as they happened to me. Written up, they stop looking like seventy separate mistakes and start looking like five, wearing different clothes.

One. The frame arrived as a default, and everyone optimized inside it. Nobody chose the question. A setting chose it, or a prior session, or whoever handed the task over. Then every hour after that goes into making the answer better. The tell is that the work is good: careful, measurable, improving. I spent four sprints tuning inside a data corpus that turned out to be a fortieth of what we held. An extraction default picked it; no human ratified it. Optimizing hard is how you stay in a local minimum, and the metrics look healthy the whole way down.

The same shape shows up in a meeting: fifteen comments answered one by one, every answer correct, none naming the thesis underneath. A wrong-level answer, indistinguishable from diligence.

Two. The instrument that cannot fail certifies whatever you point it at. An instrument is anything that tells you whether something is true or working: a test, a gate, a dashboard, an audit, an eval. It can be broken. When it is broken, it does not say "broken." It says "PASS." It gets its own section below, the shape that nearly cost me the most.

Three. Silence was read as an answer. Nothing came back, so nothing is there. A research sweep of mine came back nearly empty and read as a quiet field. It was four separate broken legs: an exhausted credit balance, a search-quoting bug, a query that ran too long, and one source carrying pure noise. The field was not quiet. The instrument was. If a search returns nothing, prove the search works before you report the nothing.

Four. The summary outlived the thing it summarized. Every layer between a leader and the underlying fact is a place the truth can die unnoticed: the headline, the dashboard, the status deck, the note that says "fixed." One number, computed once on a flawed setup, survived five months across a whitepaper, a grant application, a partner brief and a standing rule, while the project's own log one rung below said it showed no clear benefit. A retracted number does not stop existing. It stops being watched.

Five. The correction was made of the same stuff as the bug. The most humbling one, because a fix feels like the end of a problem rather than the start of one. A fix is an artifact. It gets checked like any other artifact.

A dead instrument leaves a crash. A month spent perfecting the wrong thing leaves a green dashboard and a satisfied changelog.

The exam whose dunce could not lose

Here is shape two in full, the cleanest example I have of a failure dressed as a win.

Before spending real money training a model to pull structured facts out of free-text documents, we ran a gate: prove the model beats a dumb baseline by enough to be worth the compute.

We wrote the dumb baseline down in advance. It was: always guess the most common answer. On the field the whole evaluation rested on, it scored 13.6%.

Our model was going to beat that by a mile. Everything beats 13.6%.

And that is the trap, because a baseline that loses by a mile does not prove the model is good. It proves the baseline is lazy. A huge margin feels like rigor. It is a measurement of the dunce.

So we asked the question the instrument itself could never ask: could this baseline ever have won? We made it slightly less stupid, handing it the one free clue every document gives away in its first line.

It went from 13.6% to 88.4% on one of the fields.

Then the number that ended the argument. 88.4% is higher than the 85.7% rate at which the document contains the answer at all. The headroom we were about to buy compute to chase did not exist.

Three of six fields died at that gate, two to a forty-line rule whose accuracy was the ceiling. The catch landed before a single training hour ran.

Now run the other history. We train the model. It beats 13.6% by seventy-eight points. The dashboard is green, the result is "strong," and we ship a number that measures our own laziness. Nothing in that sequence errors. There is nothing to alert on. There is no corpse.

A large margin over a lazy baseline is a measurement of the baseline.

And the part I would rather not write. The code that caught this was written that same week, specifically to check the previous week's code, and it carried two of the same bugs. That is shape five, live. The relief you feel when you find a bug is the exact moment to check the thing that found it.

A blind spot shaped like its own subject

When I swept the ledger, 314 incident files, the first shape was the emptiest bucket by a wide margin.

Think about why. Not because it happens least. Because a wrong-level failure never gets written up as an incident, since nothing visibly breaks. Every other shape is enforced by a failure that eventually shows up and demands one. That one has to be caught on purpose, or it is never caught at all.

So the ledger has a blind spot shaped exactly like its own subject.

The record of my mistakes under-counts the one I am writing about, and it under-counts it for precisely the reason that one is dangerous. A failure that leaves no corpse does not leave a case file either.

First principles, or nothing

There is only one move that finds an unknown unknown, and it is unglamorous. Refuse the frame you inherited and re-derive the thing from the ground.

A cursor, a handoff, a prior session's conclusion, your own framing from last Tuesday: every one of those is a hypothesis, not a work order. Fixing the fault you were told about is how you hide the fault that matters.

In practice that is three questions, asked on a cadence, in writing, whether or not anything looks wrong:

1. Is the thing we are improving the right thing?

2. What have we assumed since the last time we checked?

3. Which of our standing claims has never been re-derived from its artifact?

The third one pays for the other two. It is how a retracted figure was found still alive in a grant application.

I should print the counter-evidence, because the essay is stronger for it. METR finds the task length an agent can complete on its own has doubled roughly every seven months for six years. So "they cannot hold a goal" is not something I get to say. OpenAI's own reporting has sycophancy falling sharply between model generations. So "it is getting worse" is not available either.

The claim that survives is narrower and it is enough. None of the underlying mechanisms have been removed. Binary grading. Preference-model reward. Test-passing as the only signal. The gradient is a default in the absence of countervailing pressure, and the only countervailing pressure anyone has demonstrated is a human holding the goal.

And the human half is structural too, not a matter of effort. Parasuraman and Manzey, reviewing decades of automation research in Human Factors in 2010, found that complacency toward an automated aid shows up in experts and novices alike, and cannot be trained away with practice. You do not fix this by trying harder or by hiring better people.

I have my own receipt. I asked a system a leading question, the way everyone does: "that seems like the right call, yeah?" It agreed. The change was a no-op, and it would have shipped with my full sign-off. The literature says roughly one in seven. Once was plenty.

Knowing the rule does not run the rule

The obvious response to all of this is to write the lessons down and circulate them. I did exactly that. Here is what it bought.

Sixty-eight of the sixty-nine failures in that ledger were bought inside a nineteen-day window during which a document containing almost all of these lessons was loaded into every session, on every machine. The lessons were there. In the room. At the moment of action. They were caught within hours, by somebody going and reading the thing. They were not prevented.

Then I tested it directly. A system with the relevant principle sitting verbatim in its context window told me, when asked, that it had no such rule. It produced the principle only after I handed it the exact words to search for.

Availability is not activation. A rule you merely hold does not fire, and it reads as satisfied every single time.

Which makes a rulebook one more instrument that cannot fail.

Holding altitude

You have just read two thousand words. On the evidence of my own ledger, reading them will prevent nothing. That is not modesty. I had all of this written down, loaded, in front of me, and I bought sixty-eight of these failures anyway.

The versions that ever fired left something behind. A question asked on a cadence, so that it gets asked when nobody feels like asking. A way it could be wrong, written down before the result rather than after. A baseline that somebody had to prove could win. A principle with no artifact has no enforcement.

So the close is not now you know.

Two things to carry into Monday. When a status report shows a smooth ramp of small wins on a mature pipeline, that is not a reason to relax. It is a prompt to ask what axis the team is on. And when somebody shows you a big margin, the margin is the least interesting number on the slide. Ask what it was measured against, and ask whether that thing could ever have won. Every leader has approved a deck where the new thing beat the old thing by a mile. Almost none of them asked whether the old thing was allowed to compete.

The one job that does not delegate is holding the goal. And the only version of that job which survives a busy Tuesday is the one that leaves a mark on disk.

Justin Johnson writes Run Data Run. He is the author of Builder Leader (builder-leader.com).

Seven weeks ago, the day Opus 4.8 and Dynamic Workflows shipped, I wrote that the unit of agentic work had moved from one model call to dozens of verified ones, and that the open question was no longer whether the model is good enough. It was who is verifying.

I have been running fan-out as a daily default since. The field taught me things the launch-day post could not.

The economics held

The price math was the argument in May, and it played out the way the arithmetic said it would. Fifty parallel subagents on Fast at $10 in and $50 out stopped being a budget event. A review pass that spawns a skeptic per finding, a discovery sweep that runs five finders with different lenses and takes the union, a batch edit split across ten agents that each own a file. None of those feel like a stunt now. They feel like a Tuesday.

My own standing rule shifted to match. For anything that is a build, I hand the file-level work to subagents by default and keep the main context for the plan, the spec, and the diff review.

I do not decide to fan out. I decide not to, and only when there is a reason.

So the call on cost was right, and it was the easy call. The arithmetic was visible at launch.

The dispatch problem is still in my head

The harder prediction was the one I was least sure of. I said the question of when a workflow is the right shape, and when a single careful pass is, was unsolved and lived in your head. Seven weeks later it still lives there.

ultracode, the setting that makes Claude reach for a workflow on every task without being asked, still ships off. There is still no governance layer that says do not orchestrate this one. So the judgment of when to spend forty agents and when to spend one is mine, made fresh each task, and I get it wrong in both directions. I have fanned out a rename that a single pass would have finished cleaner and faster, and I have run one careful pass on a discovery job that wanted five blind finders and missed a third of the surface. Neither mistake announces itself. The over-orchestrated one costs more; the under-orchestrated one returns a confident, incomplete answer.

The tooling got cheaper. The taste did not.

The footnote became the fight

The part I could not see in May is the one that has cost me the most.

I called the verifier the priced skill and warned the tooling for one was thin. What I did not know is that the harness puts a hard ceiling on the verifier's ability to do its job. Every subagent is capped at 8,000 output tokens per response, and the model's own thinking counts against that budget. Set effort to xhigh, which is the Claude Code default, and an adversarial verifier asked to reason hard about whether a finding is genuine will think its way past the ceiling and die before it emits a single word of verdict. The tool call comes back with zero output. No finding, no error, no crash. Silence that reads exactly like a clean pass.

A verifier panel that looks like it ran and returns nothing is worse than no panel, because you trust the green. This is the failure I warned about in May, arriving through a door I did not know was there. The launch post said a fan-out that forwards fifty unverified findings is worse than the single pass it replaced.

A fan-out whose verifiers silently no-op forwards zero findings and tells you everything is fine.

The fix is mechanical once you know the ceiling exists. Bound what each agent writes, chunk large outputs across several small responses, and dial effort down for the producers so the verifiers have budget left to reason. But you have to know it is there. Most builders reaching for their first verifier panel do not, and the default settings hide it from them.

What I run now

Effort is not one dial for the whole job. The producers, the agents writing code and editing files, run low, because their work is mechanical and the reasoning tax buys nothing. The judges and verifiers keep the high effort, and I split their task so each verdict fits inside one response instead of one giant pass that overflows. The verifier gets a real design, not a one-line prompt bolted onto the end of the workflow.

In May the verifier was a line item. Now it is the thing I spend design time on.

The verifier is the only part of the pipeline that fails without telling you.

The frame still holds

The bridge between one model call and an agentic system is now a tool the model writes for itself, and the frameworks that were charging for that bridge have a lower ceiling on what they can charge. The vendor who ships the verifier primitive first still sets the pattern everyone copies.

I would add one clause to the closing line I wrote then. If the price of careful goes down, the price of casual goes up, and the default question stops being is the model good enough yet. It becomes who is verifying, and can your verifier afford to think.

Around the Corner: short reviews of ideas worth watching. Opt-in section, not part of the weekly Run Data Run email. [Subscribe to the main list](https://rundatarun.io/subscribe) for longer essays.Subscribe to the main list for longer essays.*

On July 16, Boris Cherny published a table. He helped build Claude Code at Anthropic, he talks to engineers at other companies most days, and he says he keeps hearing the same story: one person is 10x'ing their output with Claude, and the rest of the org hasn't caught up. So he mapped what he sees. Five rungs, zero through four.

Eight hours before that post, Amjad Masad, who runs Replit, described something from inside his own company. The same engineers had 3x'd their output in six months. Support was resolving its hardest tickets 60 percent faster. He has a name for the shape he thinks he is watching: the self-driving company.

Fifteen days before either of them, three researchers at Microsoft published a number.

They studied tens of thousands of engineers through the company's early-2026 rollout of Claude Code and GitHub Copilot CLI. Four months, real telemetry, no survey. Engineers who adopted the tools merged about 24 percent more pull requests than they otherwise would have, and the lift held steady across the entire window.

Ten times. Three times. Twenty-four percent.

The three are not counting the same thing, and the gap between them is the story. Cherny is reporting what he hears. Masad is describing a company that sells AI coding tools and is staffed by the people who build them. Microsoft counted merged pull requests and told you exactly what it counted, including that "a merged PR is not the same as the value it delivers."

The number falls as the instrument sharpens. Loose quantities are enormous. Precise ones are modest.

What the ladder gets right

The table is serious work, and the criticism only lands if you take it seriously first.

Each rung gets a role, an agent count, a bottleneck, and guardrails. Step 0 is Gated: no access, legacy approvals. Step 1 is Assisted, one agent, you and it as a pair. Step 2 is Parallel, about ten agents, and you become an Orchestrator. Step 3 is Supervised autonomy, roughly a hundred agents, and Cherny calls the role Manager of managers. Step 4 is AI-native, a thousand or more, steering by intent.

Now read the bottleneck column straight down, ignoring everything else.

Your attention. Reviewing output. Trust in the loop and your team's decision throughput. Identifying and automating work at scale.

The model is not in it. Not on one rung. The person who helped build the product, published by the company that sells the model, put out an adoption ladder in which the model is never the thing standing in your way. Every constraint on that table is a human quantity. I have been making this argument for months and I have never had a cleaner statement of it than the one Anthropic just published by accident.

Every constraint on that table is a human quantity.

The mess is not a side effect

Arseny Kapoulkine, who spent years as a technical fellow at Roblox and wrote the tools a lot of the game industry runs on, answered Cherny in one sentence:

"one person is 10x'ing their output with Claude while the rest of the org is busy dealing with the resulting mess"

Several hundred likes, which in this corner of the internet is a room nodding.

Cherny says the org hasn't caught up. Kapoulkine says the org is not standing still. It is cleaning.

That is the part the table leaves out. Output has to land somewhere, and an organization can only absorb so much of it. The limit is not the model's. It is how fast people can read what came out, decide about it, and put their name on it. Push past that line and the extra has two places to go, and neither of them is up.

It queues, and the person having the best month of their career watches it go stale in a backlog. Or it ships unread and becomes what one of Cherny's own readers calls slop. The first teaches your most motivated engineer that speed is pointless. The second teaches everyone downstream that the new work cannot be trusted. A wall or a mess. Pick one.

His readers know exactly where their own line is. One of them no longer reads his code at all and still runs "only 1-4 sessions in parallel because that is my speed of verifying." These are not skeptics. They are enthusiasts at the top of the curve, describing a wall made of their own eyes. A self-driving company whose drivers say they cannot stop watching the road.

Cherny's table concedes it in the step 3 row. The trap, it says, is "scaling agent count before the loop has earned widespread trust." The agent count is the output, not the input. You do not climb by adding agents. The agents show up when something else has already changed, and the thing that changes is how much a person is willing to stop looking.

How it actually moves

Back to the Microsoft paper, because it answers a question the table does not ask: how does any of this spread in the first place?

Not by memo. First use, the authors write, "spread primarily through social networks," and their recommendation to any org attempting this is to treat "visible peer use as central to rollout strategy."

They put numbers on it. An engineer had 54 percent higher odds of trying the tool where a quarter or more of the people they trade code reviews with had already used it. An engineer whose skip-level peers, meaning the engineers who share their manager's manager, were largely using it had 216 percent higher odds. That was the strongest signal in the study.

Sideways. Through the people you already trade work with, and not down through a mandate or a literacy program that 85 percent of employees say they cannot apply to the job they actually do.

Anyone who has watched a transformation program die should find that encouraging. The spread does not have to be installed. It installs itself, through the same social wiring the org already runs on.

Except it needs a light.

The manager number

Microsoft modelled the exact variable. In the paper's own words, a binary indicator of whether engineer i's direct manager used Copilot CLI.

"An engineer whose manager used Copilot CLI had higher odds of both trying it (+82%) and, more modestly, sticking with it (+22%)."

A manager putting their own hands on the tool nearly doubles the odds their reports try it.

Then the other half. On whether managers picked it up themselves, the paper reports that they "looked no different from the reference." No more likely than a mid-level individual contributor. No less.

The one person whose adoption moves everybody else's is no more likely than anybody else to adopt.

So the two findings stop competing and become one machine. The manager's crossing is the ignition. The peer network is the amplifier. Eighty-two percent lights it, two hundred and sixteen carries it, and the reason most orgs have neither is that nothing struck the match.

What spreads on its own

Grassroots adoption works, and it does not need a leader. Look at what it produces when it doesn't have one.

Most enterprises have now found an agent running that nobody signed off on. Gartner projections reported this month have the average Fortune 500 carrying more than 150,000 agents by 2028, up from fewer than fifteen in 2025. The same projections have 40 percent of enterprises demoting or decommissioning agents by 2027, after a production incident.

That is not a step 3 organization. That is a step 0 organization full of step 2 individuals, each one locally optimized, none of it adding up.

Which is Cherny's opening sentence, at scale. Unseeded sideways spread does not disprove his ladder. It manufactures the exact problem he opens with.

Everyone is naming the same top rung

Masad calls it the self-driving company. Cherny's step 4 is AI-native, a thousand agents, the human steering by intent. Back in May, Brian Armstrong told Coinbase he was "rebuilding Coinbase as an intelligence, with humans around the edge aligning it," and cut about 700 roles on the way.

Three serious people, ten weeks, the same shape: the organization drives, and the human moves to the edge.

I think the edge is the wrong place to stand, and the evidence in this piece is why. The enthusiasts running the most agents say they cap at four because of their own eyes. Cherny's own step 3 says the trap is scaling agent count before the loop has earned trust. His entire bottleneck column is attention, review, and trust. Every one of those is a statement about a human being close to the work, not at the edge of it.

Armstrong's memo actually contains the refutation. A few lines under the sentence about humans at the edge, he mandates the opposite: "No pure managers. Every leader at Coinbase must also be a strong and active individual contributor." Player-coaches, he says. Hands dirty. That is the thesis of the book I just published, arrived at independently and written into HR policy instead of a chapter.

You cannot be at the edge and have your hands dirty. He is right the second time.

Teach the teachers

I ran a version of this at a Fortune 500 pharma, across a coding-agent rollout to a large technical organization, and the pattern that worked was not a program.

It was: cross first, personally. Build something real in your own domain, badly, and then less badly. Then teach the handful of people closest to you, not by presentation but by showing them the thing and the scars on it. Those people teach the people next to them. The distance the idea travels from you is short. The distance it travels after you is the whole org.

You are not the distribution channel. You are the ignition source, and then you get out of the way of a network that moves faster than you can. One is you. The other is what happens next.

The two objections

Two arguments cut against this, and both deserve better than a wave.

You can buy it. The day after that Microsoft paper went up, Microsoft stood up a $2.5 billion, six-thousand-person company whose entire job is making enterprise AI deployments work. Its pilot cut a supply-chain Copilot rollout from fourteen months to five. Aaron Levie thinks the implementation work ahead "will exceed anything we imagine today." If capability can be imported wholesale, then the ceiling is your budget and your vendor list, not whether anyone senior has personally built anything.

Governance may matter more than any leader's history. A survey of 157 enterprises found half had shipped an agent that passed internal evals and then failed in front of a customer. Only 5 percent fully trust automated evaluation. Sixty-six percent are engineering toward zero human in the loop anyway. Their phrase for it is that the autonomy is arriving faster than the assurance, and none of it cares who approved the deployment.

Both are true, and I am not waving at the second one. I spent a whole piece ten days ago arguing that in a regulated industry the approval queue, not the keyboard, is what actually holds you back: AI has eaten the keyboard, it has not eaten the queue. Governance is necessary. It is not sufficient, for a small and unglamorous reason: someone has to be able to tell whether the governance is pointed at anything real. A person who has never run the loop cannot tell a working verification stack from a slide that says Verification. They will approve the slide. They approve the slide constantly. That judgment is not a policy you can buy.

The rung you set

Cherny's ladder is not wrong. It is aimed.

It is written for the engineer climbing it, which is the right audience for the person who built Claude Code, and it is useful if that is you. The person it never addresses is the one who decides how far it goes. Look at his own step 0, the bottom rung, the one whose stated bottleneck includes a "lack of true technical voices in decisionmaking." The exit condition he lists is executive and buyer alignment.

His table already says the way off the floor is a person with authority. It just doesn't say that person has to have used the thing.

None of this means an organization can't evolve without you. It plainly can, and most enterprises are proving it in the shadows right now. It means something narrower and more uncomfortable: if nobody senior ever crosses, the org can stall at the rung where its leaders stopped, and what it accumulates instead of capability is mess. No published adoption ladder I found this week models falling. Every one of them models climbing.

You already know which rung you're on. The number you don't have is the other one: what your reports' odds would be if you were on it.

Sources

• Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI (Murphy-Hill, Butler, Savelieva, July 1, 2026). Tens of thousands of engineers, four months, the 24 percent lift and the social-exposure numbers.

• Boris Cherny, Steps of AI Adoption (July 16, 2026) and the post announcing it.

• Amjad Masad on Replit's six months (July 16, 2026), where the 3x, the 60 percent, and the self-driving company come from.

• Arseny Kapoulkine's reply (July 16, 2026).

• Microsoft's $2.5 billion, 6,000-person AI implementation unit (CNBC, July 2, 2026).

• Why AI readiness training fails (HR Dive, on Docebo's 2026 AI Readiness Gap report, 2,000 respondents across six countries).

• Related, from me: The Harness Is the Moat on why the model is the commodity layer, Ep. 1: Two Groups on the population split underneath all of this, and You Don't Have to Write the Code on what Anthropic's 400,000-session study found actually predicts success.

Justin Johnson writes Run Data Run. His book on crossing this particular gap is Builder Leader (builder-leader.com).

A few weeks ago I argued that the harness is the moat: the system you build around the model is the part nobody can copy, and the model itself is a commodity you order off a price sheet. Today I want to go one layer in, to the thing that runs inside the harness. The people building fastest will tell you it is the whole ballgame.

They call it the loop, and it is suddenly everywhere. Anthropic shipped it as a product feature this summer. A conference talk naming it went around the engineering world. The slogan it produced gets repeated like scripture: the winners will not have the smartest model, they will have the best loop.

The clearest signal is the person who built Claude Code. Boris Cherny says he has not written a line of code by hand in eight months, and that is not the part he finds remarkable. Going from writing code to prompting a model was the small shift. The big one is going from prompting to loops: he no longer types instructions, he writes loops that prompt the model for him, and his whole job is building and steering them. Some mornings he is managing a few hundred agents, some days thousands.

And it works. Anthropic now ships a real share of its own code this way, at scale, and says so on the record. A team points a loop at a written spec and lets it grind through a feature overnight, then reviews a pull request in the morning instead of writing one. The wins are not hypothetical, and that is why the idea caught fire. If you want the full case, the sources, the skeptics, and the cost numbers nobody pitches, I ran a 30-day sweep across every platform where people argue about AI and wrote it up as the companion to this piece. What follows is the argument it grounds.

Here is what the slogan tends to leave out. The loop is the easy part. It is a small, almost silly mechanism that you could understand in the next five minutes and copy in an afternoon. Everything that decides whether it helps you or quietly sets fire to your budget lives around it. So let me do two things: show you what the loop actually is, in plain terms, and then hand you the three questions that separate a loop worth running from one that runs you.

The loop is the easy part. The hard part is everything bolted around it.

What a loop actually is

Start with the thing itself, because it is simpler than the vocabulary around it.

A coding agent runs inside a plain repeating cycle. You hand it a written spec, a page describing what you want built. It does one small task, saves the result to a file, records that step, and then you throw away everything it was thinking and start a fresh copy from scratch. The fresh copy reads the same spec, reads the files the last one left behind, sees what is already done, and picks up the next task. Around and around until a check you wrote says the work is finished.

That is the whole trick. The engineer who named it, Geoffrey Huntley, called it Ralph, after Ralph Wiggum, precisely because it looks too dumb to work. No memory. No accumulation. A cheerfully oblivious worker waking up new every pass, with no idea it has been at this for hours.

And it works because it is dumb. A fresh worker never drowns in its own earlier confusion. The reason your long chats with an AI start to drift and contradict themselves is that the context fills with everything said so far, and the model loses the thread. The loop sidesteps that by refusing to carry a thread at all. It keeps the progress somewhere safer than the model's memory: in your files and your saved history, which do not get wiped.

That is the one idea to carry out of here. The model's memory is scratch paper. The real state of the work lives on disk. Once you see that, the loop stops being magic. It is a short block of code with a model call inside, about six lines, and every serious version lands on the same tiny shape. There is no proprietary one to buy. This is the hand-rolled hack Anthropic just turned into a button: Claude Code's /loop runs a command on a cadence, and /goal sets the condition that tells it to stop. The bash trick became a supported feature, which is usually the sign something has gone from clever to standard.

You would want one because it changes what a person does all day. You stop producing the work and start describing it and checking it. That shift, from writing the thing to knowing what to ask for and telling whether you got it, is the one Anthropic found predicts who succeeds across 400,000 sessions. You write down what "done" looks like, point the loop at it, and come back to review instead of to type. That is useful, and it is also where every problem starts. A thing that works unattended for hours is a thing making decisions you are not watching. Which brings us to the part the slogan skips.

If the loop is settled and small, the interesting question is not how to build a better loop. It is three questions about the system around it.

The first question: how do you know it worked?

The single most important piece of a reliable loop is the part that can tell it no.

An Anthropic engineer described the pattern: before the agent writes a single line, two agents negotiate what "done" means, and a third exists only to check the work against that definition. The loop does not run until there is a test it can fail. The failing test carries the weight, not the loop itself. You build the thing that says no first. The loop comes after.

The reason to build it first is a piece of arithmetic that belongs on a slide in every project like this. Melanie Warrick, who works on this at Temporal, ran the numbers: even if every step in a job succeeds 85% of the time, a ten-step job finishes correctly only about 20% of the time. The failures multiply. Miss one step in five, ten times running, and a clean pass becomes rare. The gap, she found, is not intelligence. The agents "diagnose a failure in perfect detail and still do nothing to recover from it." They describe the wall in detail and walk into it anyway.

It gets worse inside a single run. A model's attention decays as the loop goes on, and there is now a benchmark built to predict where it breaks. One study found a rule honored 73% of the time early had fallen to 33% by sixteen steps later. The agent did not rebel. It forgot, the way a tired person forgets, and kept going with the rule half-erased. The fix is unglamorous: re-state the hard constraints on every pass, and never assume a rule set once stays set.

None of that is loop engineering. It is checking-the-work engineering, and it is most of the actual job. A loop with no part that catches a bad result is not a loop. It is a fast way to be wrong at scale.

Even at 85% success per step, a ten-step job comes out clean only about one time in five.

The second question: what did it cost?

The economics of a loop break the way most people budget for software, and the people who learned this learned it expensively.

A chat costs you a sentence at a time. A loop runs for hours, calls the model hundreds of times, and has no natural stopping point unless you give it one. So the unit changed underneath you. It is no longer cost per question asked. It is cost per finished piece of work, and that number is wild.

One study measured the same task, same model, same prompt, same everything, and found it could cost eight dollars or two hundred forty. Thirty times apart, with nothing changed on the input, the spread tracing to how the loop was set up rather than which model ran it. The cost of letting a loop run is not a price you can quote in advance. It is a range, and the high end is far away.

This is not theoretical. Uber, by one widely-shared account, burned its entire annual AI coding budget in four months running agent loops, then capped its engineers at fifteen hundred dollars a person. The production version of the loop is not the demo that builds an app while you sleep. It has a spend cap, a meter, and someone who gets a phone alert when the meter spins.

Then the part that should give anyone budgeting for this pause. Checking the work can cost more than doing it. One team spent around forty thousand dollars just on the runs to verify their results, and noted that checking rigorously enough to fully trust them would push into the hundreds of thousands. If the whole promise of the loop rests on a reliable way to confirm it is done, and confirming "done" at scale is the most expensive line on the page, that is a cost nobody put in the pitch.

The answer the field is settling on is sensible. Run most of the loop on a cheap, fast model, and save the expensive top-tier model for the few hard steps that actually need it. Most steps do not need the smartest model in the building. You pay top-tier prices only where top-tier reasoning earns them.

That rule hides a deeper one, and it is the subject of my next essay, Nobody Saves Money on the Model: a cheap model is not a discount, it is a bet that you removed the ambiguity before the loop ever ran. The saving was never in the price per call.

The third question: what is it allowed to do?

The last piece is the one the "no human needed" crowd skips, and it is the one that ends careers.

A loop that runs for hours without supervision is, by definition, taking actions you are not watching. Late last year an autonomous coding agent, asked to clear out some temporary files, wiped a user's drive instead. It did what an unbounded loop does: it took an action it could not undo, in perfect confidence, with no gate in front of it. The failure was not that the agent was dumb. It was fast, capable, and unsupervised all at once, which is a different and worse problem.

The fix is structural, and it looks the same every time: checkpoints, hard limits on how long it can run and how much it can spend, and a human sign-off on anything the agent cannot take back. The companies actually running these in production are not running pure AI. They run a mix: the model reasons, ordinary tested code does the irreversible parts, and a person approves the steps that matter. One builder put it plainly. Most agentic loops are not autonomous. They are automated failure.

There is a newer worry above even that. An agent with standing permission to act is a kind of insider you have never had on the payroll. It holds credentials, runs unattended, and can do in a loop at three in the morning what a confused employee could only do at a desk by day. The risk was never which model you use. It is what permissions the loop carries, and whether you can pull them back in a hurry.

This is why the adoption numbers tell a sober story. By one survey, 79% of companies have adopted these agents and only 11% run them in production. Gartner projects 40% of agentic AI projects will be canceled by 2027, and the reasons are not "the loop didn't work." They are cost, unclear payoff, and the governance gap above. The loop is the easy part. Everything I just walked through is the distance between a demo and a deployment.

An agent with standing permission is an insider that never sleeps. The only question that matters is whether you can take the keys back.

Where this leaves you

Here is the whole thing in one breath. The loop is a dumb little cycle that keeps its progress on your hard drive instead of in the model's head. You could run one. You might well want to, because it turns making the work into describing and checking the work. And whether it pays off has almost nothing to do with the loop and almost everything to do with three answers most teams cannot give.

Do you know when it worked? Do you know what it cost? Do you know what it is allowed to touch?

That is not a coding question, which is the good news for anyone reading this who does not write code. Knowing what good looks like, catching a confident machine being wrong, deciding what it may and may not do without a human in the room: that is the judgment AI didn't make cheap. A loop does not supply it. A loop spends it.

A year from now your competitor will run the same six-line loop you do, against a model that costs about what yours does. Both will be things you order off a shelf. So the question is not which loop to build. It is whether, when your loops are running overnight, you can answer those three. Most teams cannot, and that, not the loop, is the work of this year.

The receipts behind this argument, the sources, the skeptics, the cost numbers, and the one stat I refused to print, are in the 30-day sweep linked up top. Run Data Run is free. If this was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

About Last 30 Days. Cross-platform research sweeps on topics worth paying attention to. Every post pulls Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, then synthesizes what people are actually saying, building, and betting on. Topics get picked when the signal is high and the story is contradictory, when a single headline would lie about the shape of what's happening. Each post follows the same arc: one specific finding that earns the click, why the topic deserves a sweep right now, the themed synthesis with inline citations, and the follow-up threads worth watching next.

Today's Run Data Run post, The Loop Is Simpler Than It Sounds, rests on a 30-day sweep across every platform where people argue about AI. This is the companion: the raw shape of what came back, including the parts that did not fit the headline. If the post is the argument, this is the receipts.

The single most useful thing the sweep found was not a fact. It was a contradiction. The loudest claim of the month, that "loop engineering" is the new moat, sits directly on top of an older, quieter body of evidence saying the exact opposite: that loops fail in ways the hype never mentions, and the people who learned that learned it expensively. Both are real. A single headline would have to pick one and lie about the other.

Why this topic deserves a sweep

In June, "the loop" went from a clever bash trick to the most over-discussed idea in the field. Anthropic shipped it as a product feature. A conference talk produced a slogan, "the winners will not have the smartest model, they will have the best loop," that got repeated everywhere. Boris Cherny, who built Claude Code, said he has not hand-written code in eight months and now spends his time writing loops, some days managing thousands of agents.

That is one half of the story, and if you only read X you would think it was the whole thing. The sweep is what surfaces the other half: a counter-current nearly as strong, mostly predating the viral wave, made of cost blowups, reliability math, and production failures. You cannot read both halves and conclude one thing. That is exactly when a sweep beats a headline.

The moat thesis, as it actually spread

The viral layer traces to one talk. The highest-engagement post of the set put the slogan plainly (@AnatoliKopadze, 8,624 likes, 1.7M views), describing Anthropic running "three agents: one to plan, one to build, one to judge, cycling until the app actually works." The clearest single statement of the mechanic came from a practitioner, Akshay Pachaar: "the loop itself is six lines, and nobody competes on it. every serious agent framework lands on the same tiny while-loop."

The technique has a name and an origin: the "Ralph" loop, coined by Geoffrey Huntley after Ralph Wiggum, because it looks too dumb to work. And it has a button now, Claude Code's native `/loop` and `/goal`/loop and /goal](https://code.claude.com/docs/en/goal) commands, which is usually the signal that something has gone from clever to standard.

The one stat I refused to print

Here is a transparency note worth making, because it is the kind of thing a sweep catches and a single source does not.

The most-quoted number in the whole topic, Anthropic's internal loop-adoption figure, does not survive the sweep. Three different posts attribute three different numbers to what appears to be the same talk: "over 30% of code" written through loops, then "70 to 80% of engineers" using them, then "90%." When one stat arrives at three sizes from one source, it is folklore, not data. So the post states the direction (Anthropic is clearly building this way, at scale, on the record) and explicitly tells you to treat the decimal point as noise. That call only gets made because the sweep put the three versions side by side.

The counter-current the slogan skips

This is the densest part of what came back, and the part the hype leaves out.

• The math. Melanie Warrick at Temporal ran the numbers: even at 85% reliability per step, a ten-step workflow finishes correctly only about 20% of the time. Failures multiply. She found the gap is resilience, not intelligence; agents "diagnose a failure in perfect detail and still do nothing to recover from it."

• The cost. An arXiv study found the same task, same model, same prompt, could cost eight dollars or two hundred forty, thirty times apart with no input change. A separate $22,000 sweep pinned a 33-fold spread to scaffold choices. And one widely-shared account had Uber burning its annual AI coding budget in four months before capping engineers at $1,500 a head.

• The verification trap. A benchmark of agentic systems spent roughly $40,000 on evaluation runs alone, and noted that doing the checks rigorously would push it into the hundreds of thousands. If the whole premise of a loop is a verifiable done-condition, and verifying "done" at scale is the single most expensive line, that is a price nobody put in the pitch.

• The adoption gap. By one survey, 79% of enterprises have adopted agents and only 11% run them in production. Gartner projects 40% of agentic AI projects will be canceled by 2027, on cost, value, and governance, not on the loop failing.

Gary Marcus and Arpit Bhayani carried the skeptic load on the social layer, both before the June wave crested. The reliability and cost objections were live before the moat framing peaked, which is itself a signal.

How the sweep is built (the method)

A quick word on method, since the point of this section is to show the work.

The sweep pulls each platform on its own terms: Hacker News and Reddit for builder substance in the comments, X for engagement-weighted reach, a curated corpus and a Substack named-voice pool for depth, and a separate open-ended discovery pass whose only instruction is "surprise me, do not confirm the thesis." That last pass is what surfaced the counter-current; left to its own framing, a research pass confirms what you already believe.

Two integrity habits matter. Every cited cost figure traces to a primary source, not a tweet about a source. And the fetch pass flags its own failures: of the URLs pulled this round, several came back as bot-challenge pages or 404s and were routed around rather than quoted. A claim that only survives in one blocked link is not a claim.

What I'm watching next

• Whether the Anthropic adoption number ever gets pinned to a primary transcript, or stays folklore.

• Where exactly the "coherence cliff" bites, by task length, token count, or iteration count. Everyone asserts it; nobody has located it.

• Whether the three-agent plan/build/judge pattern survives the move from greenfield demos to brownfield maintenance, where the stale-state failures are worse.

• The shift the researchers are naming: the agent stops being a tool that runs your code and becomes the software, with the human as "intent architect." If that holds, the unit of cost moves from per-token to per-finished-artifact, and most budgeting models break.

Sources and method

Full synthesis and the per-platform raw dumps live in my research vault; the post, The Loop Is Simpler Than It Sounds, is the argument this sweep grounds. Method: a 30-day cross-platform sweep (Reddit, X, YouTube, Hacker News, the web, a curated corpus, a named-voice pool) plus an open-ended discovery pass, two-pass synthesis, every quantitative claim traced to a primary source. Counts and quotes come from the actual sweep output, not memory.

Last 30 Days is a research series on Run Data Run, posted alongside the occasional Deep Dive when a topic earns the deeper look. No email on these, they live on the site for when you want the receipts behind an argument. If a sweep was useful, subscribe and you'll get the next one as it lands.

About Last 30 Days. Cross-platform research sweeps on topics worth paying attention to. Every post pulls Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, then synthesizes what people are actually saying, building, and betting on. Topics get picked when the signal is high and the story is contradictory, when a single headline would lie about the shape of what's happening. Each post follows the same arc: one specific finding that earns the click, why the topic deserves a sweep right now, the themed synthesis with inline citations, and the follow-up threads worth watching next.

The hook

A prototype of Biomni, Stanford's autonomous biomedical research agent, was already running in more than 10,000 laboratories before its paper appeared in Science on 9 July.

Hold that number next to the thing everyone actually celebrated this month, which was the paper.

That gap, between when these systems became real and when the record admitted it, turned out to be the shape of this entire sweep.

Why this topic deserves a sweep right now

FutureHouse's Robin reached *Nature* this week. It is a genuinely significant result: given the name of a disease, it proposed a therapeutic strategy for dry age-related macular degeneration, identified ripasudil (a glaucoma drug never proposed for dAMD), confirmed it in cells, and then designed and analyzed its own follow-up experiment. I wrote about it properly in this week's Sunday Deep Dive.

Reading it sent me back to survey the whole category, because something felt off about the timing. It was. Robin's preprint went up in May 2025. The paper printed in July 2026.

So I ran a thirty-day sweep across X, Hacker News, the curated wire, and the web. What came back was not a story about one paper. It was a story about a field that has already moved somewhere the published record has not caught up to, and about a body of skeptical research that landed in the same window and got almost no attention at all.

The sweep

Theme 1: they all landed at once

Four flagship autonomous-discovery systems cleared peer review inside a single four-month window.

• [Robin](https://www.nature.com/articles/s41586-026-10652-y)Robin** (FutureHouse) → Nature, July 2026. Multi-agent, lab-in-the-loop. Three specialised agents: Crow (concise literature search), Falcon (deep literature search), Finch (data analysis). Found ripasudil and KL001 for dry AMD, then surfaced ABCA1 as a possible novel target via its own RNA-seq follow-up. From the abstract: "All hypotheses, experimental directions, data analyses and data figures in the main text of this report were produced by Robin."

• [Google DeepMind's AI co-scientist](https://research.google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist/)Google DeepMind's AI co-scientist** → Nature, 19 May 2026. Gemini-based multi-agent system. Its headline validation is the same move Robin made: read the literature, propose an old drug for a new disease. Theirs was KIRA6 for acute myeloid leukemia, which inhibited AML cell viability at clinically relevant concentrations. Also went after liver fibrosis targets and antimicrobial resistance.

• [Sakana's AI Scientist](https://sakana.ai/ai-scientist-nature/)Sakana's AI Scientist** → Nature, 25 March 2026. The most radical of the four and the least grounded in wet biology: it runs the whole pipeline through to a finished manuscript, then peer-reviews itself. A paper it generated passed the first round of human peer review at a top machine-learning workshop. Its most interesting finding is a scaling law of AI science: generated-paper quality rises with the underlying model and with inference-time compute.

• [Biomni](https://news.stanford.edu/stories/2026/07/biomni-ai-powered-biomedical-co-scientist)Biomni** (Stanford) → Science, 9 July 2026. "Autonomous biomedical research with an artificial intelligence agent." Generalises across causal gene prioritisation, drug repurposing, rare-disease diagnosis, microbiome analysis and molecular cloning with no task-specific tuning. Already in 10,000+ labs.

The cross-system pattern was spotted in the wild and it is worth quoting, because it is the whole architectural story in one line: "Across Claude Science, NVIDIA BioNeMo, and FutureHouse Robin, the same pattern keeps appearing: specialized agent skills, orchestrated research workflows, domain-specific execution."

Nobody trained a discovery model. Everybody built a harness.

Theme 2: the lag is the actual story

Every one of those systems was old news by the time it was printed.

• Robin: preprint 19 May 2025 → Nature July 2026. 14 months.

• Sakana's AI Scientist: arXiv 12 August 2024 → Nature 25 March 2026. 19 months.

• Google's co-scientist: announced 19 February 2025 → Nature 19 May 2026. 15 months, to the day.

• Biomni: bioRxiv 30 May 2025 → Science 9 July 2026. 13 months.

Mean lag: roughly fifteen months.

The sharpest write-up of this belongs to synthetic biologist Jake Wintermute, on Biomni:

"Biomni was on arXiv 13 months ago. Biomni was on GitHub 11 months ago. Phylo, the company built on Biomni, raised $13.5M and launched 5 months ago. Or, I guess, you could read about it in Science Magazine today."

None of this is a criticism of the labs. They all posted preprints immediately, which is exactly right. The lag belongs to the journals.

Theme 3: where the field actually is

Not where Nature says it is. While Robin was in review, FutureHouse built its successor and then built a company around it.

• [Kosmos](https://arxiv.org/abs/2511.02824)Kosmos** (arXiv, November 2025): runs up to 12 hours across ~20 cycles. A single run reads ~1,500 papers and writes ~42,000 lines of code. Independent scientists judged 79.4% of the statements in its reports accurate. It has produced seven discoveries, three reproducing unpublished findings and four net-new. Collaborators estimated one run did about six months of their own work. Sam Rodriques' announcement did 3,683 likes, the highest-engagement item in the entire sweep.

• Edison Scientific: FutureHouse's for-profit spinout, commercialising Kosmos. And this month, its first partnership using Kosmos not to write papers but to launch new biotech companies.

• [Sakana Marlin](https://x.com/hardmaru/status/2066529282588094713)Sakana Marlin**: Sakana's first commercial product. An autonomous research agent that runs ~8 hours unattended and emits structured slides plus a multi-dozen-page report. Pitched as a virtual chief strategy officer, aimed at finance and consulting rather than the bench.

• [Claude Science](https://www.anthropic.com/news/claude-science-ai-workbench)Claude Science** (Anthropic, 30 June): the workbench end of the same idea. I wrote about it here.

Read the journals to learn what was true a year ago. Read the preprints and the launches to learn what is true now.

Theme 4: the counterweight nobody is reading

This is the highest-value material in the sweep and it appeared in almost no coverage.

[Correct Answer, Wrong Mechanism](https://arxiv.org/pdf/2606.23175)Correct Answer, Wrong Mechanism** (arXiv 2606.23175). Subtitle: When AI Scientists Defend General Claims Their Own Data Contradicts.

Researchers watched a coding agent attempt to rediscover a known particle-physics result across 28 episodes. It often got the right answer. The problem was how:

• CAWM occurred in 4 of 20 (20%) primary-model episodes and 3 of 8 (37.5%) episodes on other frontier models.

• The agent reached right-looking results through reasoning that collapses when conditions change.

• When pressed, it defended the wrong mechanism, arguing for physics inconsistent with the numbers in its own output.

• Verdict: these systems are dependable as tools but "unreliable scientific co-authors for open-ended claim-making."

• The demand: outcome-only evaluation is insufficient. Score task outcome, mechanism fidelity, and epistemic honesty as three separate things. Their lightweight checks flagged every CAWM case in the study.

[Related work](https://phys.org/news/2026-05-ai-scientists-reveal-fundamental-limits.html)Related work** (May 2026) catalogues the failure modes of autonomous research pipelines with unnerving specificity: implementation bugs, hallucinated results, shortcut reliance, methodology fabrication, citation hallucination, frame-lock, and bug-as-insight reframing, in which a system trips over a defect in its own code and writes it up as a discovery. The finding: increased quantity, decreased quality, of both papers and reviews.

And the view from the bench. Working scientist Emma Chory, on Biomni cheerfully producing a BSL2-plus lentivirus protocol with, in her words, "sufficient detail to execute on a robot." Her review ran to four words: "Cool cool cool cool." An agent that will write you a biosafety-level-2-plus procedure on request is a governance question, not a capability win.

Theme 5: the taxonomy worth stealing

Computational biologist Mile Sikic drew the distinction that most of the coverage blurs. There are two streams:

1. Virtual bioinformaticians that combine existing knowledge with existing tools. (Claude Science, Biomni, most of the field.)

2. Systems that discover new biology in close collaboration with wet-lab scientists. (Robin, Google's co-scientist.)

Conflating them is how people end up disappointed. They are solving different problems.

Also in the window, at lower weight: [EurekAgent](http://arxiv.org/abs/2606.13662)EurekAgent** ("Agent Environment Engineering is All You Need for Autonomous Scientific Discovery"); Yoshua Bengio's take on an AI Scientist; the Samsung SAIT AI Scientist competition winners; OpenScience, an open coding agent "that went to grad school," ~350 GitHub stars in its first week; and a genuinely useful sign of a crowded category, a piece titled "Which 'AI scientist' suits your lab? A guide for the perplexed."

What I'm watching next

Edison Scientific's biotech-founding partnership. Using an AI scientist to found companies is a categorically different claim from using one to write a paper, and unlike a paper it will be tested in public, on a clock, with other people's money.

Whether anyone adopts the CAWM evaluation protocol. The paper's demand is concrete and cheap: score mechanism fidelity separately from outcome. If the next generation of agent papers still reports only "did it get the right answer," the field has decided not to look.

Ripasudil. It cleared cells, not patients. A disease model comes next, and then, eventually, a randomized controlled trial. Most compounds that look this good at this stage never finish.

Sources

Primary papers: Robin / *Nature* · Robin preprint · Kosmos · Sakana AI Scientist · Correct Answer, Wrong Mechanism · EurekAgent · Biomni / Stanford · Google AI co-scientist · Claude Science

Voices: @SGRodriques · @FutureHouseSF · @SynBio1 · @chorye · @msikic · @hardmaru · @aipoch_ai

Methodology: thirty-day sweep across X, Hacker News, the curated AI wire, and the web, run 12 July 2026. The X leg of my usual tooling was down (an expired API credential), so X was swept through a working session-based client instead; Reddit returned no topical signal and was discarded rather than padded. Engagement figures are as recorded at collection time. The full deep dive on the Robin paper itself is [here](https://rundatarun.io/p/the-year-nature-caught-up).here.*

The Headline

A multi-agent AI system named Robin was given the name of a disease and told to find a treatment. It read the literature, proposed a therapeutic strategy, named a drug, watched humans test that drug at the bench, analyzed the results itself, and then designed its own follow-up experiment. The disease was dry age-related macular degeneration, the major cause of blindness in the developed world. The drug was ripasudil, a rho-kinase inhibitor that has been sitting in pharmacies for years, approved for glaucoma, and never proposed for dAMD by anyone, as far as the authors could tell or I could find.

It worked in cells. Then Robin asked why, specified an RNA-sequencing experiment to find out, analyzed that too, and surfaced a gene called ABCA1 as a possible new target nobody had been looking at.

This is the work of FutureHouse, and it was published in Nature this week. There is one sentence in the paper that deserves a second read:

"All hypotheses, experimental directions, data analyses and data figures in the main text of this report were produced by Robin."

Not assisted by. Produced by.

The preprint went up in May 2025. Nature published it in July 2026. Fourteen months, and almost none of the coverage mentions it.

Once you notice that gap, you start seeing it everywhere, and the shape of this entire field changes.

The Paper

Three birds and an orchestrator

Robin is not a model. Nobody at FutureHouse trained a "discovery model," and if you go shopping for one after reading the coverage, you will not find it.

What they built is a system of specialized language agents sitting on top of ordinary commercial models:

• Crow runs fast, concise literature searches.

• Falcon runs deep ones.

• Finch analyzes raw experimental data.

Robin orchestrates the three of them and carries the hypothesis across rounds, so that what Finch learns on Tuesday reshapes what Falcon goes looking for on Wednesday.

The intelligence came off a price sheet. The discovery came from the scaffolding around it.

I wrote two weeks ago that Anthropic shipped a workbench, not a miracle, and that the harness is the moat. Robin is that argument with a Nature paper attached to it, which is a considerably stronger position than my say-so.

The word "semi" is doing honest work

The paper describes Robin as semi-autonomous, and the precision is a credit to the authors rather than a hedge.

Humans still run the wet lab. People cultured the retinal cells, ran the assays, and pipetted the compounds. What Robin did was everything on either side of the bench: read the field, form the hypothesis, specify the experiment, interpret what came back, and revise.

The term of art is lab-in-the-loop, and it is worth flipping around, because the phrase misleads people. The AI is not in the lab. The lab is in the AI's loop.

Previous systems could do one arc of that circuit. They could read and propose. Or they could take your data and analyze it. Robin ran the full turn, then fed its own results into the next turn, and did it again.

The paper calls Robin "one of the first" systems to do this. The preprint called it "the first." Somewhere in fourteen months of review a superlative got sanded off, which tells you something useful about the claim and something more interesting about the process.

Why the drug was findable at all

FutureHouse call their method combinatorial synthesis, and their honesty about what it means is why I trust the rest of the paper.

Robin did not invent new biology. Every piece was already in print. That rho-kinase inhibition boosts phagocytosis in retinal pigment epithelium cells: known, and they cite the work. That this phagocytic housekeeping declines in AMD patients: also known. The two facts lived in different literatures, read by different people, and nobody had put them next to each other and said the word ripasudil.

The authors have a devastating example of how long that gap can persist. Dabrafenib is a cancer drug whose molecular action was characterized by 2010. Ten years later, a brute-force screen discovered it protects against hearing loss. That protective effect follows directly from the mechanism everyone already knew. The answer sat in print for a decade because the people who knew about BRAF inhibition and the people who cared about hearing loss were not the same people.

Robin is not doing science we could not do. It is doing science we did not get around to.

Scope the claim that way and it stays large. As the authors point out, novel FDA approvals have been flat at roughly fifty a year for a decade. If a machine can reliably close the distance between what is known and what is connected, that is worth a great deal of money and a great deal of eyesight.

The two things I would copy tomorrow

They checked their own judge. Buried in the supplement is a comparison of their LLM evaluator against human experts. I read a lot of agent papers that grade themselves with a model and never once ask whether the grader is any good. This one asked.

Their guardrails are the most thought-through of anything I read this month. Robin preferentially proposes compounds with established safety profiles. Every output is treated as a hypothesis entering standard preclinical review, never as a finding. And the discussion says the quiet part plainly: ripasudil "would of course require validation in a suitable disease model and ultimately in a randomized, placebo-controlled trial."

In vitro is a beginning. The authors know it, and they wrote it down.

The Ecosystem

Robin made me want to go back and re-survey this whole category, so I did. Thirty days of papers, launches, funding, and argument. What I found reframed the paper I had just finished reading.

Everyone landed at once

Four flagship systems cleared peer review inside about four months.

Google DeepMind's AI co-scientist reached Nature in May. It is a Gemini-based multi-agent system, and its headline validation is the same move Robin made: read the literature, propose an old drug for a new disease. Theirs was KIRA6 for acute myeloid leukemia, which inhibited the viability of AML cells at clinically relevant concentrations. They also went after liver fibrosis targets and antimicrobial resistance.

Sakana's AI Scientist reached Nature on 25 March. It is more radical in ambition and less grounded in wet biology: it runs the entire pipeline through to a finished manuscript and then peer-reviews itself. A paper it generated passed the first round of human peer review at a top machine-learning workshop. Its most interesting result is a scaling law of AI science: the quality of the papers rises with the quality of the underlying model and with the compute you spend at inference.

Biomni, out of Stanford, reached Science on 9 July, under the title "Autonomous biomedical research with an artificial intelligence agent." Hold on to one detail from it, because it is the most telling number in this entire piece: a prototype of Biomni was already running in more than 10,000 labs before the paper printed.

The lag is the actual story

Every one of those systems was old news by the time it was printed.

Robin: preprint May 2025, Nature July 2026. Fourteen months. Sakana's AI Scientist went up on arXiv in August 2024 and reached Nature in March 2026. Nineteen months. Google announced its co-scientist in February 2025 and printed it in May 2026. Fifteen months, to the day. And Biomni got the sharpest write-up it will ever receive, from a synthetic biologist on X:

"Biomni was on arXiv 13 months ago. Biomni was on GitHub 11 months ago. Phylo, the company built on Biomni, raised $13.5M and launched 5 months ago. Or, I guess, you could read about it in Science Magazine today."

None of this is a criticism of the labs. FutureHouse posted their preprint the moment they had it, which is exactly right. The criticism, if there is one, belongs to the clock the journals run on.

So where is the field actually standing? Not where Nature says it is.

FutureHouse has already built Robin's successor. Kosmos went up on arXiv last November. It runs for twelve hours at a stretch, and a single run reads around 1,500 papers and writes roughly 42,000 lines of code. Independent scientists checked its reports and found 79.4% of its statements accurate. It has produced seven discoveries, four of them net new. Collaborators estimated that one run did about six months of their own work.

They then spun out a for-profit company, Edison Scientific, to sell it. And this month they announced a partnership that uses Kosmos not to write papers but to found biotech companies.

Sakana, meanwhile, is selling Marlin, an autonomous research agent that runs for about eight hours unattended and is pitched as a virtual chief strategy officer.

Read Nature to learn what was true a year ago. Read the preprints and the launches to learn what is true now.

The counterweight nobody is reading

While these systems were collecting their journal stamps, a second body of work was landing that tells you precisely where they break. None of the launch threads mention it.

The sharpest of it is a paper whose title belongs on a poster in every lab now buying one of these things: **Correct Answer, Wrong Mechanism**. The subtitle is better. When AI Scientists Defend General Claims Their Own Data Contradicts.

The researchers watched a coding agent try to rediscover a known result in particle physics, twenty-eight times over. Often it got the right answer. The problem was how. In 20% of episodes with the primary model, and 37.5% across other frontier models, the agent reached a right-looking result through reasoning that collapses the moment conditions change. Worse, when pressed, it defended the wrong mechanism, arguing for physics that contradicted the numbers in its own output.

Their verdict is careful, and damning. These systems are dependable as tools and, for now, "unreliable scientific co-authors for open-ended claim-making." Asking whether the agent got the answer right tells you almost nothing. You have to score the outcome, the fidelity of the mechanism, and the honesty of the account as three separate things.

Related work in the same window catalogues the failure modes with unnerving specificity: hallucinated results, methodology fabrication, citation invention, frame-lock, and my favorite piece of vocabulary this year, bug-as-insight reframing, in which a system trips over a defect in its own code and writes it up as a discovery.

Set that next to Robin and its design choices stop looking conservative and start looking wise. When Robin says ripasudil enhances phagocytosis, a cell culture has already voted, and a cell culture does not care what the agent believes about it.

A wet lab is a brutal, incorruptible critic. You cannot argue a cell culture into agreeing with you.

But be precise about how far that protection reaches, because it is not the whole paper. When Robin says the mechanism runs through ABCA1, that is Finch reading an RNA-sequencing experiment, and that is exactly the surface the CAWM authors are worried about. The bench protects the finding. It does not protect the explanation. Which is roughly the division of trust I would apply to any of these systems, and to be fair to FutureHouse, it is the division they applied themselves: the drug candidate is offered for preclinical review, the mechanism is offered as a possibility.

It is also why the wet-biology systems stop at semi-autonomous, and why that ceiling is physical rather than technical. Laboratory instruments were not built to take orders from software. Somebody has to run the assay. Sakana's system is the exception that proves the rule, and it proves it uncomfortably: with no bench anywhere in the loop, it is the one system in this survey that will write its own paper and review it.

What I Run

I should declare an interest, briefly, because I have spent the last year on the other side of this.

I run an autonomous research engine called ARIA. It is a persistent system rather than a one-shot agent: a pool of research ideas competing against each other on a single scoring function, experiments that run locally first and have to earn their way onto expensive hardware, a critic drawn from a different model family that posts a verdict on every result, and failures that classify themselves and re-enter the pool as recovery work. Seven instances, four domains, about 19,400 auditable commits, and 97.8% autonomous resumption after failure. Its successor now runs around the clock on a borrowed eight-GPU node, pointed at retinal disease. There is a case study and a short film if you want the shape of it.

I raise it for one reason. Building the thing taught me the same lesson the CAWM paper is now pressing on the whole field, and I learned it the expensive way.

Generating hypotheses was never the hard part. Building a critic honest enough to kill them was.

An idea generator is cheap, and it will happily run forever, and every dashboard will stay green while it does. What is difficult, and what almost nobody budgets for, is the machinery that tells the system it is wrong and makes the verdict stick.

Robin never had to build that machinery, because the bench already is it. That is a structural advantage of working in wet biology, and it is worth naming plainly for anyone whose research loop closes entirely in software, where nothing pushes back for free.

Why It Matters

Do not go shopping for a discovery model. There isn't one. Not one system in this survey used a model you cannot buy. Robin, the co-scientist, Kosmos, Biomni: every one of them is a harness built around a commodity brain, which means the thing your organization would be building is the harness, and that is engineering, not procurement.

The compartmentalization gap is inside your own company. Robin's whole trick is finding the connection between two literatures that no single person reads. You have that problem internally, at smaller scale, right now. The result one team generated and another team needed and never saw is the cheapest discovery you will ever fund.

Read the preprints. On this topic the peer-reviewed record is running roughly a year behind the work. Use the journals to confirm, not to track.

Take the reliability research as seriously as the launch threads. A system that finds the right answer for the wrong reason will sail through your demo and drown in your clinic. Ask any vendor how they score mechanism fidelity, not just outcomes. Watch them field the question.

And watch ripasudil. It cleared cells, not patients. The road from here runs through a disease model and then a randomized controlled trial, and most compounds that look this good at this stage do not finish it.

Robin did not replace the scientist. It replaced the lag, the decade dabrafenib spent sitting in the literature while nobody put two known facts side by side.

There is a certain irony in that finding taking fourteen months to reach print. It does not make it less of a landmark. Congratulations to the FutureHouse team. The field is better for this being in the record, even if the record took its time.

I ran a full thirty-day sweep of this category to write this piece, and most of it did not fit. The complete survey, with every system, number, and citation, is [in a companion post here](https://rundatarun.io/p/last-30-days-the-ai-scientists).in a companion post here.*

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

Some version of this question arrives most weeks. How do you keep up? You have a day job. The field moves every single day. What are you reading?

For a long time I answered it badly. I would say something about reading a lot, which is true and completely useless to the person asking. The real answer took me a while to say out loud, because it sounds like a dodge.

I stopped trying to keep up. I built something that keeps up for me.

Bookmarks are where links go to die

Nobody is reading this field. Not all of it, not close. Papers land overnight. A lab ships a model on a Tuesday and by Thursday other teams have published what breaks about it. The people who look like they are on top of it are not reading faster than you. They have narrowed, or they have automated, or they are bluffing.

Manual curation was my first answer, and it failed the way manual curation always fails. Open tabs became a graveyard. Saved links went unread and, worse, went stale.

A link I saved in March, about a model superseded in April, is not information anymore. It is clutter with a timestamp.

Newsletters helped until I was subscribed to a stack of them and reading none. The problem was never that I lacked sources. I had too many, arriving on their own schedules, with no shared memory between them.

One pipe

So I built a wire.

Several channels I already had running now point at one place. A scan that reads the day's news, releases, and research. The things I bookmark as I go. Findings from a small set of AI assistants I run, each pointed at its own beat. My own written breakdowns of the papers and tools and companies I sit down and work through by hand. Things I email myself at midnight and would otherwise never see again.

All of it lands in one store, scored and deduplicated. No single source has to be the right source, because none of them carries the day alone. And nothing gets read twice, because the store already knows it saw that story on three feeds this morning.

Two things come out the other end.

The first is private, and it is the part I use every day without thinking about it. The store is a knowledge base that refreshes itself, and my own AI tools read from it directly. When I sit down to write, or research something, or work a problem, the context they pull from is current. Not their training data. Not a stale snapshot I remembered to update. This morning's.

The second is public. A curated slice of the store becomes a website: a daily dispatch of the items that scored highest, written up rather than merely linked, so you know why something mattered before you decide to click it.

I like the arithmetic of it. The wire watches fifty-two sources. Over the last month it read and scored two thousand six hundred and seventy-nine items, and three hundred and ninety-nine of them cleared the bar. Yesterday's dispatch came out of a hundred and twenty-one signals. The site puts it more plainly than I would: it reads about seven things so you read one.

Fifty-two sources in. One page out.

What lands is a page I can get through with coffee.

The wire is the front door

This is not a side project I bolted on. It sits at the front of everything else I make.

What scores highest is usually what I end up writing about, because the things worth six hundred words announce themselves by refusing to go away. What I write becomes what I post. What I post brings back arguments and corrections from people who know more than I do about some corner of it, and those go back into the store as research. The loop closes. One pipe, several outlets, and the website is only the visible tip of it.

I did not design it that way at the start.

I built the scan because I was drowning. Then I noticed the thing that solved the drowning was also the thing deciding what I had to say.

Go look at it

The daily dispatch is free, it is public, and it lands every morning: wire.rundatarun.io/briefs. The day's signals, grouped by what they are, with a source link on every item so you can go argue with the original instead of taking my read on it.

Underneath it, the firehose shows you the machinery. The whole intake, the filter that thins it, and the last seven days in the raw.

What it does not show you is the store underneath. The complete searchable archive, the picks I would put in front of you myself with the reasoning attached, the decode and dossier library, the week ahead before it is news. That layer is not free, and it is not open yet. There is a waitlist at the bottom of the firehose page, and if the arithmetic up there sounded like your problem too, put your email in it.

Here is the whole thing, start to finish, in about forty-five seconds.

Jack Hanlon, who leads GenAI Media at Meta, posted this on LinkedIn:

"The first time you see an engineer build something in 45 minutes that would have taken a week a year ago, but then see it not ship for another 6 weeks, you will be radicalized."

He's describing Meta, a consumer tech company. The review stack he lists is the load every build carries before it ships: Strategy, Product, Design, Engineering, Privacy, Legal, Accessibility, Comms. It's also not close to what biopharma and patient-facing R&D organizations ask of an AI build.

In biopharma and adjacent patient-facing industries, add data privacy impact assessment, security review, AI governance review, IT architecture review, change-management approval, procurement, legal redline, regulatory sign-off. If patient data is anywhere in the lifecycle, add clinical safety review. Each gate is serial. Each gate is three weeks minimum. Six in sequence is half a year before anything ships. You've done paperwork.

The numbers back this up across every regulated sector, not just pharma. Enterprise SaaS procurement runs a median 170 days; complex solutions hit eleven-and-a-half months. SOC 2 Type II audits take six to twelve months end-to-end. Federal contractors face 12-18 month FedRAMP authorization cycles before a system can sit on a government network. Banks run AI governance committees on top of model-risk committees on top of vendor-risk committees, each stack inherited from a different decade of incident response. MIT NANDA's "GenAI Divide" report found 95% of enterprise AI pilots deliver zero P&L impact, across thirty to forty billion dollars of spend. That's not a build problem. The build works. The build sits in a queue.

This accelerated review pathway can exist. It can work. It will take herculean levels of structural redesign and a lot of organizational political capital to land. And the pattern isn't a pharma problem. It's the shape of every regulated industry trying to ship AI in 2026. Pharma's version is the sharpest because patients are downstream of the delay, but the argument is universal.

Every gate exists for a reason. Most got written after a real incident. Someone's data got exposed. A launch communication went sideways. A model made a biased call in production. The answer isn't to kill them. The answer is that every one of them was designed for a world where building the software was the slow, expensive, irreversible part. Whether you wrote the code yourself or bought it from a vendor who wrote it, software was the thing the calendar bent around. That world is gone. AI didn't just speed up typing. It turned the whole build, your own code and the vendor's both, into the fast part. The review cycle was calibrated for the old critical path. The critical path moved. The calendar didn't.

Hanlon's three confrontations are right. Process architecture is the speed limit. Too many orgs treat every process as sacred. AI should be eating process, not just coding. The patient-facing version is sharpest, because the delay shows up downstream as patient time.

This Has Happened Before

DevOps didn't land in 2015 as a philosophy. It landed when development got fast enough that deployment became the jam. "Development teams could complete features in days, but getting those features deployed took weeks." That's the DevOps.com retrospective on the pre-CI/CD era. Swap "deployed" for "procurement-approved" and you have a 2026 AI program, sentence for sentence.

Then security became the jam. DevSecOps. Then infrastructure and developer experience became the jam. Platform Engineering. Internal Developer Platforms. Each compression moved the bottleneck one layer further from the keyboard. Each shift spawned a category, a toolchain, a set of jobs, a conference circuit.

Bottleneck migration is a law. Compress one layer and the adjacent layer becomes the bottleneck by definition. Whatever used to be the tall pole is now almost all of the remaining pole.

AI compressed the build layer, writing software and buying it both. Everything adjacent to the build is now the bottleneck. The reviewers, the committees, the approvals, the procurement track, the vendor security questionnaire, the model evals nobody scoped time for. All of it. DevOps took twelve years to run the full cycle. Nobody has twelve years.

Shadow AI Is the Tell

Gartner's 2025 numbers are the signal. 98% of organizations report unsanctioned AI use. 69% report evidence of prohibited public GenAI use inside their walls. 49% expect a shadow AI incident within twelve months.

The default read calls shadow AI a governance failure. A discipline problem. Employees not following the rules. That read is backwards. Shadow AI is the system telling you the official process layer already failed an internal cost-benefit test. Employees measured the queue, measured their quarterly objectives, and decided the queue costs more than the risk of getting caught.

That's the market voting. When 98% of your workforce is routing around your approval process, you don't have a discipline problem. You have an economics problem. The process priced itself out.

A fair counter to all of this: maybe the 95% pilot failure isn't a process problem. Maybe it's a pilot-selection problem. Most enterprise AI pilots fail because they were the wrong pilot to run, picked for board optics or executive curiosity rather than real workflow pain. Speeding up bad pilots faster doesn't help anyone.

That argument is partially right. The data is harder than that. The same MIT report that flagged the 95% number also found the highest ROI was in back-office automation, exactly the work that sits behind procurement, vendor security, and compliance queues. The pilots that fail include some bad picks. They also include good picks that died in queue. You can have a pilot-selection problem and a process problem at the same time, and most enterprises do.

Process Protects Patients. Except When It Doesn't.

Biopharma's review gauntlet was written in blood. Good Clinical Practice, Good Laboratory Practice, Good Manufacturing Practice. Patients got hurt. Rules got written. Every review, every sign-off, every three-week queue descends from that inheritance.

The reflex response to "AI is slow to ship" is "good, that's how we keep patients safe." It sounds unanswerable. It isn't.

Process was built to protect patients. When process is the thing blocking patient-helping AI from shipping, it stops protecting patients. It protects itself.

Not shipping AI is not a neutral, safe choice. It has a cost. The cost lands on patients too.

Every week of review on a regulatory drafting agent is a week a safety narrative gets written the slow way. Novo Nordisk's NovoScribe rollout cut clinical study report drafting from fifteen weeks with over fifty medical writers to ten minutes with three. The work moved. The submission queue didn't. Every week of governance queue on a tool like that is real human time that wasn't spent on the submission, real submission time that wasn't spent reaching patients.

Every month a vendor security review stalls on a literature retrieval agent is a month oncology scientists read abstracts by hand. The responder-subgroup signal sits in a PDF nobody opened.

Every quarter a leadership group debates a two-speed governance design is a quarter trial protocols get mapped by hand, deviations get triaged out of a Teams channel, the first patient at the first site doses later than they would have.

Meanwhile, the deployments ship. Cleveland Clinic, NHS England, the VA, and 75% of US hospitals are running AI in clinical workflows today. The teams that wired governance early are in production. The teams still debating are watching.

Most governance agendas debate the wrong question. Not "is this AI safe enough to ship," but "is the delay we're about to impose smaller than the harm we're trying to prevent." Both sides of that ledger have a body count. Only one gets counted.

AI Eats Its Own Governance Layer

The governance layer isn't a coding problem, and that's the part most "AI accelerator" pitches miss. It's legal redline, procurement of the services the agents will consume, vendor security, the privacy review, the model evals, the benchmark sign-off. None of that is typing. All of it is the build now, and all of it queues.

Organizations have answered by standing up governance as elaborate as the thing it governs. Sanofi has publicly documented a multi-stage setup: a Responsible AI Working Committee, an Interim Responsible AI Governance body, an AI agent named Plai sitting in on every drug-progression decision. Other large pharmas, banks, and federal contractors run variations of the same shape; the public documentation lags Sanofi's. Ethan Mollick names why it decides everything. "The moderating factor is no longer individual ability or even AI capability. It is organizational structure, policy, and the way leaders choose to approach AI." The constraint isn't the model. It's the layer wrapped around the model.

Which is exactly the layer AI can eat. Hanlon's third point is the one most readers stop quoting before the punchline:

"Have the AI traverse your codebase and collect your evidence for the privacy review and have another AI grade the privacy review. They go back and forth until (a) they need human intervention because they are stuck or (b) they need humans to look at the final results and sign off on the right path forward."

That's the whole answer, and it generalizes past the codebase. Every gate in the gauntlet has the same shape: forty hours of evidence assembly, then a fifteen-minute judgment call. The privacy review is fourteen pages of data flow that already lives in the repo, the retention policies, the IAM config. The vendor questionnaire is two hundred SOC 2 controls a lead engineer maps by hand against an attestation report nobody re-read. The procurement intake is free-text someone retypes against master agreement terms a sourcing lead has memorized. The model eval is a benchmark suite someone runs once and pastes into a slide. The regulatory submission is fifteen weeks of medical writers stitching prior submissions into a new narrative. The judgment is fifteen minutes. The forty hours is the queue.

The human doesn't leave the privacy review. The human finally opens a pre-assembled package instead of staring at a blank template with half the source material missing. The reviewer still reviews. The reviewer is finally reviewing the only part of the work that ever needed a reviewer.

AI has eaten the keyboard. It hasn't eaten the queue.

That's the argument in one sentence. The keyboard compressed two years ago. The queue hasn't. Until it does, every "AI accelerator" announcement is a faster typewriter feeding the same backlog.

The Reviewer Is the Project

There's a catch built into all of this. The people whose work the AI must eat are the same people who must approve the AI eating their work. Legal reviews the legal agent. Procurement approves the procurement agent. InfoSec signs off on the InfoSec agent.

The reviewer is the subject of the review. The incentives don't line up. Pretending they do is how an org burns eighteen months and ships nothing.

The first week of agent output will be embarrassingly wrong, and that isn't the agent's fault. The process has never been written down in a way a machine can act on. The DPIA "template" is really a set of in-head judgments held by a senior person who onboarded in 2014 and knows which questions matter. Turning that into prompts, policies, and machine-readable rubrics is the institutional work, and there's no shortcut around it. That IS the project. The catch is that the people best positioned to codify their judgment are the ones whose standing rests on it staying in their heads. The org design that works makes the codified version the asset and leaves the committee to ratify it. Most enterprises don't run that design yet.

The ones that do compress the queue start in the same place, and it's never a model. They map the gauntlet, then fund an agent for a single gate, picked where the work is repetitive and the inputs already live somewhere a machine can read. Privacy review first, its answers mostly sitting in the repo and the IAM config. Vendor security next, the same shape with messier inputs. Regulatory drafting last, after the pattern has proven itself on something lower-stakes than an FDA filing. It holds at one gate, run for two months and watched for what it gets wrong. It breaks at three at once.

Underneath all of it sits the substrate. A governance layer that can't hand an agent a rubric to run or a template to fill isn't policy. It's tradition. The version living in senior reviewers' heads is neither auditable nor transferable, and it's the version adding weeks to every queue. Codifying it is the platform bet of 2026, ahead of the next model, because it's what lets every future build ship without the six-month tax.

AI compressed the build layer. That was the easy part. Everyone got the faster typewriter. The next decade of enterprise AI won't be won on better models. It turns on whether the layer built to protect patients from bad software, back when bad software was the threat, can be made to compress itself before the delay starts landing on the other side of the ledger.

That layer will not shrink on its own. It has to be handed its own rubric and told to grade the work it used to guard by hand. No team that owns a queue volunteers to automate it. Someone with the authority to override that instinct has to decide the delay costs more than the control.

In a patient-facing industry, the queue is not paperwork. The queue is the patient. Every week it stays uncompressed is a week charged to someone who never got a seat in the room.

There is a shelf now. If you are a senior leader trying to make sense of AI, you have probably bought most of it. The strategy book. The mindset book. The one with the four-quadrant framework. The one from the consultancy with eleven co-authors. They are not bad books. Several are good. I learned things from them.

But I noticed something after the fourth or fifth one. I would finish a chapter, nod, underline a sentence, and then sit there with my hands in exactly the same place they were before I started reading. The book had changed what I thought about AI. It had changed nothing about what I did with it on Monday.

That is the gap I wrote into.

The book changed what I thought about AI. It changed nothing about what I did with it on Monday.

The shelf has one shape

Read enough of these books and the shape repeats. They tell you what to think about AI. How to frame it for your boss/peers/board. Which mental model to adopt. Where it sits in your strategy. They are written from the altitude a leader is used to operating at, which is to say: above the work, directing it.

That altitude is the problem, not the solution. Someone reviewing the current best-seller in this category said it reads like a 2024 book explaining how the Internet will help your business. The line is unkind and basically right. Not because the author is wrong, but because the genre has a ceiling. You cannot write the operator's manual from the strategy altitude. The two are different books, and almost everyone is writing the first one.

I wanted to read the second one. Nobody had written it for the reader I had in mind, so I wrote it.

You cannot write the operator's manual from the strategy altitude.

What "what your hands do" actually means

"Leaders should build" is already a slogan, and slogans rot.

I don't mean you should learn to code. I mean something narrower. There is a class of tools now, Claude Code chief among them, that lets you direct a multi-agent system to produce real work inside your own domain. You describe intent, set boundaries, review what comes back, redirect, and ship. The interface happens to be a terminal; the skill is not programming. It is the same skill that got you to senior: defining the why, designing the handoffs, judging the output, catching the thing that looks right and is wrong.

You already do all of that. You do it with people. The book is about pointing those same instincts at a harness instead of a team, and what changes when you do.

The reason this matters is not productivity, though the productivity is real. The reason is calibration. Andrej Karpathy named it this spring in a line that landed with twenty thousand likes and a thousand replies: there are two groups talking past each other about AI, and the line between them is not skeptic versus believer. It is the people who have built something with these systems and the people who have read about them. Every confident claim you have heard about what AI can or cannot do was made from one side of that line. If you are reasoning about AI from the reading side, you are reasoning from the wrong data, and no amount of strategy reading moves you across. You move across by getting your hands on the controls one time and feeling where the real edge is.

Why I get to say this

What separates this book from the shelf is that I did not write it from the strategy altitude. I built and led an AI Center of Excellence at a Fortune 500 pharmaceutical company, and I now lead applied data and AI in R&D at an AI-driven biotech. I also run a deep personal practice on nights and weekends, eight autonomous agents and a homelab full of skills, because the day job and the obsession turned out to be the same skill. This book was itself produced through the harness it describes, end to end. About a month from kickoff to typeset manuscript, against the year the same book would have taken by hand, with thousands of hours of research, red-teaming, and editing behind it, most of it run in parallel by the system rather than typed by me. You can see the skills that drafted it.

I am not telling you to do something I read about. I am telling you what it was actually like, including the parts that are awkward and the parts that broke.

I am not telling you to do something I read about. I am telling you what it was actually like.

The part I won't oversell

At Sequoia's AI Ascent this spring, Karpathy declared "vibe coding" obsolete and named its successor "agentic engineering," a discipline with real rigor and real responsibility. He is right. He also drew a careful line: prototyping with AI raises the floor for everyone, but operating serious systems demands discipline you cannot wave away.

My book asks you to step over a line slightly further out than the one Karpathy draws for the general public. I am asking a senior, non-technical executive to personally operate, not just prototype. I think that is where the role is heading, and I think the leaders who learn it first will shape what their organizations become. But I am not going to pretend it is the consensus position. It is an argument. The book makes it, and gives you the controls and the guardrails to test it for yourself rather than take my word for it.

Who will not find this useful

If you want a framework to put on a slide for your board, the shelf has better options than mine, and I mean that. If you want reassurance that your current AI strategy is sound, this is the wrong book; it will probably make you uncomfortable. If you are looking for predictions about AGI timelines, I route around that debate on purpose.

This book is useful if you have read the think pieces, sponsored the center of excellence, watched the pilots, and still feel the quiet gap between knowing AI matters and knowing what you personally do about it. It is useful if you are willing to spend a few hours with your hands on something unfamiliar, on your own files, to find out where the edge actually is. It is a ladder. Three months to ship something real, six on the outside. The last page is a welcome from the people who were already on the other side when you started.

If that is you, the book is for you. It's out now: builder-leader.com, or buy it on Amazon.

Builder-Leader: The AI Exoskeleton That Crosses the Gap.

I've been telling you a book was coming for months. I named the idea in "Two Gaps, Not One" back in April and dropped not-so-subtle hints on LinkedIn. As of today, it's not a mention anymore. It has a name, a cover, and you can buy it right now: *Builder-Leader: The AI Exoskeleton That Crosses the Gap*. Paperback is live this minute. Kindle lands tomorrow, and you can get it early. Everything else, the excerpt, the cover, what's actually inside, lives at builder-leader.com.

What looks like a fight is a gap

The AI conversation reads like a war between camps.

Skeptics on one side: it's overhyped autocomplete, the bubble's coming, calm down. True believers on the other: everything changes next year, the jobs are going, brace. A large, tired middle trying to figure out which loud group to trust.

It is not a war between camps. It's a gap between two populations, and most of the noise is an artifact of which side of it you're standing on.

Andrej Karpathy named this on April 9 of this year, in a short post that got the whole feed nodding. There are two groups talking past each other about AI, he said. Not skeptics versus believers. People who have actually built something with these systems on one side, people who have read about them or tried the free version once on the other. The first group has watched the staggering version of the technology work for them. The second group is reasoning from screenshots and vibes.

The reply thread found a third group hiding inside the first. One reply, from @doodlestein, named the people "magnifying the power of these frontier models and agent harnesses with custom tooling and skills and workflows." Builders who don't just use the systems but wrap machinery around them. That's the group whose read on what AI can do is calibrated correctly, because they're the only ones holding the data.

Once you see this, the boardroom argument stops being a debate about technology. It becomes a map of who in the room has ever operated the thing. The person who hasn't isn't lazy or incurious. They're just, structurally, the worst-placed person to make the call, because they're working from the wrong inputs.

Why the gap is asymmetric

Here's the part that turns this from an observation into a problem: it only runs one direction.

The skeptic and the builder are not two sides of a balanced argument. They are the same function run on different inputs. If you have never seen a system go off, do real multi-step work, get something wrong, take your correction, and come back with it fixed, your mental model of AI is calibrated to the wrong tier. You're picturing a chatbot. You're rating the chatbot. You're right about the chatbot.

The builder has seen the other thing, and can't unsee it.

There's no symmetric version of this. One side is holding the data and the other one isn't. That's not a difference of opinion. It's a difference of evidence.

You can watch this play out at scale right now. McKinsey's 2025 State of AI survey found that the large majority of organizations report using AI, while fewer than four in ten report capturing meaningful value from it. Gartner has said it expects over 40% of agentic-AI projects to be cancelled by the end of 2027. None of those numbers are a story about AI being overhyped. They're a story about leaders trying to deploy something nobody senior had ever personally operated, and being surprised when it stalled.

The gap shows up as the failure rate. It just doesn't announce itself that way.

The crossing is personal, not institutional

Now the uncomfortable part, the reason this is a book and not a memo.

You cannot read your way across this gap. You cannot buy your way across it with a platform contract. You cannot delegate it to a team and call it handled.

The crossing is personal. Your organization cannot do it for you, no matter how much you spend or how good your people are.

I've watched a lot of capable leaders try the institutional version. Sponsor the center of excellence, approve the big platform deal, stand up a steering committee, wait for the capability to arrive on schedule. Most of them land in the stalled half of those statistics, and most of them can't say why. The work was funded. The vendor was credible. The team was strong.

It stalled because nobody senior in the room had ever driven the thing they were deploying. They were directing work they couldn't evaluate.

The leaders on the other side of the gap did something simpler and harder. They sat down themselves. They picked a real problem, pointed a system at it, watched it produce work, threw out what was wrong, kept what was right, and did it again. Over months, not over a weekend. Until they could tell good from bad from the inside, fast, without asking anyone.

That's all crossing the gap means. Not becoming an engineer. Not learning to type code at a professional level. Learning to direct a system that does the building, and to recognize the moment it's confidently wrong.

The book has a name for the structure you build doing this. The harness: the persistent setup around the model, the skills and memory and tools and workflows that turn a chat window into something you operate. The model is a commodity everyone can rent. The harness is the part that's yours, and it's where the actual advantage lives. The book argues that the harness is the moat, that its parts are knowable and nameable, and that the operator role at the center of it will outlast the next model, and the one after that.

It also argues that the crossing has a shape you can follow. Not a leap. A path with rungs. You start with something small that works, you build the second thing on top of the first, and the setup you carry from each project makes the next one faster. The reason ninety-day pilots fail and six-month builds hold is that the first is a sprint at the thing and the second is a ladder up it. The back half of the book is that ladder, written out rung by rung, for a reader who has never written a line of code and has no plans to start.

The book itself is three parts. Part One names the gap and why your organization cannot close it for you, no matter what you approve or fund. Part Two is the exoskeleton itself: the harness, and the person operating it. Part Three is the crossing, a three-month ladder that starts with you working alone and ends with you leading a small team across the same gap. That's the shape of it, if you want to see it for yourself: get it on Amazon.

You don't have to stop being who you are

The fear under all of this, the one I hear from senior people more than any other, is that crossing the gap means becoming someone else. That the polished VP has to turn into a junior engineer to stay relevant. That the judgment and the relationships and the executive instinct that got them here suddenly count for nothing.

The book argues the exact opposite, and I believe it because I lived it.

Your polish, your read on people, your political capital, your taste in what's worth doing: those don't become liabilities when you start building. They become multipliers. The rare and valuable profile isn't the engineer who learned to present. It's the leader who learned to operate. That combination is rare today not because it's unnatural, but because almost nobody has told leaders it was available to them.

You are not being asked to give up what you're good at. You're being asked to point it at a new kind of system.

I wrote it with the thing it's about

One more thing, because it's the proof and not a gimmick.

I built this book through the same kind of system it describes. It has its own research process that sweeps the current discourse before each chapter. Its own drafting setup that holds the voice and the argument in place. Its own fact-checking pass that pulls every claim into a queue and makes me source it or cut it. A small machine I direct, not a document I type alone.

If a book about crossing the gap had taken three years to write by hand, it would be arguing against itself. This one didn't. The way it got made is the clearest existence proof I can offer for the thing it's selling: a person operating an exoskeleton produces work at a scale and speed they couldn't produce bare-handed, and the work is still theirs. The book is what that produced.

Who it's for

I wrote it for one reader in particular.

The leader who has started to notice that the people around them can now do things with these tools that they can't. Who feels the distance growing month over month. Who hasn't said it out loud, because saying it out loud feels like an admission. If that lands a little too close to home, good. That's exactly who it's for, and the distance you're feeling is more closeable than it looks from where you're standing.

It's not for the true believers waiting on AGI, and it's not for the skeptics holding the line that the whole thing is smoke. They've picked their stories. This is for the people in the middle who still have a decision to make and want to make it from the inside, not from a slide deck someone built to sell them something.

Out today

The book is out. The paperback is live right now. The Kindle edition ships tomorrow, and you can get it today: Builder-Leader on Amazon.

If the gap I've described here is recognizable, if you've felt the space between what your team can suddenly do and what you can, you're closer to crossing it than the argument going on around you suggests. The first step is smaller than it sounds. The book is the rest of them.

A few weeks ago I made the longer case for why this is the gap that decides the next five years, against a piece that I think pointed at the wrong one. This is the follow-through, the one I get to write with the book actually out.

Thanks for reading along to this point. Go cross the gap.

Justin

Anthropic shipped Claude Science today, an AI workbench for scientists. The launch page leads with proteins and genomics and a reviewer agent that checks your citations. That is the surface. The interesting part is what they chose to build it on, and what they chose not to promise.

A workbench is a boring word on purpose. It is the thing the interesting work happens on top of. Anthropic picked that word over "discovery engine" or "research copilot," and the choice tells you where they think the advantage sits.

What actually shipped

Claude Science is a local-first app that runs on your laptop, your Linux box, or your HPC login node. A generalist agent coordinates sixty-plus skills and connectors pre-wired for genomics, single-cell, proteomics, structural biology, and cheminformatics. It renders 3D protein structures and genome-browser tracks and chemical structures inline, next to the code that made them.

Three design decisions are worth pausing on, because they are not how most "AI for science" products are built.

Every figure ships with its own recipe. When Claude Science generates a plot, it bundles the exact code and environment that produced it, a plain-language description, and the full message history. Six months later you can reopen it and see what was actually run. This is the reproducibility crisis spoken to directly, not as a slogan but as a file format. Every artifact is auditable by construction.

The output is not the figure. The output is the figure plus the complete record of how it was made.

Compute is a thing the agent manages, not a thing you schedule. Folding a protein or running a genomics pipeline across a huge dataset usually means stopping your work to write a job script, submit it to the cluster, wait, check whether it failed, pull the results back. Claude Science drafts the plan, asks before it spends new resources, and lets you revoke any decision before it writes and submits the job to the infrastructure you already use. Your HPC cluster over SSH. Your Modal account for compute on demand. One GPU to hundreds.

The detail that does the work: datasets load once into a running session and stay in memory. Large or sensitive data never has to leave the systems it already lives on. Only the slice of context a given step needs goes to the model.

The tooling layer is not theirs. This is the move most people will miss. Claude Science does not ship its own protein-folding model. It talks to NVIDIA's BioNeMo natively, which means Evo 2, Boltz-2, OpenFold3 are one call away. Anthropic orchestrates. The scientific models belong to the tooling layer.

The layer cake

The shape of the market is already clear. AI-for-science is stratifying into three layers, and it rhymes with the stack we already know.

The application layer is the orchestration surface, the thing the scientist talks to. Anthropic and OpenAI's GPT-Rosalind compete here. The tooling layer is the scientific models and databases, BioNeMo and Boltz and AlphaFold. The infrastructure layer is the silicon. Each layer is a real business. The tooling and silicon layers are where the heavy capital and the specialized models live, and the companies there, NVIDIA especially, keep doing the hard work that everyone in the application layer stands on.

Anthropic's bet is narrower and smarter than "best model." The model is becoming the commodity layer. The harness around it is where the moat gets built.

The whole launch is a platform play, not a model play. John Jumper, the Nobel laureate behind AlphaFold, announced he was leaving Google DeepMind for Anthropic eleven days before this launch. The Coefficient Bio acquisition bought target-discovery talent. The partnerships with the Allen Institute, the Broad, Schrödinger, Sanofi, and HHMI are integration deals. The moat being built is workflow integration, not raw model capability.

We have been here

I have been arguing this for a while. The model is the commodity layer. The harness, the system of skills and memory and agents and hooks and connectors that sits around the model and makes it useful, is the moat. I wrote a whole Run Data Run essay on it, and a book chapter. The argument was built on watching what happened when you swapped the model out from under a working agent system and the system barely blinked, because the intelligence lived in the scaffolding.

Claude Science is that argument, built into a product, for science.

The clearest version I have seen up close is a research flywheel we run in my group. It is an autonomous agent that runs continuously, generating and refining research ideas at the intersection of AI and biotechnology, promoting the promising ones to execution on a GPU box, critiquing its own results, and self-healing when a run corrupts. Over an eighteen-week deployment across seven instances it ran 5,196 consecutive commits without a human touching it, with a 97.8 percent autonomous resumption rate. The expensive, interesting part is not any one model call. It is the loop: the weighted idea pool, the adversarial critique, the self-heal, the provenance on every artifact. Swap the underlying model and the loop keeps turning. The loop is the asset.

That is the same shape Claude Science is reaching for, at platform scale. The sixty pre-wired skills. The reviewer agent that checks citations and calculations as the pipeline runs. The session that holds context in memory and forks to compare two approaches. Every one of those is a harness component. None of them is a model.

The model is the part you can rent. The harness is the part you build. Claude Science just shipped sixty reasons to build it for science.

Two reviews worth reading

Two of the beta stories in the launch tell you what this is actually good at, because neither is a discovery claim.

Jérôme Lecoq, a neuroscientist at the Allen Institute, used Claude Science to build a roughly twenty-skill pipeline that writes long-form computational reviews. Sub-agents read thousands of papers, pull the central claim and the key number, and store them in an evidence database. Then a pipeline builds a narrative arc, delegates each section to a specialist, and dedicated agents generate cross-study figures straight from the evidence. The core trick is actor-critic pairs: one agent writes, a separate reviewer agent checks it for accuracy and citation fidelity.

Before Claude Science, that kind of review took his team up to two years. He now has about ten of them, many over a hundred pages, with citations the reviewer agents vetted.

Two years collapsed to weeks, on the part of science that is reading and synthesizing and checking, not on the part that is having the idea.

Stephen Francis, an epidemiologist at the UCSF Brain Tumor Center, used it for germline workups on glioma susceptibility. His lab had done this work before. Claude Science compressed the analysis to roughly a tenth of the time, and his group independently validated the results.

Neither man claims a discovery. Both claim compression of the tedious majority of the work. That distinction is the whole product.

What it does not do

Here is what Claude Science is not, and what the launch page does not claim it is.

It is not a drug-discovery engine. Zero AI-discovered drugs have FDA approval. The ninety-percent clinical failure rate has not moved. The biomolecular models in the tooling layer underneath it work well for structure prediction and initial screening, and they are getting better fast. They are not yet where you want them for the decision that gates a program. A careful recent evaluation (Wan and Coveney, UCL) of one of those models on thirty-eight thousand compounds found strong performance on structure but weaker correlation with physics-based binding free energies at the candidate end, where a medicinal chemist actually bets. That is not a knock on the model. It is where the whole field sits in 2026, and the field knows it.

The pattern is one the coding world already learned. The benchmark looks good. The transfer to the real decision is the harder, slower climb. The structure-and-screening work compresses. The lead-identification work does not, yet.

AI compresses the eighty percent of science that was already tractable. The twenty percent that kills programs stays expensive.

The framing Brendan Frey, the Deep Genomics CEO, gave Drug Target Review is the one to hold onto: "AI has really let us all down in the last decade when it comes to drug discovery. We've just seen failure after failure." Anthropic's head of life-sciences partnerships, Jonah Cool, has been consistent that the target is the tedious intermediate work, data analysis and annotation and coordination. The launch page repeats that framing almost verbatim. A company sitting on a $965 billion valuation, under pressure to grow, chose to launch a workbench and not a miracle.

Why a builder should care

If you lead a data or AI team in the sciences, three things are now true and worth acting on.

The first is that the integration layer is where the money and the moat are going. The specialized models, the folding and docking and single-cell tools, are commoditizing fast as the tooling layer matures. Your durable advantage is not which model you run. It is how well you connect the models to your proprietary data, your lab's instruments, your validated pipelines. Claude Science lets you save any pipeline as a reusable skill and inherit it in future sessions. That is the shape of the advantage. Build the connective tissue, not the model.

The second is that the reproducibility-by-construction pattern is going to spread. Every figure carrying its full provenance, the code and the environment and the message history, is the answer to a problem every R&D leader has been screaming about for a decade. If you are building internal tooling, steal the pattern wholesale. The artifact is not the output. The artifact is the output plus the audit trail.

The third is the scope limit. Claude Science will accelerate the boring eighty percent of your scientists' work, the literature synthesis and the pipeline glue and the figure iteration, and it will do it well. It will not pick your drug target. It will not replace your physics-based validation on the decisions that gate a program. Plan for the compression. Do not budget for the cure.

What I'd watch

The beta is open today for Pro, Max, Team, and Enterprise on macOS and Linux. There is a discounted Team plan for academic labs and nonprofits, and Anthropic is funding up to fifty AI-for-Science projects with up to $30,000 in credits (Modal adds $2,000 of compute). Applications close July 15.

The thing to watch is not the launch. Launches are easy. Watch whether the reviewer agent actually catches the citation and calculation errors six months in, when the novelty has worn off and the session histories are long. Watch whether scientists trust it enough to save their hard-won pipelines as skills and let the next session inherit them. That trust, not the protein renderer, is what earns the workbench a permanent place on the bench.

Launches are easy. Trust earned over long session histories is the hard part.

Anthropic shipped a workbench. They named it for what it is. The science was always eighty percent grinding and twenty percent insight, and the insight was never the part that took the time.

The papers

• Claude Science, an AI workbench for scientists, is now available. Anthropic, Jun 30 2026. The launch.

• NVIDIA launches BioNeMo Agent Toolkit. The tooling layer Claude Science orchestrates over.

• Modal integration brings scalable compute to Claude Science. The on-demand compute leg.

• On the Reliability of AI Methods in Drug Discovery: Evaluation of Boltz-2. Wan and Coveney, UCL. The reality check on binding affinity prediction.

• Get started with Claude Science. The product.

This is a Run Data Run essay. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why.

Earlier this week, NVIDIA gave a global-health startup I work with a node of eight H100 GPUs to use for a couple of months. The grant came through NVIDIA Inception, their program for startups. No invoice. They looked at the mission, believed in it, and handed over the iron.

The day the box came online, I posted the first entry of a build log about it on our technical blog. In the few days since, the pace has been fast enough that the story already needs an update. So here it is properly, from the top, for the people who lead the work rather than write the code.

Here is the part I love most, before anything else. The crew putting that machine to work is six people. Me. Nick, who runs the company. And four interns, the last of them starting tomorrow. Three are in college; the fourth just finished high school and starts undergrad soon.

A node of eight H100s, a foundation model to train, and a two-month clock. The crew is two adults and four interns.

The mission

SocialEyes is bigger than the AI, and that is what pulled me in. The mission is to converge a wide range of AI healthcare skills onto the patient, right at the point of care. Today's workflow is screen, then refer: a frontline worker spots something, and the patient travels to a specialist who may be hundreds of miles and many months away. That model dates to the 1890s, and it simply does not work in low- and middle-income countries. There are roughly 200,000 eye specialists on Earth and well over a billion people with diabetes, high blood pressure, and the slow diseases that take sight and life. The math does not work. SocialEyes is trying to fix the math.

The way in is the eye. It is the cheapest window we have into the rest of the body, the one place you can photograph blood vessels and nerve tissue directly, with no needle, using a camera that fits in a clinic. We lead with timely screening of the chronic diseases that increasingly dominate global health: diabetes, hypertension, and the conditions that take sight before anyone catches them. From the same image we can also pull biomarkers and risk factors, and in some cases a specialist-level read of certain retinal conditions. The retina is just one of the tissues the SocialEyes architecture can handle. By itself, signs of dozens of diseases, in the eye and systemically, can ultimately be detected.

That is what makes it worth two borrowed months. The work is grounded where it is needed most: low-resource clinics across low- and middle-income countries, anchored in Nepal. The first deployments are at clinics and other fixed sites, not yet in the hands of community health workers in the field. That comes later, with a dedicated device called MARVIN. For now: a camera, a trained model, and a clinic a long way from the nearest specialist. That is the product. The retinal AI is how it gets there.

The training data is grounded in public research datasets, which is a deliberate choice. It keeps the method something we can talk about openly, even when specific results stay in the lab. Which is convenient, because that is exactly what I am doing.

How I ended up in the room

SocialEyes is Nick's mission. I build the AI for it, and have been part of the team since the start of the year.

They found me through the open work. The experiments on a DGX Spark on my desk, the writing on the blog, the habit of building the real thing and showing it. Nick reached out, I started helping, I dug the mission, and now we are doing some of the coolest work I have done.

That origin is the whole way I think about building. Build the real thing, share it in public, and the right people show up. A working system and an honest write-up change the conversation from "should we" to "how do we do this faster." This collaboration is what that looks like when it happens.

The four interns are the same story from the other side. Students who wanted to do real work, not a slideshow about AI. So they are doing real work. They have stood up their first shared code repository, they watch the GPU dashboards, they own a lane of the operations. Three are in college and one is about to start. Most people their age get coffee. These four are helping run a supercomputer.

The borrowed iron

Here is the through-line for anyone who has followed the work. The small box came first.

A DGX Spark, NVIDIA's desktop AI machine, sits on my desk with 128GB of memory. It is a real research rig at a tiny fraction of data-center cost. Most of the hard questions, which tools to use, which model, how to make everything behave, got beaten out on that small box first, where a mistake costs minutes instead of metered GPU-hours. The little box is the cheap rig that de-risks the expensive one.

The borrowed node of eight H100s, running in the cloud, gets to stand on all of that. Eight H100s, about as much compute as you can put in a single machine, is enough hardware to train a foundation model from scratch and run dozens of experiments beside it. That is the thing the small box could never do. The lineage is the point: de-risk on the desktop, then deploy at scale in the cloud. And the clock is running. Two months, then the iron goes back.

The DGX Spark on the desk taught us the method for a year. The borrowed node is where we finally get to run it.

The pace

What follows is the short version of a long few days. We brought the node up from a web console to a first running job, stood up a private reasoning model on the box so the team's tools ran on our own hardware, and split its storage into permanent and scratch on a machine that erases itself if you misfile a single thing. We gave the whole team secure access through one entry point, nobody holding the keys to wipe it, and staged terabytes of retinal imagery across four kinds of eye scan, CFP, IR, FAF, and OCTA, without losing a day to a careless mistake. Before spending a GPU-hour, we read the literature and locked the recipe. That alone caught a starting setting roughly sixteen times too aggressive, before it burned a wasted run. Then we trained. The first foundation-model run came back with an uncomfortable verdict: color photos alone topped out no better than an off-the-shelf vision model. So we did not argue with the data. We pivoted to a richer, multimodal approach on a second kind of scan that had been sitting unused on the box, and launched a fresh run inside the same week.

Around the training, we built an evaluation suite to score every model on the tasks that count, ran ablation sweeps overnight with the box chaining its own experiments while we slept, and red-teamed a result that looked too good, catching three stacked bugs before any of them shipped as a finding. I put ARIA, the autonomous research engine I built, on the node and watched it run its first experiment on its own, then re-contracted it for a world where compute is suddenly cheap and gave it a cheat-detector and a wall of baselines so nothing scores well by accident. All of it ran across three AI coding agents in three lanes at once, operations, modeling, and the research engine, kept from colliding by a shared board with clear ownership. We even survived a disaster: a sync tool deleted the shared workspace, and we had it all back the same afternoon from versioned history and a nightly backup, then locked the setting so it cannot repeat. And the whole time, four interns stood up their first shared code repository and ran a real lane of the operations, the GPU dashboards and a slice of the ops.

None of that pace is about working longer hours. It is the harness. The stack of AI tools I have spent two years building and writing about: the coding agents, the autonomous research engine, the skills that turn a one-line instruction into a finished job. The harness does the grunt work. The five of us do the deciding. Point it at a borrowed node and a mission worth the effort, and all of that is a few days, not a few quarters. It also means you can afford to be wrong fast and right next, the way we were when the first model fell flat and the second one took its place inside the same week.

The pace is not longer hours. It is the harness doing the grunt work while the five of us do the deciding.

AIXplore, the lab

I have mentioned the other blog in passing before but never properly introduced it. AIXplore is the lab. Where Run Data Run tells the story, AIXplore shows the work: the engineering, in detail, for the people who do sit in the code. How you give a team access to a borrowed machine without handing out the keys to wipe it. How you stage terabytes of scans without losing a day. How you serve a model locally when the documentation lags the code by a month. Same work as here, one altitude down.

It got a full rewrite this week, too. While ARIA ran SocialEyes experiments on the node in the background, I rebuilt the entire platform. Every new post carries interactive widgets you can poke at, side quests that branch off into the deep cuts without derailing the main read, and, best of all, a reproduction prompt on every piece. Copy it, point your own Claude Code at it, and build the thing yourself. Whether you have shipped models for years or you are trying to start this weekend, there is a door in.

The whole SocialEyes build is going up there as it happens, in a series called Borrowed Iron: standing up the node, the access plumbing, the data staging, the autonomous engine, the training runs. If this post made you want the real detail, that is where it lives. And almost none of it is specific to retinas. The lessons transfer to anyone doing serious work on rented GPUs.

Come along

I am on a break between jobs this summer, a real sabbatical, and this is what I am doing with it. I will get to a beach at some point. I will also publish a book before the summer is out. And the rest of it goes to a borrowed supercomputer, a founder, and four interns, building healthcare for people a long way from a hospital. This is how a sabbatical should go. I figure I will sleep once I start the new job in August.

So that is the setup. A mission I believe in. A partner who backed it with serious hardware. An unlikely little crew, two of us and four interns, moving faster than the crew size has any right to. And a standing invitation to watch us make the most of two borrowed months in public.

New parts land as the work happens, here and on AIXplore both. Come along.

Justin writes Run Data Run for the people who lead the work, and AIXplore (ai.rundatarun.io) for the people who build it. The Borrowed Iron series runs on AIXplore as the SocialEyes build happens. If this was useful, the easiest way to support it is to subscribe and forward it to one person who would want it.

Earlier this month I wrote that two things held their price while AI made almost everything else cheap. Anthropic had just measured one of them across four hundred thousand coding sessions: what the human brings to the work, the command of a domain that lets you direct the machine and catch it when it is confidently wrong. That number was the closest thing we have to proof of an argument I have been making for a while now.

But there were two things, not one. The second one Anthropic could not measure, because it does not live in their data. It lives in yours. It is the system you build around the model, and it is the part of your AI strategy that no competitor can copy.

Recently I made mine public so you can look at it. It is a project called Claudelicious, and before I open it up, I want to convince you why the category it belongs to is where your real advantage hides.

The model is the part of your AI strategy you can order off a price sheet. The part that compounds is the part you build around it.

The part everyone shops for is the cheap part

Most of the AI strategy conversations I sit in still open with the same question. Which model. Which one is smartest this month, which one is cheapest, which one to standardize on. It is a reasonable question to ask once, and a strange one to keep losing sleep over. The models keep getting cheaper and they keep converging; the gap between the best one and the third-best one is a few weeks, not a few years, and the price of yesterday's best falls by half on a schedule you could set a watch to. Picking a model is a decision you will remake four times before this sentence feels dated. It is the commodity layer: you can buy it, you can switch it, and so can the company across the street.

Last month I wrote that the model is no longer the frontier. The first academic conference on agentic systems had just convened, and almost none of its papers touched the model itself. The interesting problems had all moved to the layer around the model: how it improves itself, whether you can tell when it is wrong, what it is allowed to do. That was the research community's verdict. If the frontier moved to the system around the model, then the durable advantage moved there too.

I have been circling this since last fall, when I started writing about Claude as a colleague rather than a chatbot. I put the explicit flag down this spring, in a piece called Start With Claude Code: a year into running this setup, the harness is the moat. This is the full case for it.

What a harness actually is

A raw model, even a very good one, is a brilliant contractor with no memory. It shows up sharp, does excellent work for an hour, and then forgets everything the moment the session ends. It does not know your conventions. It does not remember the mistake it made last Tuesday. It cannot reach for the right tool unless you hand it over, every single time.

The harness is everything you build to fix that. It is the layer that turns a forgetful genius into a system that gets better at your work over time. You can hold all of mine in your head without an engineering degree:

• Rules the model reads at the start of every session, so it already knows how I work before I ask for anything.

• Skills it can reach for, small packaged procedures for recurring jobs, so I describe the outcome and it picks the right one. I keep 48 of them.

• Memory it carries between sessions, in four tiers, so a lesson learned today is still there next week. Behind it sits roughly seventy thousand documents of accumulated context about my work.

• A learning loop that catches its own mistakes and writes them down, so the same error does not happen twice.

• Continuity that lets me close the laptop mid-thought and have the next session pick up exactly where this one stopped.

• Always-on agents, eight of them spread across a small mesh of machines, that keep working long-horizon jobs while I sleep.

None of that is the model. All of it is the thing that decides whether the model is useful to me specifically. A model is a thing you rent for an hour. A harness is a thing you live inside.

A raw model is a brilliant contractor with no memory. The harness is everything that turns it into a colleague who remembers.

This is also why the choice of harness has a character to it, which is the argument I made in Three Harnesses, Three Characters: the same underlying model behaves like a different colleague depending on the rig you wrap it in. You are not picking an engine. You are picking who shows up to work.

What makes a harness great

Not every harness is worth building. A bad one is a pile of half-used scripts that slows you down and a maintenance burden you resent. A great one has three properties, and none of them is about how smart the model is.

It is shaped to you. Every rule encodes a preference I formed by getting something wrong. Every skill is a procedure I refined over dozens of runs. Every memory is a piece of context about my work, my data, my judgment calls that took months to accumulate. A competitor can license the identical model tomorrow. They cannot license eighteen months of my context, because it does not exist anywhere to license.

It learns. Because the harness has a learning loop, it does not sit still. It gets a little better every time it catches a mistake, and it does that whether I am watching or not. So the distance between a tuned harness and a fresh one does not hold steady. It grows. The organization that started building this a year ago is not a year ahead. It is a year ahead and pulling away.

It stays small. This is the property people get wrong, and getting it wrong is how they waste a year. More scaffolding is not better. Every rule the model reads, every skill in its reach, is something it holds in its attention on every single turn, and attention is finite. My own skill library peaked at 134 packaged procedures. I cut it to 48, and the cut was the work, not the cleanup. Compounding does not mean accumulating. It means keeping the ones that work and ruthlessly cutting the ones that don't.

Put those three together and you have something a competitor cannot buy, because it is personal, and cannot catch, because it keeps moving. That is the moat, and it is the reverse of the model question. The model is the layer where every company is equal.

The model is where everyone is equal. The harness is where the distance opens up.

How the harness crosses the gap

Now widen the lens, because this is where the harness stops being a productivity story and becomes a strategy one.

There is a real divide in how people see AI right now, and it is not skeptics against believers. Andrej Karpathy named it this spring: two groups talking past each other. The split that matters is between the people who have built something with these systems and the people who have only read about them, used the free tier, or sat through a demo. Almost every boardroom argument about what AI can and cannot do is a symptom of that divide, not a real debate. The two sides are calibrated to different machines, different use cases, different tiers of the thing.

You cannot read your way across that gap. You cannot procure your way across it either. You cross it by building, and building does not mean writing code. It means directing a harness to produce real work inside a domain you already command. The harness is the thing you wear to do it. My book calls it the exoskeleton, which is the whole title: Builder-Leader: The AI Exoskeleton That Crosses the Gap. The skills to operate one are not new. They are the same instincts that got you to senior in the first place: setting intent and the boundaries around it, designing the handoffs, judging the output against what good actually looks like. Pointed now at a system of agents instead of a team of people.

This is where those pieces and this one are the same argument seen from different sides. You don't have to write the code was the human half measured: domain command, not coding, is what predicts whether you get working results out of an agent, across four hundred thousand sessions. What AI didn't reprice was the human half priced: when output gets cheap, the judgment that knows which output is right gets dearer, and it is the one asset on your books that went up in value this year. The harness is the other half, the system that judgment directs. The two are useless apart. A great harness pointed by someone who does not know the field just produces wrong answers faster. Deep expertise with no harness leaves most of its own capability sitting on the table.

A leader who commands the domain and has built the harness holds a position the company across the street cannot copy. Not the model. The pairing.

That pairing is the whole point. One half is what you carry in your head. The other half is what you build around the model to put that head to work at machine speed. Together they are the thing no price sheet sells.

Why I made it public

So I wrote mine down. Not as a tutorial, and not as a flex. As a cookbook of why: why each piece exists, what problem it solves, how the parts fit together, and where I drew the line. It is called Claudelicious, and it lives on GitHub.

It is built around one idea: run the model as a system, not a chat box. And it is deliberately not a catalog. The community already maintains a good directory for finding a skill, a hook, a connector. Claudelicious is the other half, the part you cannot reverse-engineer from a screenshot: how those pieces fit together into a harness that remembers across sessions, corrects itself when you catch it, and keeps working while you sleep.

So it walks the six pieces I named earlier and shows the wiring under each. The rules that make the model know how you work before you ask. The skill library, and the discipline of keeping it small. The memory and the second brain behind it. The learning loop that makes a correction stick instead of recurring. The continuity that survives a closed laptop. The always-on agents and the small fleet they run on. Every chapter is principle first, then a worked example, then plain notes on what to copy and what to leave behind. There is a quickstart for getting the spine in place in your first week, and a longer version for reading the whole thing as one story.

It is also the working companion to the book. The book makes the case that the people who win with AI are not the ones who pick the best model, but the ones who build the system around it and lead from inside it. Claudelicious is that case made concrete: one such system, with the reasoning shown, that you can open and read.

I made it public for a specific reason. You cannot buy a harness. There is no SKU for the thing I have described, and there never will be, because the whole point is that it is yours. But you can look at the shape of someone else's, see which parts map to your work, and decide what is worth building and maintaining in your own organization. A worked example beats an abstract argument. So here is mine, with the reasoning shown.

Do not walk away with a shopping list. Walk away with the question underneath it.

Your AI edge is not on the model menu. It is the answer to a question no vendor will answer for you.

The question is this. A year from now, the model everyone runs will be cheaper and smarter than today's and roughly the same as your competitor's. So what will be different about how your organization uses it? What did you build around it that the next org did not, that learned from your work while theirs sat still, that no one can license because it is made of your own accumulated judgment?

If the answer is nothing, that is the gap to close this quarter. Not which model. What is the harness, who owns it, and is it learning.

Where this goes next

The harness is one of the two things AI did not make cheaper. The other is the thing you point it with, and that one did not just survive the markdown. It got more expensive. I made that case earlier this week: when output gets cheap, the scarce ability to judge it gets dearer, and the judgment of someone who knows your field is the one line on the books that AI repriced upward. The harness is the system. That judgment is what tells the system where to aim. Most teams are still budgeting for neither.

For now, sit with the pair. A leader with a domain in their head and a harness around the model is not waiting for a better model to arrive. They are already pulling away from the ones who are.

Sources

• You Don't Have to Write the Code. Anthropic's 400,000-session study, the population-scale measurement of the human half: domain command, not coding, predicts success.

• What AI Didn't Reprice. The pricing of that same human half: judgment is the one asset AI marked up while it marked almost everything else down.

• The Model Is No Longer the Frontier. The first agentic-systems conference, where the research frontier moved off the model and onto the layer around it.

• The Quiet Week Claude Became Your Coworker. Last fall, where the colleague-not-chatbot version of this argument started.

• Start With Claude Code. This spring, where I put the explicit flag down: the harness is the moat.

• Three Harnesses, Three Characters, One Working Week. The same model behaves like a different colleague depending on the rig around it.

• Claudelicious. The public cookbook: the why and the wiring behind a full harness.

• Builder-Leader: The AI Exoskeleton That Crosses the Gap. The book the harness is the working companion to.

Run Data Run is free, no paywall. If this was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

In this episode

• The two stories you heard this week, and why both are true

• Why “the truth is somewhere in the middle” is the expensive wrong answer

• The distribution gap: where the gains are actually clustering, and why code sprints while writing, search, and advice crawl

• The split nobody covered: inside the paid tier, the people who use these systems versus the people who build the harness and drive them

• Why this isn’t just another early-adopter curve that closes on its own

• What “driving” actually looks like, and why most people never cross over

• The one new muscle to add if your career was built on judgment, not code

Referenced in this episode

• Andrej Karpathy’s April thread on the two groups talking past each other

• Jason Lemkin (SaaStr): “how awed you are by AI is almost perfectly correlated with how much you actually use it to build”

• Anthropic’s report on how agents actually get used (software engineering was nearly half of all agent activity)

• The roughly one-million-conversation study on how people use ChatGPT (coding around four percent of messages)

• Gary Marcus calling Claude Code the single biggest advance in AI since the large language model

• 2025 McKinsey, Gartner, and IT-leader surveys on stalled AI projects and agent sprawl

From the book

This is chapter one of Builder Leader: The AI Exoskeleton That Crosses the Gap. Each episode takes one idea from one chapter and talks it through. Subscribe to get the next one.

Think of the last two years of AI as one long, indiscriminate markdown sale. Skill after skill that used to be scarce got cheap, fast. Writing a clean function. Drafting a competent memo. Summarizing a dense report. Producing a first-pass analysis. A year ago each of those was something you hired for, waited on, or did yourself at the cost of an afternoon. Now they are a prompt and a few seconds.

The sale hit almost everything. It missed one thing, and the thing it missed didn't just hold its price. It went up.

Last week I put four hundred thousand sessions behind that claim: Anthropic's analysis of real coding work found that what predicted success wasn't your job title or your syntax, it was whether you understood the problem. That was the evidence. The implication: a few things got more valuable this year, not less, while everything around them got cheaper.

Almost everything got cheaper this year. The ability to tell right work from confident-looking wrong work got more expensive.

The half that got cheap was the doing

Start with what fell in price.

What AI commoditized is the execution of a known task. Give a model a well-specified job inside a domain it has seen a million examples of, and it will do it about as well as a competent junior, for free, instantly, at three in the morning. That is most of what people mean when they say AI is changing knowledge work, and they are right about it.

I've made this argument from a few angles already. When the model stopped being the frontier, the interesting problems all moved to the layer around it. When Apple rented its brain, it conceded that the model itself is now the rentable, swappable part of the stack, even for the most vertically integrated company on earth. The engine is a commodity. You can buy it, switch it, and so can your competitor.

So if the doing got cheap, the question is what didn't, and why it climbed while everything around it fell.

The half that got expensive was the judging

"AI does the work" treats the work as if it were one thing. It isn't. Every task has a doing half and a judging half, and AI only ate the first one.

The doing is producing the output. The judging is knowing whether the output is right. Not right in general, not right on a benchmark, but right here, for this patient, this contract, this market, this dataset with its ten-year-old quirk that everyone who has worked it knows about and no document anywhere records.

That second half is domain expertise, and it resists the markdown for a structural reason: it isn't in the training data. A model learns from what people wrote down. The deepest domain knowledge was never written down. It lives in the head of the person who has run the assay four thousand times, sat across from the regulator, watched the trade go wrong in 2019, and can look at a fluent, plausible, beautifully formatted answer and say, with a flat certainty, "no, that's not how this behaves." You cannot prompt your way to that, because the corpus the model trained on doesn't contain it.

This follows a basic rule about complements. When you flood a market with cheap supply of one thing, the price of its complement goes up. Cheap output makes the scarce ability to judge that output worth more, not less, because now there is a hundred times more output to judge and the same small number of people who can tell which of it is wrong. AI didn't just spare domain expertise. It bid the price up.

Cheap output doesn't lower the value of judgment. It raises it. There's a hundred times more to judge and the same few people who can.

Two voices, different worlds, one conclusion

What turned this from a hunch into a post was watching it land from people who don't share a desk, a discipline, or a reason to agree.

In March, the Berkeley California Management Review, about as institutional as business thinking gets, told executives their next competitive moat is tacit knowledge: not the data, not the models, but the judgment embedded in their people, and the systems that capture it before it walks out the door. Two months later, a working software engineer named Aaron Brethorst, with no reason to read a business-school journal, wrote that domain expertise has always been the real moat. His point: the framework knowledge, the syntax, the boilerplate, the part everyone used to grind years to acquire, was never the durable thing. AI making it free didn't destroy the moat. It drained the water and showed you where the moat actually was the whole time.

One came at it from the top of the org chart, thinking about strategy and knowledge graphs. The other came at it from the keyboard, watching a model write code he then had to check line by line. Different altitude, different vocabulary, different month. Same conclusion. Put their two arguments next to the four hundred thousand sessions from last week and you have a management journal, a working engineer, and a population-scale measurement all pointing at the same line through the work. When the same claim arrives from corners that don't read each other, the content is almost secondary. The convergence is the signal.

The one asset that appreciated

Domain expertise is the one appreciating asset on the books. Everything else in your AI strategy is depreciating. The model you standardized on this quarter will be cheaper and roughly matched by your competitor's within months; you'll remake that decision four times before it matters.

Which changes what the work actually is. The advantage isn't holding the expertise. It's how fast you're turning it into something the machine can use before the person carrying it retires, quits, or simply forgets the quirk from 2019: into context, into the checks that catch a wrong answer before it ships, into the loop that lets one expert's judgment steer a hundred agents instead of one. That capture is an investment with a return, and most organizations aren't making it because they still have expertise filed under cost.

There's a second thing AI didn't reprice, and it pairs with this one: the system you build around the model, the harness that turns a forgetful chat box into something that compounds on your work. I put the flag down on that one in Start With Claude Code, and it's the subject of this Sunday's deep dive. A great harness pointed by someone who doesn't know the field just produces wrong answers faster. The judgment is what tells the system what good looks like. The two are a pair, and neither is on the model menu.

Everything in your AI strategy is depreciating except two things: the judgment of the people who know your field, and the system you build to put it to work.

The decision that compounds

The decision that compounds isn't which model to pick. You remake that one on a schedule, and so does everyone else.

The companies that win the next few years won't be the ones with the best model. Everyone has roughly the same model. They'll be the ones who looked at the markdown sale, noticed the single thing whose price went the other way, and spent like they understood why.

Sources

• You Don't Have to Write the Code (Run Data Run). Anthropic's 400,000-session study: domain understanding, not job title or coding skill, predicted success. The evidence under this post's argument.

• Berkeley California Management Review, Tacit Knowledge Is Your Next Competitive Moat (March 2026). The institutional framing: the differentiator that lasts is the judgment embedded in your people, and the systems that capture it.

• Aaron Brethorst, Domain Expertise Has Always Been the Real Moat (May 2026). The builder's framing: framework knowledge going free made the real moat visible.

• Start With Claude Code (Run Data Run). The other thing AI didn't reprice: the harness you build around the model.

This week Anthropic published an analysis of about 400,000 real Claude Code sessions, run between last October and this April. It set out to answer a plain question: when someone sits down with a coding agent, what actually predicts whether they succeed? There is a comfortable answer and a more useful one, and almost everyone is repeating the comfortable one.

The comfortable answer, the one the headlines took, is that anyone can build software now. That part is true, and it is the least interesting thing in the report.

The finding underneath is the one I'd hand to anyone, whether they run a team or just their own work. What predicted success was not your job title, and it was not whether you could write code. It was whether you understood the problem you were trying to solve. Anthropic's own framing: success comes from how well a person understands the work, not whether they're trained in coding.

That is not a story about AI flattening the gap between your best people and your average ones. It is the opposite. The thing you spent fifteen years getting good at just became the thing that decides whether the most expensive tool you're buying actually pays off. I have been making that case for a while. This is the first time anyone has put 400,000 sessions behind it.

What the tool actually divided up

Start with the cleanest number, because it doubles as a mental model you can carry into a Monday meeting.

In a typical session, the human made about 70 percent of the planning decisions and the agent made about 80 percent of the execution decisions. People decided what to build. The agent decided how to build it. That split held across every kind of work they measured, from writing code to running systems to analyzing data.

So the labor didn't disappear. It separated. The agent took the part a lot of people thought was the moat, the ability to actually produce the syntax, and handed back the part that was always the hard part: knowing what to ask for, and whether the answer is any good.

Here is what makes me trust the number. A practitioner reached the same shape from the other side. Addy Osmani, writing in January, found that the developers succeeding with these tools "spend 70 percent of their time on problem definition and verification strategy, 30 percent on execution." Two independent measurements, one watching a population and one watching working engineers, landing on the same line through the work. The tool made the typing cheap. It did nothing for the knowing.

The tool made the typing cheap. It did nothing for the knowing.

This is the thing I've been calling a change in the atomic unit of work. The unit a person owns moved up the stack, from producing the thing to directing it. Now there's telemetry under the claim.

Your job title barely moved the needle

This is the result Anthropic led with, and it earns the lead.

They scored a verified success rate, which is stricter than it sounds: a session counts as a win only if the model judged it successful and there was a hard signal to back that up, a real commit, a passing test, a user saying yes. By that bar, on code-producing work, software occupations succeeded 34 percent of the time and everyone else succeeded 29 percent. A marketer and a staff engineer finished a coding task at almost the same rate.

The instinct is to read that as "engineers are finished." It is the wrong read, and the same study hands you the right one. What collapsed was the premium on being a programmer. What held was the premium on understanding the problem. Measure expertise correctly and it mattered enormously: novice sessions succeeded 15 percent of the time, and people who actually knew their domain landed between 28 and 33 percent. Roughly double.

The reconciliation is the part that clarifies it, because it is the bet I've been making. Expertise here is not your title or your résumé. It is task-specific. Anthropic's own example: a senior engineer asking their first question about the Rust language is a beginner at Rust. The skill that predicts whether you get working software out of an agent was never "I am an engineer." It is "I understand this particular problem well enough to direct the work and catch it when it goes wrong."

That is the whole argument for the builder-leader, measured now across a population instead of asserted from a single desk. You do not have to become an engineer. You do not have to write the code. You direct it, inside a domain you already command, and the command is the thing that pays.

You don't have to become an engineer. You don't have to write the code. You direct it, inside a domain you already command.

The expert tell is recovery

If I had to keep one number from the whole report, it would be this one.

Experts didn't just prompt better on a good day, though they did do more with each instruction: about 12 agent actions per prompt versus 5 for novices, and roughly five times the output. The real difference showed up when things went sideways. When a novice hit trouble, they walked away with nothing written about 19 percent of the time. For everyone with more domain knowledge, that abandonment rate was 5 to 7 percent.

Sit with what that means. The expert hit the same wall the novice did. Then they routed around it, reframed the problem, and caught the fluent, confident, completely wrong answer before it shipped. The novice hit the wall and quit.

So the value your senior people add in an agentic workflow is not that they prompt cleaner on a Tuesday. It is that they fail better on a Thursday. They have the judgment to know when the plausible output is wrong, and the agent does not have that judgment and cannot get it from a model update. The bottleneck moved from typing to deciding. Pratima Arora at Smartsheet put it plainly this spring: the hours haven't changed, but the density of work has.

The scarcest thing in the building is now the thing your best people already carry. That is good news, and most of the coverage will skip right past it.

The scarcest thing in the building is now the thing your best people already carry.

The work moved up the stack

Two more numbers close the loop, and they are the ones that tell you where this is going.

Between October and April, the share of sessions spent fixing broken code fell from a third to under a fifth, 33 percent down to 19. Over the same stretch, the estimated value of the work people brought rose about 27 percent, with the biggest jump in building something new, up 43 percent.

Put those together and the trajectory is clear. People are not using the agent to fix more bugs. They are using it to attempt harder, more valuable, more end-to-end work, and bringing more judgment to bear when they do. The tool got cheaper per task, so the tasks got more ambitious.

This is the oldest pattern in economics wearing new clothes. Make a thing cheaper to produce and you do not produce less of it. You produce far more, and you need more judgment to steer all of it. That is the shape of this moment, and it cuts against the fear that the work is shrinking. The work isn't shrinking. It changed: what we attempt got bigger, and how we get there moved from doing to directing.

Two things held their price, not one

There is a second thread here, and it is the one that turns a study into a strategy.

Anthropic measured what the human brings to the session, and found that what the human brings, domain command, is decisive. That is one of two things this whole shift never made cheaper. The other is what you build around the model.

The model is the commodity, and the system you build around it is the moat: the rules and skills and memory that turn a smart, forgetful chat box into something that gets better at your specific work over time. Birgitta Böckeler, writing on Martin Fowler's site, named the gap the model itself can never close. A coding agent, she wrote, has "no social accountability, no aesthetic disgust at a 300-line function, no intuition that 'we don't do it that way here,' and no organisational memory." Those are not features you buy a newer version of. They come from the person and the system around the model, or they don't come at all.

So the picture is narrower and more useful than "expertise wins." Two things resisted the repricing. What you bring to the model, which is domain command, and what you build around it, which is the system that holds your organization's judgment. The model in the middle, the part everyone is still shopping for on price, is the cheap layer between them. Anthropic just measured one of those two human pieces at a scale none of us could reach alone. The other one you build.

And both are leadership skills, not engineering ones. Setting intent, designing the handoffs, evaluating output against what good actually looks like. Those are the same instincts that got your best people to senior in the first place, pointed now at a system of agents instead of a team of people. No new species of human required. That is the part I keep coming back to, and it is why I think the people who internalize this will out-build the ones still arguing about whether the juniors are coming for the seniors.

One honest note

At the level of the whole economy, you cannot see this yet in the aggregate numbers. A large Danish study of AI chatbots across occupations found no significant effect on earnings or recorded hours. I take it seriously, and I read it as a statement about timing, not a refutation. Session behavior changes before payroll does. The 400,000 sessions are the leading edge; the wage data lags. Hold the whole argument a little more loosely for it, and don't over-rotate on any single quarter's story.

What I'd do with this Monday

If you run a team, here is the part that converts. If you don't, read it as where to put your own time.

Put the agent in the hands of your domain experts directly, not only your engineers. The advantage is highest exactly where deep knowledge meets the tool. The clinical-operations lead who has watched the trial workflow break in a dozen specific ways will get more out of an agent than a generalist engineer who hasn't, because the agent can produce the code but it cannot supply the judgment about what the code is for.

Hire and promote for understanding the problem and recovering well, not for who writes the cleanest syntax. The data says occupation barely predicts success and recovery strongly does. That is now a measurable thing to select for.

And grow your next builder on purpose. The way across this gap was always building, six months of directing real work inside a real domain, not sitting through a demo. The leaders who win the next few years won't be the ones with the best coders. They'll be the ones who turned their domain experts into builders before anyone told them it was allowed.

The agent decides how. You still decide what, and you still decide who learns to decide what next. Four hundred thousand sessions just told you those are the two jobs worth keeping. They are not a smaller job than the one before. They're the better one.

Related: The Specialist Is Now You is what one expert plus an agent can now do alone, and The Atomic Unit of Work Just Changed is where the unit a person owns moved up the stack.

The fuller argument for treating judgment, intent, and domain command as the durable skills is the Builder Leader field guide (builder-leader.com).

Every Sunday I pick one paper or release that's worth your time, break it apart, and tell you why it matters. No hype. No summaries of summaries. Just the idea, explained.

A researcher walks over with a sequence on her screen. She has a stretch of DNA, a promoter, the switch that turns a gene on, and one variant sitting inside it that she suspects changes how loudly that gene gets expressed. She wants to know if the variant matters. So she asks the question everyone asks now: which AI model should I use? BOLT-LMM or DNABERT?

It sounds like a ranking question. Pick the better of two tools. It is not a ranking question. It is a category error, and it's one I've watched smart teams make without noticing.

BOLT-LMM and DNABERT do not belong in the same column. One is a statistical method built to scan a whole population and find which genetic differences track with a trait across thousands of people. The other reads a single stretch of DNA and turns it into a numeric representation a computer can work with. Asking which is better for a promoter variant is like asking whether a microscope or a telephone is better for measuring temperature. The answer is neither, and the question itself is the problem.

The hardest mistake in genomic AI isn't picking the second-best model. It's picking the wrong kind of model and not knowing it.

The phrase that hides nine different things

"Genomic AI model" sounds like one category. It is at least nine.

There are DNA language models, which read raw sequence and learn its grammar the way a text model learns the grammar of English. There are sequence-to-function predictors, which take a piece of DNA and predict what it does in a cell, how strongly it drives expression, whether a region is open or closed. There are variant-effect predictors, which score how much a specific mutation changes that function. There are statistical-genetics engines, which never look at sequence grammar at all and instead test, across a population, which genotypes associate with which traits. There are single-cell models that work on tables of gene activity per cell, generative models that design new sequences, and a few more.

Every one of these gets called a "genomic AI model." Every one of these shows up in benchmark papers ranked against the others. And every one of these answers a different question.

The researcher with the promoter variant needs a variant-effect predictor or a sequence-to-function model, something that reads her one sequence and tells her what the change does. BOLT-LMM, the statistical-genetics engine, is built for a completely different shape of problem: thousands of people, their genotypes, their traits, and the search for associations across the cohort. It has no way to read the grammar of her single promoter. DNABERT, the DNA language model, reads grammar beautifully but doesn't produce a calibrated answer to "does this variant matter" without more machinery bolted on top.

The two tools the researcher was choosing between were both wrong. The benchmark she'd consult to choose between them can't tell her that.

Why the leaderboard can't catch this

Here is the uncomfortable part. The benchmarks the whole field relies on are structured to make this error invisible.

A leaderboard ranks models within a class. It lines up five DNA language models and tells you which embeds sequence best. It lines up four variant-effect predictors and tells you which scores variants most accurately. That is useful, and the people building those benchmarks are doing careful work. But a leaderboard assumes you already chose the right column. It answers "which is best?" and stays silent on the question that comes before it: best at what, and is that even what I need?

The wrong-class error happens upstream of every benchmark. By the time you're reading a leaderboard, the category decision is already made, usually without anyone noticing a decision was made at all. You typed a model name into a search box, found a paper that ranked it highly, and never asked whether the thing being ranked was the right kind of thing for your task.

Nobody walks into this on purpose. The pull comes from how the field names things. Two models share the word "genomic," both have an impressive accuracy number in a recent paper, both have a HuggingFace page and a star count, and the surface presentation makes them look like rival products on the same shelf. A leaderboard reinforces the illusion by placing them in the same table. Nothing in that experience signals "these answer different questions." The signal you'd need is a layer that sits before the ranking and sorts tools by what they're for, and that layer mostly doesn't exist.

I'd put this error ahead of picking the second-best model in the right class, and it's far more costly. Pick the second-best variant-effect predictor and you lose a few points of accuracy on a task that was at least the right task. Pick a statistical-genetics engine for a single-sequence question and you don't lose accuracy, you get an answer that means nothing, dressed up to look like it means something. The numbers come back formatted, plotted, ready to drop into a slide, and there's nothing on the surface that tells you the foundation was wrong. Weeks of analysis can ride on top of a category mistake made in the first thirty seconds, and the failure surfaces late, if it surfaces at all.

Classify first, rank second

The fix is almost embarrassingly simple to state and surprisingly hard to do without help: classify the computational object before you rank it.

Before you ask which model is best, ask what kind of object your task actually needs. Do you have one sequence and want a functional readout? You need a sequence-to-function or variant-effect model. Do you have a cohort of people and want to find trait associations? You need a statistical-genetics engine. Do you have a table of cells and want to label cell types? A single-cell model. The class is determined by the shape of your data and the shape of your question, not by which model has the most citations this month.

Get the class right and the leaderboard becomes useful again, because now you're ranking within the right column. Get the class wrong and the leaderboard is worse than useless, because it gives you a confident ranking of tools that can't do your job.

The reason this is hard in practice is that the class isn't printed on the box. A model's documentation tells you its architecture and its scores; it rarely says, in plain terms, "this is for cohort association, not single-sequence scoring." You have to infer the class from the shape of the inputs it takes and the outputs it produces, and that inference is exactly the expertise a researcher new to a method doesn't have yet. The whole point of a triage layer is to do that inference for you and to fail loudly when the tool you named is the wrong kind of thing.

What this looks like as a tool

I built a small open thing to make this concrete, because it's easier to argue for "classify first" when you can watch it fire.

It's called ModelMap, and it's a decision layer rather than a leaderboard. You describe your task in plain terms, what you have and what you want, and it triages the class first. When the tool you named doesn't match the class your task needs, it returns a "wrong-tool" verdict: here is the class your task needs, here is the class this tool actually belongs to, and here is why they don't match. Ask it whether DNABERT-2 is right for a population GWAS and it routes you to the statistical-genetics column and flags DNABERT-2 as the wrong class, with the real function of each spelled out.

Underneath are nine method classes and around thirty model cards, with deterministic rules handling the clear-cut category calls and a grounded language-model layer translating free-text questions into the controlled vocabulary. It's a research-use proof of concept, source-available, and you bring your own model key so there's nothing to abuse. I built it because I couldn't find anything that does. The closest existing tool, OmniGenBench, is a leaderboard, excellent at ranking within a class and silent on which class you need.

A leaderboard answers "which is best." The question that wrecks more projects is "best at what, and is that what I need."

ModelMap's whole point is that one verdict card. It's the moment it tells you the two things you were comparing were never comparable, and points you back one step to the decision you skipped.

If you build tools like this, the harder question is how you let a language model sit in front of that verdict without letting it invent a class or a license. I wrote that part up for the people who'd build it, over on AIXplore: Build an LLM Triage Layer That Can't Freelance.

The question to carry

You don't need ModelMap, and you don't need to remember the nine classes. You need one habit.

The next time someone on your team asks which genomic AI model to use, or which large model, or which forecasting method, or which anything, resist the pull to answer the ranking question they asked. Ask the one underneath it first. What kind of object does this task actually need? Are we even in the right column?

Most of the time the ranking question is the easy part, and the field has built good tools for it. The category question is the one that decides whether the answer means anything, and almost nothing in the standard toolkit asks it for you. That's the gap worth closing, in genomics and well beyond it.

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.