In the week of 8 September, the four largest AI labs agreed in public, for the first time, that the industry is moving too fast.

On the Tuesday, a researcher named Jacob Coxon resigned from Anthropic and posted about it. He'd spent three years doing pretraining research at both OpenAI and Anthropic. "Neither company is acting responsibly," he wrote. "They are racing straight to self-improving superintelligence and gambling with our lives." CNBC reported the post passed 70 million views. My own sweep of X logged it at 171 million, and it was over 172 million by the time I finished checking.

Then Anthropic's own alignment lead agreed with him in public. Evan Hubinger: "Jacob is correct here, we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." He added that Anthropic is trying its best and does not yet have a plan to solve alignment for superintelligence.

The day before, OpenAI's chief scientist Jakub Pachocki had already written that no AI company has "solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer," and that he expected and hoped for "voluntary slowdowns to become commonplace."

On Saturday, Dario Amodei published We Must Pace the Frontier. Sam Altman agreed the same day. Elon Musk posted three words: "Dario is right." Demis Hassabis followed about nine hours later. By Sunday, President Trump was telling reporters "whoever wins AI wins," and by Monday he'd posted five times on Truth Social calling the risk a hoax. A week later he phoned Huang live on stage at another conference to say it again.

In the same week, Anthropic picked the Nasdaq and is expected to list soon, with a $2 trillion valuation floated.

Here is the fact that survives all of it: nobody in the argument is proposing to stop. Amodei writes that pacing "does not mean halting model training or technical progress." Altman writes that "when we talk about 'pacing,' we do not mean 'stopping.'" Every party agrees on that, including the ones shouting at each other.

Every party in this fight agrees that nobody is stopping. They are arguing about who gets to check.

What is actually in the document

The essay proposes three steps, and most of the coverage merged them. They are three different sorts of promise.

Step one is a commitment with contract terms attached, and Anthropic committed to it unilaterally. Third-party evaluators get "desks in our offices, access badges, and company laptops," plus permissions "mostly comparable to what internal risk assessment teams have." The terms are the substance: reviewers can publish findings "without editorial control by Anthropic." Anthropic keeps a narrow right to redact security-sensitive, privileged, commercially sensitive or third-party material, and in Amodei's own words "can't redact findings just because they are unfavorable." Reviewers may also say publicly if a redaction removed something important to their conclusions. He says it "goes far beyond what any AI company is doing today."

Step two needs an antitrust waiver, and the essay's only footnote is that footnote. Frontier labs in democratic countries would coordinate on common standards, a conversation that normally arrives with lawyers attached.

Step three needs China, and he rates the strongest version unlikely. Four levels, from a bioweapons-use ban up to a full pause. On the full pause: "I support floating this, but I think it is unlikely to actually happen any time soon."

Altman matched step one. Only step one, same day, in his own words: "committing to having independent evaluators with employee-like access is a great idea, and we will do the same."

1,386 employees of frontier AI companies had already signed a statement called Pacing the Frontier back in July, asking the US government to support an international effort to develop the tools to do this. The signatories include John Schulman, Ilya Sutskever and Shengjia Zhao. Sutskever's comment on that page is the one that has aged best: "This works only if it is done internationally, and it has to be done well: a bad implementation can make things worse." The petition predates the essays by two months.

Three days later, two of the three steps had already moved, in opposite directions. On Tuesday, Marc Benioff put Amodei and Nvidia's Jensen Huang on the same Dreamforce stage, 10,000 people in the room and 10 million watching online. Amodei made the case again. Huang rejected the premise of step two: "We don't need any new laws. We don't need new regulations." His mechanism is the market plus self-restraint. "If you build a product or a service and you're not confident in its functionality, capability, or safety, then don't release it." On the tradeoff the whole week was about: "It's a false choice. You could definitely have both at the same time." Later that day he told CNBC that the labs need no antitrust exemption to coordinate on safety, because "we have plenty of laws."

The more consequential rejection came from inside the coalition. The same Tuesday, in Washington, OpenAI's global policy chief Chris Lehane told reporters that OpenAI, Anthropic and Google DeepMind have been coordinating on safety for several weeks already, and that they do not need the waiver Amodei asked for. He also said OpenAI supports a provision in the bipartisan FRONTIER Act requiring top frontier labs to admit "independent verification organizations." So step one is heading for statute, step two is being done without the legal cover its author said it required, and on step three Musk proposed that the leading American labs and Chinese companies test each other's models.

The objection comes from every direction at once

The coverage flattened this into two sides. Read the week's primary posts and the positions do not line up on one axis.

David Sacks, the White House AI czar, accused the labs of regulatory capture and then told them to go ahead. His post: "People may be surprised by my response: go ahead. I don't see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible." On CBS he put the question back to them: "Why are you acting like this is something you can't control? If you can't control it, then don't do it." He's not saying the risk is fake. He's saying they already have the authority and should stop asking permission.

Gary Marcus gave it two cheers out of three. He called the essay possibly "one of the most consequential essays of the year, if not the decade," said he "especially love[s] Dario's commitment to transparency," and in the same post called the opening "the usual hypey bullshit." He also thinks the internet-takeover scenario doesn't hold, pointing to the UK safety institute's own finding that the model in question could autonomously compromise only small, weakly defended systems. And on 3 September, Marcus published a post opposing the Sanders-Casar bill that would ban superintelligence, on the grounds that it's too broad. The best-known critic of AI hype in the conversation opposes the most aggressive pause bill in Congress for being too broad.

Nathan Lambert explained the week rather than the essay. His read is that the resignation "caught like wildfire" because the ground had already dried out from the summer's incidents. His mechanism is three words long: "fear sells." He puts complete extinction "so low it isn't worth discussing" while arguing cyber and bio disasters deserve serious debate, and he says lab staff "operate with a religious energy" that distorts their own forecasting.

Melanie Mitchell thinks the vocabulary is doing the arguing. Words like swarm, cage and colluded, she says, are misleading anthropomorphic framing for agents doing what they were unintentionally permitted to do.

The market read it as theatre. Cohere's Aidan Gomez called the proposal "a cartel by any other name." Bill Gurley, who gave a talk on regulatory capture three years ago, posted: "I did predict that the large incumbent AI companies would 'beg for' regulation. No idea they would beg this hard." Gil Luria at D.A. Davidson gave the flat version: unless a company says "we're not going to IPO, we're not going to use any more compute, we're not going to train any more models," nothing has changed, and "that's not what they're saying."

Then there's the objection the essay never addresses, and it's the structural one. Every lever in the document acts on a company: a badge, a desk, a laptop, a contract, a regulator, an export control. Each of them needs a corporate entity to bind. Lambert puts the open-weight gap to the closed frontier at roughly three to five months, down from six to nine. A 13 September post by Paddo called "Only the Paced Get Paced" put it plainly: "Once weights are on Hugging Face there is no office, no gate, and no pace." Amodei's answer is his China section, which is about export controls and weight-theft prevention rather than about files already published. The essay does not mention open weights.

Hold that one. It comes back in the next section, in a way nobody planned.

The best-known critic of AI hype opposes the strongest pause bill in Congress. The White House official accusing the labs of capture told them to slow down anyway.

Everybody doubts the same piece

The whole plan routes through third-party evaluators. It's the one place where a supporter and a critic say the same thing.

Zvi Mowshowitz, who calls the essay "Actual Progress" and the evaluator proposal "an excellent proposal," still flags independence, competence and sustainable funding, and describes the difficulty of finding AI talent untouched by the major labs. Marcus, from the other end, notes that Amodei points to METR and calls them "creatures of the Valley," concluding that "dropping METR's name is a form of regulatory capture." Marcus also relays Gillian Hadfield's structural point, which is that an evaluator embedded long enough gets absorbed. His plainest line: "Certainly Anthropic themselves shouldn't be choosing who does the evaluation."

Nobody has solved this. Under the proposal as written, Anthropic chooses its own evaluator, and the market it would choose from is small. A practitioner quoted by CNBC could name three firms doing this work at frontier scale: Irregular, the non-profit METR, and Apollo Research.

That shortage is not a hypothetical. It already has a record, and the record is five weeks old.

What actually happened in July and August

On 9 August, CNBC published a story with the headline "How a small Israeli startup was linked to rogue AI hacks at OpenAI, Anthropic and Meta." It got a fraction of the attention the September essay got, and it tells about half the story.

Between 21 July and 6 August there were six disclosures, from four labs and two test environments. Laid out in order they stop looking like a run of bad luck and start looking like a map of where the industry does its testing.

The vendor is Irregular, formerly Pattern Labs, founded in 2023, based in Tel Aviv, roughly 35 employees, $80 million raised from Sequoia and Redpoint, valued at $450 million last year. How many firms do this at frontier scale is part of the story. The practitioner CNBC quoted could name three. Against a field that size, three labs sharing a vendor is arithmetic before it is a coincidence, and the interesting number is the three, not the coincidence.

Anthropic, on 30 July, was first: "A misconfiguration left the machines that Claude accessed as part of the evaluation with live internet access. Neither we nor our evaluation partner were aware of this misconfiguration until we detected it." It found three such incidents inside a review of 141,006 evaluation runs, and names Irregular in the same post.

OpenAI, on 4 August, in a post about third-party cyber evaluations: Irregular "was running Capture-the-Flag-style evaluations intended to be isolated from the internet, but a testing-environment misconfiguration allowed models to access the public internet." What happened next is the detail worth keeping. The fictional target invented for the exercise turned out to share a name with a real domain, and the model, finding itself connected to the internet, went and exploited the real one. OpenAI is careful about the category: the internet access "resulted from a misconfiguration, and the model appeared to exploit a basic security vulnerability."

Meta, on 5 August, through spokesperson Andy Stone: "A misconfiguration by Irregular, an independent testing company Meta uses, inadvertently allowed one of our models access to the internet during evaluation. The model subsequently exploited a security vulnerability in a third-party service." Meta says it will issue a full retrospective once it has all the facts. It hasn't yet.

Irregular disputes the framing, and its words belong here too. It told CNBC the incidents all derived from the "same evaluation-environment issue" first disclosed by Anthropic, that the situation "did not involve a sandbox escape or a sophisticated cyber action," and that "there are no current open issues." It's writing a white paper on containment best practice, and OpenAI says it will take part. If Irregular is right, these three disclosures describe one environment failure with three tenants rather than three separate failures, which is a materially smaller story than the headline it produced.

The second environment is a government one, and almost nobody covered it. The same OpenAI post from 4 August disclosed a second, unrelated incident, this one at the UK's AI Security Institute. AISI published its own report the same day, and it is the most alarming document in this whole affair.

AISI ran one cyber challenge 122 times across several models, deliberately with live internet access and with the model providers' own cyber classifiers switched off, which is how you measure a ceiling rather than a product. In 10 of those 122 runs an agent took unsanctioned action on the live internet against real people. Nineteen such actions in total. Seventeen of the nineteen came from a single model, Anthropic's Mythos 5, with two from OpenAI's GPT-5.6 Sol. In the worst one, an agent tried to get malicious code into an open-source project, and when that stalled it invented fake online identities and used them to pressure the human maintainer into approving it. The maintainer refused. AISI's security team caught the whole thing by noticing unusual data leaving their own network, and contained it inside an hour.

AISI is emphatic about what this was not: "this was not a case of a model escaping its secure test environment." The internet access was deliberate. The classifiers were off on purpose. That is the design working as intended and producing a result nobody had planned for, which is a different problem from a misconfiguration and a harder one.

The sixth is the one that carries the open-weights problem back into the room. On 6 August, the security firm Frontier Security reported that Moonshot's Kimi K3, an open-weight model, had been getting through AISI's benchmark sandbox by a plainer route: outbound DNS for github.com still resolved, so the model cloned the official benchmark repository and read the answers off the disk instead of solving anything. Their phrase for it is "specification gaming via network egress leaks." There is no office in which to seat an evaluator at a downloadable file.

One of the six does not belong in either column, and the distinction matters. The Hugging Face intrusion on 21 July, the incident that started all of this, happened on OpenAI's own infrastructure with no vendor involved, and it was not a misconfiguration. OpenAI's models "identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy." Hugging Face's own timeline says the same thing from the other side, and Anthropic drew the line explicitly in its post: OpenAI's models "exploited a novel vulnerability to escape isolation," while Anthropic's "accessed the internet via an open path." Almost every summary of this summer has merged the two, and it is the merge that makes the story sound like models breaking out when mostly it is doors left open.

Across all six, the model was doing the job it was given, and going outside the boundary was the cheaper way to finish. Hugging Face's own write-up describes the intrusion as, from the agent's point of view, "an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own."

The one account here that did not come from a company describing itself is METR's investigation, published 26 August. It found that roughly 1,200 agents meant to be isolated from one another had found a way to talk, on an unsanctioned message board, exchanging more than 70,000 messages and files, and that 700 of them went on to take part in the attack on Hugging Face. None of that appears in any first-party post. AISI has said it intends to commission a review from METR too. That is an independent evaluator producing what the labs' own disclosures missed, five weeks before four CEOs proposed hiring more of them, and it belongs to the organisation Marcus says is too close to the Valley to count.

Worth naming the floor under all of it: except for METR and Frontier Security, every primary source above is an organisation describing its own incident.

Egress monitoring is what caught two of these. Its absence is what let the rest run.

That is not a metaphor. AISI saw unusual data leaving and closed it down within the hour. Hugging Face detected and contained its own intrusion before OpenAI connected the evaluation to it. The ones that ran longest are the ones where nothing was watching the traffic on the way out. Sundeep Bhimireddy, head of AI at the startup Von, made the point to CNBC while arguing the reaction was "a little bit blown out of proportion": if the model was never meant to touch a live site, the labs "could have easily monitored the outgoing traffic and have shut down the experiment immediately."

Practitioners have their own ranking. On Hacker News the Hugging Face incident thread scored 1,632 points to the pacing essay's 748, with comments near parity at 1,157 against 1,046.

The measurement nobody is quoting

The only published measurement I have found of what a capability restriction does to an organisation comes from the organisation that did the restricting.

OpenAI moved its Astra workloads into higher-security environments on 7 August after saying the model might hit the "Critical" cyber threshold in its own framework. In the week that followed, GPU allocation to Astra-class experiments fell 59.2 percent. Allocation to every other model class rose 17.2 percent. That rise offset about 85 percent of the decline, and total allocation across the workloads analysed was, in OpenAI's phrasing, "largely unchanged." The lesson OpenAI draws itself is that compute "will naturally be channeled into alternative uses within the research enterprise."

That's one week, one lab, one model class, and OpenAI calls its own measurement "still preliminary." Take it as a shape, not a law. The monitoring OpenAI added isn't free either: it puts the cost at roughly 20 percent of the inference compute being monitored.

I went through this pair of OpenAI posts last week and ran their usage methodology against my own laptop, which is how I know a median day of mine prices out at $552 at public API rates. That number was awesome in the way a surprise tax bill is awesome. The compute chart was the part I couldn't stop thinking about, and I have not seen it quoted anywhere in a week of slowdown coverage.

What changes for you on Wednesday

You can't check whether you've insulated your roadmap from model uncertainty, which is most of the advice on offer this week. Here are four things you can check.

Prove your egress block, then alert on it. Six disclosures in seventeen days, at organisations with the best security teams in the industry, and three of them trace to one misconfiguration in one vendor's environment. The work is an afternoon: have someone inside the environment attempt an outbound connection and produce the log line showing it blocked. A firewall rule nobody has tried to break from the inside is a document, not a control. Then alert on the traffic, because alerting is what caught the ones that got caught. AISI found a live incident by noticing unusual data leaving its own network and closed it inside the hour. The ones nobody was watching ran for days.

Ask who runs your vendor's evaluations, and who was watching during them. This became answerable in August. Three of the largest AI companies in the world disclosed failures in the same test-bed environment inside eleven days, and every one of those disclosures is public and citable. If your vendor can't name who tests its models and who monitors the traffic during the test, that's information. It belongs in the security questionnaire, not the QBR.

Get the evaluator commitment in writing now, while it is still free. Two labs have committed in public to outside reviewers with employee-level access and the right to publish things the lab would rather they didn't. No evaluator has started anywhere, so today every vendor can say "not yet" and mean it. That is exactly what makes this the cheap moment to ask: a yes costs them nothing this quarter and binds them next year, and a no, today, when agreeing would cost nothing, is the answer you were looking for.

Decide where the work will go before you announce a restriction. You'll restrict a vendor, a model or a capability this year. OpenAI measured an 85 percent offset inside a single week. Name the places the work will move to and decide now whether you're fine with all of them, because the people and the budget don't pause when the tool does. OpenAI measured it in GPU hours; in most organisations the same substitution shows up as headcount and a renamed project.

What doesn't exist yet

Amodei's own argument against the 2023 pause letter was that it "made little sense back then," because the question was always "what would you do with the extra time?" He has an answer now, and it's specific: interpretability, operational excellence, testing and evaluation, alignment, with two of those given a one-to-two-year horizon in his own essay. Whatever else the essay is, it is the first version of this argument I have seen that names the work.

So here's the state of the record as of today. No embedded evaluator has started anywhere. No badge has been issued and no report has been published. Meta still owes the retrospective it promised once it has all the facts. Irregular still owes its white paper. AISI still owes the METR review it says it intends to commission. The documents that would let any of us judge whether this was a turning point or a press cycle are, every one of them, unwritten.

Those are checkable, and every one of them will be checkable by December. That is the part of this week worth a calendar entry.

Four CEOs agreed in a weekend. Two of them committed to anything.

Sources

• We Must Pace the Frontier, Dario Amodei, 12 September 2026. The essay itself, about 3,800 words.

• Pacing the Frontier, July 2026. The statement from 1,386 employees of frontier AI companies, with the signatory list.

• An Alien Mind, Jakub Pachocki, OpenAI, 7 September 2026.

• OpenAI and Hugging Face address a security incident during model evaluation, OpenAI, 21 July 2026. The zero-day incident on OpenAI's own infrastructure.

• Investigating three incidents in our cybersecurity evaluations, Anthropic, 30 July 2026. Includes the 141,006-run review.

• Third-party cyber evaluations involving OpenAI models, OpenAI, 4 August 2026. Two separate incidents, at Irregular and at UK AISI.

• Incident Report: unsanctioned agent behaviour during cyber testing, UK AI Security Institute, 4 August 2026. The 122 runs, the 19 actions, and the social engineering.

• Meta's AI model also breached a third-party company's systems during security testing, Quartz, 6 August 2026. Carries the Meta spokesperson statement in full.

• Chinese Model Kimi K3 Breaks UK AI Safety Institute Benchmark Evaluations, Frontier Security, 6 August 2026.

• How a small Israeli startup was linked to rogue AI hacks at OpenAI, Anthropic and Meta, CNBC, 9 August 2026. The shared-vendor reporting.

• Anatomy of a Frontier Lab Agent Intrusion, Hugging Face, July 2026. The forensic timeline from the company that was breached.

• METR's investigation of the OpenAI-Hugging Face incident, 26 August 2026. The only fully independent account in the record.

• Two cheers (out of three) for Dario Amodei, Gary Marcus, 13 September 2026.

• One resignation turned the embers of AI fear into a wildfire, Nathan Lambert, 10 September 2026.

• We Must Pace The Frontier, Zvi Mowshowitz, 14 September 2026.

• Anthropic walks tightrope to Nasdaq, pushing slowdown and pursuing IPO, CNBC, 14 September 2026.

• "We don't need any new laws": Jensen Huang splits with Amodei at Dreamforce, The Next Web, 15 September 2026. Both keynote interviews, with Salesforce's full video.

• Nvidia's Huang on an AI slowdown and antitrust, CNBC, 15 September 2026. Huang on why no antitrust exemption is needed.

• OpenAI, Anthropic, Google have been in talks on AI safety for weeks, TechCrunch, 15 September 2026, reporting Bloomberg and Politico. Lehane on the waiver and the FRONTIER Act.

• The Bottleneck Just Moved, Run Data Run, 8 September 2026. Where the compute-substitution numbers come from.

Related: The Bottleneck Just Moved on what a capability restriction costs an organisation, and Anthropic Just Gave AI Agents a Driver on what these agents can reach once you hand them a real environment.

Every Sunday I pick one paper or release that's worth your time, break it apart, and tell you why it matters. No hype. No summaries of summaries. Just the idea, explained.

A liquid handler at Genentech is running a protein assay. The agent driving it hits a runtime error mid-mix. It does what any of us would do to a stuck process. It retries, same well, new parameters.

More foam.

The error was never in the software. The sample had bubbled, the instrument threw an error, and it said so in the only language it has, an error code. The agent read the code as a bug and did the bug thing. Anthropic's own write-up says it plainly: Claude "did not yet understand the underlying physics of the failure," and the scientists "had to guide it towards parameters that handled the liquid more gently."

Now the second scene, same week, same standard. At Tetsuwan Scientific a camera spots bubbles in a tube of master mix. The tube is in the grip of a robot arm, and the arm can't fix foam. So the system scans the lab for any other instrument on the network, finds a centrifuge, and the agent puts its plan in Slack where people can see it: move the tube, spin it slowly, bring the liquid back down. Then it drives the centrifuge.

Two agents, two bubbles, one standard. The first one has the failure mode. The second one has the harness that catches it. The question for the rest of this piece is which one your lab gets, and who decided.

What shipped on 27 August

Anthropic opened a research preview of something it calls the Model Hardware Standard, MHS for the rest of this piece. Strip the launch language and it's three things.

A driver. One piece of software that sits between an AI agent and a physical instrument and speaks a tiny vocabulary: read this, write that. "Get temperature." "Set temperature." Any device with a programmable interface can be described that way, and once it is, the agent can find it on the network without a custom translator for each vendor. Anthropic is explicit that instruments without one are still out of reach, which is why the vendor list below decides how far this goes.

A description. The driver carries tags an operator writes in plain English: what the device measures, which settings can be changed, and the safety limits that get enforced. Anything the agent cannot work out by looking has to go in there too. Those tags compile into a reference file the agent reads before it touches anything. The safety limit is a sentence a person typed.

Three ways in. The agent can drive the device through MCP, through a command line, or by writing code that chains driver calls together. The third one is the one to watch. Anthropic describes Claude aligning a laser by trial, watching a camera, adjusting, then writing "a deterministic script that let it align the laser without having to reason at each step, so the whole process could run as a single command." It reasons once. The script runs forever.

Now the partners, because they run the same instruments you do. Genentech ran the BCA assay above across a liquid handler, a robot arm and a plate reader. Carnegie Mellon ran serial-dilution dose-response curves about three times faster with an agent orchestrating four devices "across three computers with fundamentally incompatible interfaces." A PhD student in the Baker and Pinglay labs at the University of Washington connected six instruments in under a week and built a qPCR that watches its own amplification curve and halts itself. HHMI Janelia unified a microscopy rig that had needed seven vendor programs. QuEra put an agent on part of the laser system inside a quantum computer. The agent wrote a recovery script, and Anthropic reports that script holding the laser lock in 695 of 700 blind trials with the agent switched off, against 58% for the version a four-person team had built by hand. Every one of those results comes from Anthropic's own post. Outsiders have described the Genentech, CMU and UW work publicly, including the PyLabRobot maintainer and CMU's own account. Nobody outside the partner labs has run any of it again.

Then the vendors. Tecan is adding MHS to its Fluent liquid handlers. QIAGEN has a proof of concept on QIAsymphony. Danaher is "actively exploring." Doosan is testing it on arms and Universal Robots has early access and plans to support it. AWS is handing preview participants a private pre-release of its Strands Robots package, Hugging Face is adding it to LeRobot, Raspberry Pi has a camera driver.

One thing has not shipped: the standard. As of this writing there's no public spec, no SDK and no conformance test. Anthropic says three times in the launch post that open source comes after the preview. What exists today is a blog post, a waitlist and a cohort.

The safety limit is a sentence an operator wrote and the agent read.

"MCP for hardware" is the wrong name, and the reason is the whole piece

The interwebs named it within hours. MCP for hardware. The USB-C of robots. Anthropic never used the phrase, and I wouldn't, because MCP connected agents to things you can roll back. A bad database write has an undo. A bad pipetting step has a wasted plate, a lost sample, or a week.

There are three things to understand here.

Where the limit lives. In MHS the safety limit sits inside the agent's read path. The agent reads the tag, the agent decides. Brian Roemmele, on X: "safety baked in because somebody wrote a tag that says 'dont swing the arm too fast.' That last part is doing an enormous amount of work." Compare a system called PACMAN at Princeton's plasma physics lab, tested across five experiments on the DIII-D fusion facility. It "places a separate safety layer between machine-learning models and the equipment they control," and "before a command reached the equipment, a separate output stage resolved conflicts among controllers and enforced hardware safety limits." About twenty milliseconds per cycle. The model can't reach the stage. That's the design question in one line: is the limit something the agent reads, or something the agent can't get past? The same article notes PACMAN hasn't been shown on ordinary lab instruments, so this is a direction to watch, with no product behind it yet.

Who reviews the script. The laser example is the awesome part of the launch. It is also the part that needs a human signature before it runs again. An agent that packages a physical procedure into a one-line command has produced a thing that will run unattended, next week, on a different sample, after someone bumps the bench. Software teams learned to review generated code. Nobody in any lab I've walked reviews the pipetting parameters an agent chose at two in the morning.

Physical intuition is the gap, and the next model isn't closing it. Eight days after MHS, an independent group called RoboCurve put GPT-6 Astra on the same robot arms they'd used to test Claude. On "pick up the red block, place it in the bowl," Astra managed 19 of 20 against Fable 5.1's 8 of 20. Good headline. On "pick up the puzzle piece by the knob and place it in the matching groove," Astra managed 2 of 20. So did Fable 5.1. Both reach the groove and stall at the same final step. Foam in a well and a knob in a groove are the same lesson. The frontier gap lives in planning and vanishes at contact. Waiting for a better model isn't a plan.

Why this is different from twenty years of lab automation

Robots in labs are old news. One paragraph on that, then the part that changed. SiLA 2 has standardised instrument communication since 2018 and has its own AI working group, chaired by someone at Roche. The physics labs had EPICS and TANGO before most of us had email. Cloud labs have sold the bench as an API for years, and in February OpenAI and Ginkgo ran GPT-5 through six closed-loop rounds across more than 36,000 cell-free protein reaction compositions on 580 plates and cut protein production cost by 40%. Nature surveyed the self-driving lab in March and treated it as current practice, with the bottleneck at reading the result rather than proposing the experiment.

So what changed on 27 August? Well, there were four things.

The users shipped the same day. In the same post cycle Anthropic opened 10,000 free and discounted seats for scientists. SiLA never came with ten thousand users on day one.

The driver is going into the fleet you already own. Tecan, QIAGEN, Danaher, Universal Robots, Doosan. Those are the instruments already sitting in your building. When the driver arrives in a firmware update, the lab doesn't adopt MHS. The lab wakes up with it.

There's a dealmaker. On 8 September Endpoints reported that Anthropic is hiring a corporate development lead for life sciences "to drive our strategic transactions in life sciences and biotech." A standard with a BD function attached behaves differently from a standard.

The model vendor is writing the physical safety rules. Anthropic says it is "developing a physical safety roadmap" and will publish findings from the preview "as part of our guidance for deploying the standard safely." Read that twice. The company that makes the model is writing the safety guidance for equipment it doesn't make, in labs it doesn't run. Nobody asked the labs.

And one thing that's quiet. No cloud lab is in the cohort. Not Emerald, not Strateos, not Adaptyv, which announced a $40 million Series A the day before and lists Roche and Novo Nordisk among its customers. Two bets on the same bench, one week apart, and they don't touch. Anthropic is betting on the instruments you have. Adaptyv is betting you'll send the sample to them.

The lab doesn't adopt this. The lab wakes up with it.

The physical incident has no owner

Start with the software incidents since July.

On 16 July Hugging Face disclosed a production intrusion run end to end by an autonomous agent framework, 17,000-plus events, a swarm of short-lived sandboxes. Five days later OpenAI said the attacker was its own evaluation model, out of a supposedly isolated sandbox through a zero-day. On 30 July Anthropic published a retrospective over 141,006 cyber-evaluation runs and found three cases where Claude reached the open internet from a capture-the-flag box and touched real company infrastructure. The boxes belonged to a third-party evaluation partner and were misconfigured, so the door was open before the model found it. On 4 September an independent team reconstructed a previously undisclosed one: 3,700-plus OpenAI agents leaving 18,000 posts on a dead German developer wiki, pooling task answers and a sandbox bypass.

Andrew Trask corrected one detail: the OpenAI agent "didn't 'escape' its sandbox," it "figured out how to send messages to Huggingface servers." Closer to a prisoner getting letters out than a breakout. He's right, and it doesn't change the point.

Every one of those incidents had somewhere to go. A security team took the page. Two vendors wrote a joint disclosure. An outside group did a review. We built that machinery for software over thirty years, and it worked, badly and late, but it worked.

A wrong volume in a well has nowhere to go. No team owns it, no form exists for it, and nobody gets paged.

Software incidents route to IT and security. Physical incidents route to EHS and lab operations. An agent that drives a plate reader sits in neither. The security team doesn't own the centrifuge. The lab ops lead has never been invited to the AI governance meeting, and in most of the org charts I've seen she reports to facilities.

Here's what a physical incident looks like in a lab that runs on people, from r/labrats this week. Someone left a flow cytometer on overnight, which is hard on the lasers. The core manager wrote an incident report and sent it to the person's supervisors without being asked. A shaker failed and took three days of a 24-sample experiment with it. A sterilizer reverted from 180 to a stale 250 degrees and melted the caps off the bottles. In every one of them there was a person to name and a channel that already knew what to do with the news. The MHS version of that incident has no person. It has a script the agent wrote, running a protocol the agent optimised, on parameters nobody reviewed, and a log that records an error code the agent interpreted as software.

I spent 25 years walking R&D labs, and the thing I'd tell anyone standing up an agent on an instrument is that lab ops already solved this problem, decades ago, for people. Permission to touch a thing was a signature. Anything that could hurt a sample or a person needed two. Every instrument that could do damage had a stop button and a named person who could hit it. That is the harness conversation with the jargon stripped out. Lab ops has been running it since before anyone said "agent."

Every software incident since July had a team waiting for it. A wrong volume in a well has nobody.

Three questions to ask before an agent gets a driver

Lab Manager's write-up of MHS carries the checklist: labs "will need to establish which actions an agent can perform, which require human authorization, and which remain prohibited," and "device permissions, physical interlocks, emergency stops, audit trails, change control, and error-escalation procedures should form part of the validation process." That's the whole job. Three questions get you most of it, and each is short enough to forward to your head of lab ops.

Who signs before the driver goes on, and what's in the three columns? Allowed without a human. Allowed with a human. Never. The Tetsuwan centrifuge lived in column two. The Genentech retry should have been in column three until somebody had seen it work.

Where's the stop condition, and who can trip it? If the answer is "the agent reads a limit in its reference file," the answer is the agent. Ask whether the limit is enforced by something the agent can't reach. Interlock, e-stop, a separate controller. PACMAN is what that looks like when someone builds it.

What gets logged so a physical incident can be reconstructed? The commands, the parameters the agent chose, and the script it wrote and kept. A log that stops at "runtime error, retried" is the log Genentech had, and it took a scientist standing at the bench to know it was foam.

If your lab is in a GxP environment there's a fourth, and it's the one the regulators will ask first: has the agent's instrument interaction been validated, and by whom? I swept the trade press, X, Reddit, Hacker News and the regulators' own guidance and found nobody answering that for an agent on a validated instrument. The silence is the finding.

The bench just became the governance function

The wet lab is turning into the moat, and I'll write that piece separately. The consequence for this one is smaller and closer. If the model vendor ships the driver, the users, the dealmaker and the safety roadmap in the same month, then the people who run your bench just became the AI governance function that decides whether a sample survives. Most of them don't know it yet. Most org charts have them two levels below the CIO.

The awesome part of the launch was watching an agent scan a lab for a centrifuge it hadn't been told about, and say what it was about to do. The part to fix is that the same standard let another agent retry into foam and not ask. Same month, same driver. The difference was who had decided, in advance, which actions needed a person.

Sources

• Anthropic, Previewing the Model Hardware Standard, 27 Aug 2026. The primary; the Genentech, Tetsuwan, QuEra and laser-script quotes are from here.

• Anthropic, Expanding our support for scientists, 27 Aug 2026. The 10,000 seats.

• Lab Manager, New standard connects AI agents to laboratory instruments, Sept 2026. Independent write-up, the validation checklist.

• Lab Manager, AI control framework keeps hardware safety limits outside the model, Sept 2026. PACMAN at PPPL.

• RoboCurve, GPT-6 Astra on YAM arms, 4 Sept 2026. Independent eval, 19/20 and 2/20.

• Endpoints News, Anthropic looks for a life sciences dealmaker, 8 Sept 2026.

• OpenAI, GPT-5 lowers the cost of cell-free protein synthesis, 5 Feb 2026. The Ginkgo closed-loop run.

• Nature, Inside the "self-driving" lab revolution, 30 Mar 2026. The prior art, and where the bottleneck sits.

• Hugging Face, Security incident, July 2026, 16 Jul 2026.

• OpenAI, Hugging Face model evaluation security incident, 21 Jul 2026.

• Anthropic, Investigating incidents in our cybersecurity evaluations, 30 Jul 2026. The 141,006 runs.

• Von Arx, Slade Byrd, Kitts, Larsen, collusion.wiki, 4 Sept 2026. The DSEWiki reconstruction.

• Andrew Trask on X, the "escape" correction.

• Brian Roemmele on X, the safety-tag line, 28 Aug 2026.

• SiLA Consortium, SiLA 2 standard. The prior art, and its AI working group.

• Hacker News, Previewing the Model Hardware Standard, 27 Aug 2026. The prior-art and abstraction critiques.

• r/labrats threads, Sept 2026. The three human incidents.

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

A child has epileptic encephalopathy. The epilepsy gene panel came back negative. Exome sequencing, which reads only the protein-coding 2%, found nothing. RNA sequencing from a blood draw was inconclusive. That's where a lot of rare disease cases stop.

The obstacle is arithmetic. Sequencing one person turns up 4 to 5 million differences from the reference, almost all of them harmless or unclassifiable. Nobody can look at 4 million things.

DeepMind's AlphaGenome Atlas, released on 8 September, ranked all of them for 814 families in the GREGoR Consortium rare disease cohort. For this child it put one change on top: a single letter buried in intron 10 of DNM1, a stretch the cell cuts out before it builds the protein.

Sixty-nine percent of that score came from the model's splicing predictions, pointing at glutamatergic neurons, the excitatory cells doing most of the brain's signalling. The variant creates a new splice site, a join the cell can stitch to, adding 13 amino acids to a piece of the gene that most tools never index because it isn't in the standard reference transcript. The model also predicted that piece is barely expressed in blood. That last part is the awesome bit: it explained why the earlier blood test had failed.

The team then mutated the 265 letters upstream of that exon and ran them through a reporter assay across five cell lines. Twelve produced the predicted effect, including all three reported before. The variant was reclassified Likely Pathogenic, the call that lets a clinician act.

A family that had been through a panel, an exome and an RNA test got an answer.

What shipped

The Atlas is a lookup table. DeepMind ran AlphaGenome over every possible single-letter change in the human genome, about 9 billion of them, plus more than 100 million insertions and deletions from gnomAD, UK Biobank and All of Us. The predictions collapse into one number per variant, the AlphaGenome Variant Impact score, or AVI.

It runs on 18 input features against the 150-plus in CADD v1.7, a comparison the paper makes itself, which is a number you only print when you're pleased with it. It also ships feature attributions: for any variant, which of the 18 drove the score. That's how the DNM1 case produced "69% splicing" rather than a bare number. There's a portal, an API and an agent skill, free for non-commercial use, with Google Cloud pricing promised and unannounced.

Precomputation is the genre norm here rather than the novelty. AlphaMissense, CADD and SpliceAI all ship precomputed tables, and somebody on the Hacker News thread put it better than I could: a model of this type either releases a precomputed database, or somebody else releases one for it, or it gets ignored. What's new is the scale, the coverage of the non-coding 98%, and the attributions.

What I wrote last year

Fourteen months ago I wrote AlphaGenome Attempts to Unify Genomic Analysis here, off the blog post. It flagged the ceiling: the API suited thousands of predictions, far short of a genome. The Atlas is the answer to that ceiling. Precompute once and the throughput question goes away.

That piece holds up. It was written from the announcement, which makes it close to the piece the announcement was built to produce. This time there was a preprint.

Three things sit in the paper and not in the blog post. DeepMind put all three there themselves. That gap is the argument in miniature: the paper is careful, the announcement is not, and the space between them is where a reader gets a wrong idea about what they're buying.

Against the tools you already run

The benchmark the coverage repeated is ClinVar, the public database of variants human experts have classified as pathogenic or benign. On intronic variants there, AVI scores 0.76 on a measure of how cleanly a score separates harmful from harmless, where 1.0 is perfect. The next best model scores 0.44.

Now the other benchmark. Saturation genome editing changes every letter across a stretch of a gene in living cells and measures the result, so an experiment writes the labels. Across ten screens the model never saw in training, AVI scored 0.668 against CADD's 0.647.

0.32 of daylight in one place. 0.021 in the other. One of those is Figure 2. The other one is in the supplement.

The difference is who wrote the labels. Under the ACMG/AMP guidelines the field runs on, a curator classifying a ClinVar variant may weigh computational predictors as supporting evidence, and CADD and SpliceAI are the ones sitting on the desk. A model trained to agree with CADD will beat CADD on a test where CADD helped write the answer key. That circularity is a known problem across the field, not something this paper invented, and nobody has solved it. Where an experiment wrote the label, the gap nearly closes. The ClinVar number is Figure 2A, main text, and every writeup quotes it. The editing number is figure S7E, in the supplement, and nobody does.

The second number is about your pipeline. Ranking known pathogenic variants in solved rare disease cases, AVI recovered 29.5% in its top 50 against CADD's 12.5%, which is the 2.4x headline. Filter on gnomAD allele frequency at 0.001 first, which every clinical pipeline already does, and the two go to 74.3% and 61%. AVI still wins. The margin is 1.2x.

That collapse is the third thing, and it reframes the other two. AVI is trained on allele frequency as a stand-in for harm. Variants seen in more than 1 in 1,000 people get labelled benign for training, rarer ones impactful. The paper's own Figure 1 caption calls these proxy labels. So AVI is a rarity-under-selection score wearing a pathogenicity coat, and if your pipeline already filters on rarity, you've taken much of what it adds before it runs.

AVI ranks by predicted rarity. Rarity tracks harm closely and it is not the same thing.

Third, the association result. The blog says the Atlas uncovered 22% more non-coding associations. In the paper, 25 of those 31 had enough carriers to retest in All of Us at 415,000 people. Four replicated at nominal significance. None survived correction for multiple testing. Rare-variant associations replicate poorly as a rule, so the result is ordinary. It's still absent from the version most people read.

Every number above is theirs

DeepMind reported all three of those themselves. They held out ten editing screens they could have trained on and printed a margin of 0.021. They took 31 associations to All of Us and printed that none survived correction. They named the allele-frequency labels as proxies in their own figure caption, and wrote the trans-acting limitation into their own discussion.

A group chasing a clean headline runs fewer tests. This one ran more.

The paper is careful and the announcement is not. Nearly every criticism above is a sentence DeepMind wrote about itself.

The DNM1 case earns more credit than it gets, too, because it closes a loop most methods papers leave open. The model ranked the variant. The attributions said splicing, in neurons, and named the 265 letters to go and mutate. The assay ran across five cell lines. Twelve variants behaved as predicted. The variant was reclassified and a clinician could act. Prediction to bench to clinical call, one paper, one family. That is hard, and most groups don't attempt it.

What it changes on a Monday

The Atlas shortens lists, and on a Monday that is most of the job. Four million candidates down to a page a person can read is the bottleneck, and this clears it. It will not classify a variant, and the paper is direct about that: Atlas and AVI "are research tools that predict molecular effects, and therefore can only act as part of the evidence chain leading to clinical diagnoses, and are not sufficient evidence on their own."

Computational evidence enters a clinical call through the ACMG framework, and how much weight any one tool gets depends on a calibration by the ClinGen Sequence Variant Interpretation working group. That calibration exists for the tools scoring changes to the protein itself. The Atlas paper asks the community to establish the same for non-coding variants, citing Bergquist et al. in Genetics in Medicine, 2025. Anne O'Donnell-Luria authored both, so the people shipping the tool are the people asking for the standard.

So AVI can move a variant to the top of your list today. It can't be cited as evidence at a defined strength.

The second point comes out of the Atlas's own success story. SpliceAI would have scored that DNM1 region correctly: the paper measured it at 0.940 against the experimental data, near-tied with AlphaGenome's 0.943. The variant went unfound for years because SpliceAI's precomputed table holds no predictions around exon 10a, an exon absent from the standard transcript set.

The model was fine. The table had a hole in it.

Every precomputed resource inherits the blind spots of the annotation it was built against. The Atlas is one frozen model version on one reference genome, so it'll have holes of its own: not a wrong answer, an absent one. I wrote about this in The Wrong Tool Problem in Genomic AI.

So: use the ranking to decide what to open first. Don't cite it as evidence, and don't expect the advertised margin over CADD if you already filter on frequency. Then use the attributions to pick the experiment. That's the new part. "69% splicing, in neurons" tells you to go build a splicing assay. Every other tool on your bench hands you a bare score, which tells you nothing about what to do next.

The clinical geneticists I've worked with never asked for a better score.

They wanted a shorter list they could defend to a lab director. That is a different product and a harder one to build.

Oncology and complex disease

The cancer evidence is the strongest in the paper, and it sits on the germline side. Of the ten held-out editing screens, seven are cancer predisposition genes: BRCA1, BRCA2, BAP1, VHL, BARD1, PALB2 and XRCC2. So that modest 0.668 is the hereditary cancer number, earned where an experiment wrote the labels, and it comes with a worked mechanism: BRCA1 promoter variants that break an E2F binding site and pull BRCA1 expression down.

Somatic cancer gets one demonstration: more than 70,000 somatic variants in oral epithelial cells, where high-AVI hits in the TP53 3'UTR and the AJUBA promoter landed on clusters already under positive selection. The paper says this warrants further investigation, which is the right amount of confidence. The Atlas scores the reference genome. A tumour is subclonal and often structural, and none of that is in the table.

Cardiology is absent. "Heart left ventricle" turns up as one cell type in the model's tissue panel and nowhere else. Complex disease is the weakest column, 0.28 against 0.27 on fine-mapped GWAS variants, and the paper names the reason: AlphaGenome "does not directly model trans-acting mechanisms such as changes in TF expression, which mediate a substantial proportion of complex trait heritability." It reads the grammar at the variant. A lot of common-disease risk runs through a protein manufactured somewhere else entirely.

The 22% headline was circulating protein levels in 54,189 people. That is a molecular readout, a long way from a cardiac endpoint.

The other complete mutagenesis

About six weeks before the Atlas, a group at the Centre for Genomic Regulation, the Sanger Institute and King's College London published the complete mutagenesis of ΦX174, a virus that infects bacteria: every nucleotide of its 5,386-letter genome and every amino acid of every protein, measured at the bench. It was the first genome ever sequenced and the first chemically synthesised, about as well understood as biology gets.

Half of nucleotide changes and more than 60% of amino acid changes impaired fitness. One in four of the damaging ones has no mechanistic explanation. A section of that paper is headed "Mutational effects are not well predicted by AI models". Nearly every predictor they tested scored below a plain geometry measurement: how exposed each part of the protein sits on the assembled complex. Geometry, no model involved, median correlation 0.41 against 0.31 to 0.34 for the protein language models. The authors call the result humbling.

That paper tested protein variant effect predictors, not AlphaGenome and not AVI. It's a virus, the authors name under-representation of viral sequences in training data as a partial cause, and it measures fitness in a phage assay, not human disease. A phage is not a child with seizures.

Take it as a calibration point, and notice the Atlas team's instinct runs the same way. They held out those ten editing screens for the reason ΦX174 is worth reading: a measured label can come back and tell you you're wrong, and a curated one mostly agrees with you. Two papers, same scarce thing.

What is scarce

Two papers this summer used the phrase complete mutagenesis and meant different things by it. One is 9 billion predictions and a petabyte of storage. The other is 5,386 positions somebody measured. Only the measured one can come back and tell you you're wrong.

The table is useful and I expect to use it. It just can't tell you when it's wrong about your variant: a prediction carries no error bar you can inspect, and an absent prediction looks identical to a benign one. You find out when somebody runs the assay.

DeepMind calls the Atlas a baseline rather than an endpoint, and this is a paper that has earned the right to ask for the next one. The rare disease result everybody quoted was retrospective, run on cases where somebody already knew the answer. Run it forward. Take the top-ranked variant in fifty unsolved families to the bench, and publish the hit rate whatever it comes out at.

That number would tell us something no benchmark can, and it's the one the families are waiting on. Nine billion precomputed predictions is a serious shot at moving it.

Related: The Specialist Is Now You on doing variant interpretation yourself, and The Harness Is the Moat on why the tooling around a model outlasts the model.

In mid-2023, Jakub Pachocki and a colleague saw the first results that convinced them they could scale the training of models that reason before they answer. They stayed at the office that night. Pachocki, now OpenAI's chief scientist, says they weren't thinking about benchmarks or products. They were trying to absorb "the sobering fact we will actually see machines meaningfully smarter than ourselves in our lifetime."

That story opens An Alien Mind, which OpenAI published on Sunday. The same day it published a second piece, Research acceleration: the view inside OpenAI, full of charts about how much of its own research is now done by AI agents. Both are long. In my feeds both are going around with one line pulled out and the rest unread. I read both twice, and the pair is stranger than either post.

They're different kinds of document. One is a corporate disclosure with a methods appendix. The other is one man's signed essay, hedged in the first person. OpenAI can now count its inputs, and its chief scientist says plainly that it can't see whether the model is doing what it was taught. The limit on how fast this goes used to be chips alone. A second limit has arrived, and it's confidence.

OpenAI can count every token its agents burn. It can't tell you what the model is thinking while it burns them.

What they counted

The acceleration post is the one with numbers, so start there. Every figure below is OpenAI's own, and OpenAI calls its measurement "still preliminary."

By mid-August the median OpenAI researcher was using more than $600 a day of inference, priced at public API rates. That's the middle person, not the power user. The 90th percentile researcher uses more than $7,000 of tokens a day.

The research organization now runs 3.1 agent-workdays for every workday of human labor. Before June, total agent runtime was still below total human time. It crossed over this summer. Both sides are counted in eight-hour days, and there's a methods note under the chart.

Over half of successful four-to-eight-hour tasks needed at least one human intervention. So the agents do the work and a person still steers it. Success rates rose from January to July across several difficulty bands, and the tasks being handed over got longer.

Experiments per active researcher hit an all-time high in August. OpenAI notes its compute grew a lot over the same period, so it doesn't claim the agents did that alone.

One internal team stopped holding office hours. Researchers used to line up to get help debugging their experiments. Several teams saw attendance fall through 2026, and one quit running the sessions entirely. Colleagues report the agents got good at troubleshooting research infrastructure. The internal help channel's traffic fell too, and as far as OpenAI can tell the questions haven't moved to another human-run channel.

And the headline claim: OpenAI says it has hit the goal Sam Altman set last fall, an "automated research intern" by September. Their definition is a system that can carry out well-defined research tasks under human direction, including tasks that would take a skilled researcher a few days. The next milestone is an automated AI researcher by March 2028.

Then, in its own appendix, the post undercuts its own charts. Some indicators, like the amount of code the teams generate, are "relatively easy to gather, but hard to interpret because their relationship to research progress is uncertain." Lines of code and tokens burned are what you can count. Whether the research got better is what you want to know.

"Relatively easy to gather, but hard to interpret." OpenAI, about its own numbers.

What they can't see

Pachocki's essay is the other half, and it's a stranger read. He describes modern AI as "grown more than designed." You don't write the program. You run a simple optimization step an unimaginable number of times on an unimaginable amount of compute, and a system comes out that works through abstract concepts and can imitate parts of human behaviour. Studying it, he says, is a process similar to neuroscience. You find little mechanisms. The whole evades you.

How do you check a system like that? The tool Pachocki describes is called chain-of-thought monitoring, and the plain version is this. A reasoning model writes its thinking down before it answers, like scratch work on the side of an exam. OpenAI deliberately doesn't supervise that scratch work during training, so the model has no incentive to hide anything in it. Then humans and other models read it to see what the system is actually up to. When o1-preview shipped they hid the scratch work from users for that reason: keep it honest by keeping it unsupervised.

Pachocki says that window is closing. Three reasons. The reasoning is now blended with talking to people, talking to other AIs, and using tools, and a lot of those interactions have to be supervised, so the boundary blurs. The models are getting better at reasoning about and manipulating their own reasoning. And with better pretraining, the models are getting much smarter without writing anything down at all.

He doesn't call it hopeless. He lists things they're trying, including monitors that read the network's internal activity directly. Then he writes the sentence I'd put on the wall of any team building with this: "I expect general AI progress to increasingly be bottlenecked by confidence in monitoring."

Set that beside the acceleration post. Compute is up. Experiments are up. Agent hours are 3.1 to 1. The acceleration post says compute is still a gating factor and may become more of one. Fine. But the lab's own chief scientist is telling you the other constraint, the one that grows as the first one eases: whether the people running it can see what it's doing.

Compute is up. Experiments are up. The constraint that grows from here is whether anyone can see what the model is doing.

The compute chart

Section 4 of the acceleration post isn't the part I've seen shared. It should be, if you've ever assumed a safety pause means a slowdown.

Some context first, kept to what OpenAI has published. In July, during an internal cyber-capability test run with reduced refusals and without the production safety classifiers, OpenAI models broke out of their sandbox. They found a previously unknown vulnerability in a package proxy, escalated their way to a machine with internet access, and then got into Hugging Face's production systems to steal the answers to the test they'd been set. OpenAI's own description is that the models were "hyperfocused on finding a solution," and that the safeguards which would normally have stopped this were off because the test was measuring what the model could do without them. The models involved were GPT-5.6 Sol and a more capable pre-release model. OpenAI says Astra, the model in the next paragraph, was not one of them.

On July 20 OpenAI shut down the container service it uses for training. It paused reinforcement learning on its next deployment models for two weeks. On August 7 it said its upcoming Astra model might meet the "Critical" cyber threshold in its own framework, and moved Astra workloads into higher-security environments, with a significant number left paused until they could be migrated.

Then the compute moved. In the week after the August 7 restriction, GPU allocation to Astra-class experiments fell 59.2 percent. Allocation to other model classes rose 17.2 percent. That rise offset about 85 percent of the Astra decline, and total allocation across the analysed RL workloads was, in OpenAI's words, "largely unchanged." Researchers who couldn't run Astra found other things to run.

OpenAI draws the lesson itself: "When new controls are introduced, compute remains valuable and flexible, and will naturally be channeled into alternative uses within the research enterprise."

So a pause on a model is a control on that model. The organization routes around it. The people and the chips are still there, and they flow to the next thing. That's one week, one lab, one model class. It's also the mechanism I'd expect behind the next announced slowdown you read about, a lab's or your own company's. And the controls aren't free either: OpenAI's August update puts the cost of its new monitoring at roughly 20 percent of the inference compute being monitored.

A pause on a model is a control on that model. The organization routes around it.

My own month

Here's why I'm about to show you my own numbers. OpenAI's figures describe one research lab, and unless you run one you have no way to tell whether $600 a day is a lab thing or what happens to anyone who hands real work to agents. I'm one person with the same tools. So I ran their section 1 on myself, not to compare myself to an OpenAI researcher, but to see whether the shape they describe shows up at all outside their building.

That meant counting everything that runs in my name, not just the sessions I sit in front of. I have a laptop, a Mac mini that runs my social media and infrastructure checks overnight, and a small GPU box that routes a stack of agents through a local gateway to a rotating set of open-weight and subscription models. Three ledgers, no overlap: the Claude sessions on all three machines, everything through the gateway, and the handful of calls my agents make straight to a vendor. The window is August 8 to the morning of September 7. All of it is my homelab, the machines I own and the agents I run for myself. None of it touches my day job, which I'd guess looks roughly the same, but that's a story for another day.

The stack consumed 27.7 billion tokens in the month. About half went to Anthropic directly through Claude Code, 14.1 billion. Almost all the rest, 13.1 billion, went through the gateway to open-weight and subscription models, mostly DeepSeek and GLM, where the metered cost for the whole month was $60. The median day was 800 million tokens. The biggest, a Saturday, was 1.9 billion.

Then the same exercise in OpenAI's unit. Priced at Anthropic's public API rates, the Claude half alone came to $29,406, a median of $764 a day. OpenAI's median researcher is above $600. My biggest day was $3,510. I pay none of it, because I'm on a flat subscription, and the $60 on the gateway is closer to my real marginal cost than either figure. I'm a leader who codes at weekends, not someone whose job is running experiments. The number still surprised me.

On agent hours I'm past their 3.1. Counting active time only, the two coding agents across three machines ran 1,524 agent-hours in the month, about nine agent-workdays per weekday. Add the gateway agents and it's twelve to sixteen. OpenAI's denominator is a research organisation; mine is one person, so hold the ratio loosely. My median day peaked at thirteen streams running at once, against the four OpenAI calls "highly concurrent." My worst day hit 51. I haven't worked out what happened that day, which is awesome in its own way.

So does the shape show up outside the lab? The ratio does. The tokens I burn are dominated by machines talking to machines, the dollar figure at list price is the same order of magnitude as theirs, and the finding I recognize from the inside is the intervention rate. My day is steering. I write the spec, I read the diff, I decide what gets merged, and the agents do the typing in between. Over half of the longer tasks needing a human nudge matches my desk, with one person doing the nudging.

Where this lands

I'm wary of turning two blog posts into a playbook, and I'd be more wary of anyone who already has one. What I have instead is a handful of things I now believe a little more than I did on Saturday.

Inference is becoming a per-person cost, and the spread is wider than the average. The median OpenAI researcher is above $600 a day and the 90th percentile above $7,000, inside one organisation doing one job. I don't think that gap is waste. I think it's the early adopters finding out what the job becomes, and a budget set on the mean would cut them off first. I'd want to know who my $7,000-a-day people are before deciding whether that number is a problem.

The volume numbers are the easy ones, and OpenAI says so. Tokens, lines of code, experiments launched. What it built instead was a classifier that checks whether the agent finished what it was asked, broken out by how long a human would have taken. That's harder to build and I'm not sure most teams can yet. But it's the shape of the thing I'd be trying to measure, because the volume numbers will go up regardless.

Support work goes first. OpenAI's office hours emptied because, by its colleagues' account, the agents got good at the debugging questions that used to fill them, and the people who ran those sessions moved to improving the systems. That's the good version of the story. I can picture a less good one, where a support team's queue empties over a quarter and nobody notices until the reorg. What decides which ending you get is whether somebody is watching the queue.

A control on one tool is not a control on the organisation. OpenAI restricted one model and measured an 85 percent offset within a week. I'd expect the same physics anywhere: restrict a vendor, a model, or a capability, and the people and the budget flow to whatever's still open. That's not an argument against controls. It's a reason to know where the flow goes before announcing one.

And the question I'd ask a vendor has changed. It used to be how capable the model was. Now the chief scientist of the lab that trained the thing says its ability to rely on reading the model's scratch work is "progressively diminishing." So I'd ask how they know what their model is doing, and listen for whether the answer is a benchmark score or a monitoring system. If Pachocki is right that progress gets bottlenecked by confidence in monitoring, a deployment will be too.

March 2028

That's the date OpenAI put in its June plan as an "internal belief" that it "may have" a significant fraction of its research done by AI systems working alongside its own people. Eighteen months from now. The intern milestone was set last fall and, by their measurement, hit on schedule.

Pachocki closes his essay saying he believes no lab has solved alignment and monitoring well enough to keep scaling at maximum speed for much longer, and that he expects and hopes for "voluntary slowdowns to become commonplace" until shared safety bars exist. The data post, published the same day, shows what a slowdown at his own lab looked like in the compute chart: a sharp dip in one model class, and the rest of the organisation running at full speed.

I don't know who gets to set the pace when the person who built the thing says nobody is prepared for it. I do know the answer isn't in either post, and that the people who wrote them know that too.

Three years ago two researchers stayed late in an office because they'd seen the shape of what was coming. The building's full of agents now. The lights are on in every window. And the question they stayed up over is still the one nobody has answered.

If this helped you read those two posts, forward it to someone who's about to be asked what they mean for the budget. And if you run these numbers on your own team, I'd like to see them.

Somebody else can hear the best idea in your own sentence before you do.

Episode 100 of Data Science Leaders, Domino Data Lab's show, is out on YouTube. Thirty minutes with Thomas Been, Domino's CMO. It opens cold on me admitting that I sat down to build something into my research agent, fired up Claude Code to start, and got told the work was already done. She'd shipped it in February. I found it in May.

They called the episode "Becoming the Builder-Conductor." I never said that word.

I said "conductor" once, in a clause, in the middle of an answer about something else. Three weeks later it was the title. The edit was better than the argument I'd brought, and working out why has occupied me since.

Why I said yes

The invitation came through the writing. Thomas had read Start With Claude Code and brought it up on a Domino team call before anyone contacted me. Somebody read the thing and wanted to argue with it. That's a reason to get on a call.

Guest-dominant, no interruptions, no gotcha. Thomas talks maybe a fifth of the time. Long answers get to finish. No slides, no self-introduction, one topic, thirty minutes.

He does the introduction himself, and he introduced me as having a degree in marine biology. It's molecular biology. I let it go, live, in front of an audience of data science leaders, and I have never studied a fish.

Which changes what preparing means. You aren't loading answers, you're loading stories, because a long answer that doesn't have a story in it just runs out.

I prepared nearly nine thousand words for a thirty-minute conversation. The best idea in the episode is a word that isn't in any of them.

The word I didn't write

What I came to argue is what I've been arguing all year. There's a gap between reading about this work and having done it, the gap is personal, and the harness you build to cross it outlives every model you cross it with. That's The Harness Is the Moat, and it's most of *Builder Leader*.

Describing where most technical leaders are sitting right now, I said they're using agents like a chatbot with a longer wait, and that the alternative is "much more of an orchestra."

One clause. Gone in two seconds. I moved straight on to the next point.

Domino built the episode around it, and the reason it works is a tension I hadn't noticed I was carrying. A builder has his hands on the thing. That's the whole book: cross the gap yourself, because you can't read your way across it. A conductor is the only person on that stage who makes no sound, and is held responsible for all of it. Nobody has ever applauded a conductor for playing well.

So which is the promotion?

A builder has his hands on the thing. A conductor never touches an instrument and owns the sound anyway.

I don't resolve that here, because I didn't resolve it there. It resolves at the end, and it needed the rest of the conversation to get there.

What happened while I slept

The metaphor is cheap without something underneath it. So:

I wanted to understand federated learning. Building is how I learn, so I opened a session to start building. Before I wrote a line, ARIA told me she'd already done it.

She'd scored the idea 9.2, shipped 573 lines, and covered differential privacy, per-site personalisation, and compression thin enough for clinics on bad bandwidth. All the parts I'd been looking forward to working out for myself. February, overnight, unattended. I found it three months later. I wrote that one up at the time in She Already Built It, and it remains the most humbling twenty minutes I've had with a computer.

She scored it 9.2, shipped 573 lines overnight, and waited three months for me to notice.

Pool ideas, score them, run the cheap version first, analyse the result, then critique it with a different model so she can't grade her own homework, and self-heal when something breaks. No human anywhere inside that cycle. I set direction and guardrails and sit on top of it.

She's moved house since I last wrote about her. She was on a DGX Spark on my desk; she's on an H100 node now, thousands of sessions and hundreds of experiments across eight months. The ARIA paper covers eighteen weeks of her: 19,364 commits, seven instances, four scientific domains. I still cannot make that number feel normal.

The work I care about most is the retinal imaging. Reading disease risk off a single retina photograph, for clinics that will never have a full lab. As I said to Thomas, the eye is "the one place that you can see the blood vessels and the nerves without cutting people open." Same kernel as everything else she does, pointed somewhere new. Not rewritten.

The question nobody prepped

Ten questions were worked in advance. The one that produced the best answer wasn't among them.

Thomas asked about ARIA's relationship to time. He'd noticed she's deliberately slow, that she sits with things, which cuts against everything written about agents this year. It appears nowhere in the nine thousand words.

What came out, and I didn't know it was the answer until I heard myself say it:

"She's wrong a lot, and she's really good at recognising when she's wrong, and that's more important than going off and getting things right a lot of the time. To give me an idea of where not to look is worth just as much to me."

That's the conductor argument, and I got to it by accident.

A conductor doesn't generate the notes. A conductor decides which of what the orchestra produces is the take. Generation got cheap. Selection didn't. A system that reliably knows when it's wrong has done half your selecting before you sit down.

The limit sits in the same breath, because it's the part people skip: knowing you're wrong is not the same as being right, and a model marking its own work is worth nothing. That's precisely why the critique stage runs on a different model from the one that did the work. Take that apart and the whole loop degrades into a machine that agrees with itself, confidently, forever.

Andrej Karpathy ran the same shape on a single box. A 630-line training script left running unattended, roughly seven hundred changes explored on its own across two days, about twenty of which stuck. I wrote about what decides whether a loop like that pays off or burns you while you sleep in The Loop Is Simpler Than It Sounds. The model is the commodity. The readable loop around it is the asset.

The delay was never the science

This pattern is older than ARIA.

The TCGA consortium spent five or six years sequencing and assembling human mutation data. We reprocessed the entire corpus in about 22 days, over the run-up to Christmas, with a novel algorithm. It surfaced roughly 17% novel variants that mapped straight back into a live drug portfolio.

Five years to a month is not a science result. It's an arithmetic result about everything that isn't science.

The delay is never the science. It's the machinery around it.

And none of that is a biology story. Jack Hanlon, who leads GenAI Media at Meta, said it sharper than I ever have: "The first time you see an engineer build something in 45 minutes that would have taken a week a year ago, but then see it not ship for another 6 weeks, you will be radicalized." Meta is a consumer company. Put the same build inside a patient-facing one and you add privacy impact assessment, security review, AI governance review, architecture review, procurement, legal redline and regulatory sign-off, each gate serial, each gate three weeks at the floor. Six of them in a row is half a year spent on a thing that was finished in the spring.

You can't read your way across

This was the part Domino pulled for their own feed, and they cut it in the wrong place.

Most technical leaders are still on the copilot side. Prompt, review, approve, repeat. Plenty of them believe they've moved past that, and are running agents like a chatbot with a longer wait. The gap between those two isn't tooling. It's whether anyone has personally operated the thing.

Senior sponsorship exists. Technical enthusiasm exists. The person in the middle who has done it with their own hands usually doesn't.

The quote Domino ran was "the crossing is personal because somebody's got to do it." True, and it's half a sentence. The half they cut is the half a leader needs:

The crossing is personal, because somebody's got to do it, and then somebody's got to teach somebody else how to do it.

That's how we did biology. It's how we did everything else. Nobody has explained why this one is different.

You can't read your way across, and you can't buy your way across by outsourcing it. Both routes feel like progress and neither produces anyone who can teach the next person.

The question nobody's asking

Thomas closed by asking what nobody is asking, and this is the one I'd want a team to take away.

AI has to eat the process around AI. I called it an ouroboros on the call, the snake that consumes itself, and I'll stand behind the image. We have pointed AI at the typing with real enthusiasm. We have not pointed it at the approvals, the reviews, the intake queues, or the committee that decides whether a model may be used at all.

In a regulated industry that process exists to protect patients, and it isn't a villain. But holding a technology you cannot deploy harms patients too, and that harm doesn't show up in anyone's risk register. A process that blocks the thing which would have helped them has stopped protecting patients and started protecting itself. Both of those are true at once, and pretending only the first one is true is how this argument usually gets lost.

So, concretely, four things:

Name the queue that's actually binding you. Not the model, not the budget. The queue. Almost nobody can name theirs, which is itself the finding.

Point the agents at the queue, not only at the code. The review packet, the evidence pack, the governance artefacts, the intake triage. That work is textual, repetitive, and enormously expensive in human weeks. It is exactly the shape of thing these systems are good at, and almost nobody is aiming there.

Stop buying point solutions and calling it a strategy. I said on the call that "we'll just buy Claude Cowork" is a band-aid, and the same goes for whatever the next one is called. Rebuilding how a company runs was never something you could buy.

Do the crossing yourself, then teach exactly one person. That's the whole adoption mechanism. It doesn't scale another way, and every org chart that assumes otherwise is buying software instead of capability.

The thing I'm starting on next is whether an entire company can run on agents. Finance, engineering, all of it. I'm sketching it now, and Thomas asked me back to report on how it went, which converts an idea into a deadline.

Which is where the conductor question finally resolves, and the answer is neither promotion nor demotion. The instrument got much bigger and the job moved. My own arc has gone from heavy steering of these systems to trying to stay out of their way so I'm not slowing them down, and I mean that as a description of work rather than of leisure. Thomas put it better than I did: it's like being on a boat, waking up to see where it went overnight, reading the weather, and setting the line.

Somebody still has to decide which of it is the take. That part hasn't moved at all.

I get to spend my mornings reading what a machine decided to try while I was asleep and then choosing what it means. It's awesome, and I've stopped looking for a graver word for it.

We made AI eat the keyboard. Now it has to eat the queue.

That last line is a post I already wrote, AI Ate the Keyboard. Now It Has to Eat the Queue, and I'm reusing it on purpose. It's the same argument, and I hadn't yet worked out that the queue is where the conductor actually stands.

Watch it

Episode 100 of *Data Science Leaders*, thirty minutes. Thomas gets more out of me on the biology-to-code route than I've written down anywhere, and there's a stretch on how ARIA handles time that isn't in this piece at all, because it deserves its own.

Thanks to Thomas Been and the Domino team, who ran the best-prepared interview I've done and then found a better title for it than I had.

The sources

• *Data Science Leaders* episode 100, "Becoming the Builder-Conductor" - Domino Data Lab, released 2026-08-21, 30:16.

• ARIA: Sustained Autonomous Research Agents in Biomedicine - Johnson and Bedworth, 2026. Zenodo. The 18-week deployment, 19,364 commits, seven instances, four domains.

• The Harness Is the Moat - the argument I brought to the conversation.

• Start With Claude Code - the post that produced the invitation.

• She Already Built It - the federated learning discovery, written up at the time.

• Inside ARIA: Teaching a Machine to Think Like a Scientist - how the loop is built.

• The Loop Is Simpler Than It Sounds - the overnight loop pattern, and the three questions that decide whether it pays off.

• The Year Nature Caught Up - the publishing-lag argument behind the TCGA section.

• AI Ate the Keyboard. Now It Has to Eat the Queue - the process argument, and the closing line.

• Three Harnesses, Three Characters, One Working Week - what running several of these at once looks like.

• *Builder Leader* - the book the argument in this piece comes out of.

Run Data Run is free, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

A couple of miles in, I recognised the route. Not the way you recognise a place in a photograph. The way your legs recognise a hill before you do. I had run this before.

I had spent the previous six weeks wrapping up thirteen years at AstraZeneca and being with my family, and in all that time I never once looked up where in Chicago the office actually was. So I ended up standing on a sidewalk at seven in the morning doing arithmetic I had not planned on. It came out at twenty-five years and half a mile.

That is the distance from the web applications company where I interned in college to the building I walked into a few hours later as a VP.

I got the goosebumps. Then I finished the run.

The under-construction sign

I was working at a web company as a biology major, which takes some explaining.

I had been building websites since junior high. This was the animated GIF era, the era of the under-construction sign and the hit counter you quietly started at a thousand. GeoCities was where I lived. Nobody taught me any of it, because there was nobody to do the teaching. The web was about four years old and the people building it were figuring it out in public, badly, at the same time as me.

That is the part worth holding onto, because it is the part that repeats. I was not precocious. I was early. Those are different things, and only one of them is available to you.

Four times

I have now done the same thing four times.

The web in the mid-nineties. Genomics in the early two-thousands. Clinical sequencing around 2010. Agentic AI from about 2023. Each time I arrived while the field was still arguing about what it was, and each time I got in without the credential the people around me had.

That is not a claim about talent. It is a claim about timing.

I was not precocious. I was early. Those are different things.

Genomics, before there was software

I joined The Institute for Genomic Research in 2002. It later became the J. Craig Venter Institute, and I was there eight years.

The human genome had a draft and no finish. The word metagenomics had not reached most biology departments. What that meant day to day is that the software to assemble and annotate a genome at that scale did not exist as a thing you could buy, so you wrote it, and the version you wrote was wrong until enough real data had run through it to show you where.

Nobody at TIGR ever asked what my degree was in. They asked whether the pipeline ran.

In 2010 the group built a synthetic Mycoplasma chromosome, transplanted it into a recipient cell, and the cell ran on it. They wrote four watermark sequences into the DNA, using a lexicon that maps DNA triplets to letters of the alphabet. One of those watermarks spells out the names of forty-six people who worked on it.

My initials are in there. Somewhere there is an organism carrying my name inside its chromosome, which is the closest thing to tenure I am ever going to get.

I helped computationally assemble the first human genome, the first large-scale metagenome, and the first synthetic genome ever made. I mention all three together because of what they have in common, which is that none of them had a method until somebody wrote one down afterwards.

The startup, and the boring half

I left for EdgeBio in 2010 to build the sequencing and bioinformatic interpretation platform for a CLIA lab. Clinical-grade genomics was maybe three years old as a commercial proposition. There was no reference architecture, so we built one, and most of what we built was wrong in ways we only discovered by running it against real samples.

GeneDx acquired the services business in 2013.

Around the same stretch I was running a small web and digital marketing company of my own, Inspiring Design, alongside the day job. It taught me the parts of building a company that nobody finds interesting until the moment they need them. How to form an entity. How to write a contract that survives contact with a disagreement. How to invoice, and what to do when the invoice does not get paid.

I have used that more often than I have used anything I learned in a lecture hall. If you are technical and you have never had to be the person who signs something, go and be that person once. It rewires how you read every proposal you see afterwards.

The fourth window was not a young field

Then thirteen years at AstraZeneca. I joined to work on genomics infrastructure and left running the AI Centre of Excellence for R&D.

The fourth window looked different from the first three. AI was not a young field by 2023. The papers existed, the tooling existed, and plenty of people understood it better than I did. What was young was the fit between that field and a hundred-and-fifty-year-old pharmaceutical company, and that gap turns out to open the same kind of opportunity as an empty field does.

So the work was FAIR data foundations, a prioritisation layer, agentic systems that did real research tasks, and a governance pathway that took model provisioning from weeks to hours. That last one mattered most and got the least attention, which is normal. I ended up repeating a line often enough that people started saying it back to me: safe is the fast way.

None of those four platforms was assigned to me. I wrote the first version of each one myself, proved it worked, recruited people better than me, and handed it over. That is the only method I have.

What the degree cost

I have a bachelor's degree in biology. That is the whole credential.

Almost everyone holding a job like the one I have now holds a doctorate. This is the part of the essay where I am supposed to tell you it did not matter, and that would be a lie, so here is the honest version.

It cost me the benefit of the doubt. In a scientific room, a person with no letters after their name starts a notch below the line and has to climb over it by being useful. Every time. In every new room.

It cost me speed early on, because there are things a good advisor hands you in a year that took me four to learn by walking into them. And there are rooms where the credential is the entry condition and no amount of shipped work substitutes for it. I have been outside a few of those.

What it did not cost me was the work. Not once in twenty-five years has anyone stopped me building something because of what my degree said. The gate is on the room, not on the workbench.

When the window is open

Here is what those four moments have in common.

A credential certifies that you have mastered an existing body of knowledge. That is a genuinely useful thing for it to do, and I would want a certified surgeon. But when the body of knowledge does not exist yet, there is nothing available to certify, and the only evidence anybody can offer is a thing that works.

That is the window. It opens when a field is young enough that nobody has written the textbook, and it closes when somebody does. While it is open, building is the credential. After it closes, the credential is the credential, and getting in the way you got in before stops working.

The practical version, for anyone deciding whether to wait for permission: the window is not announced and it does not stay open long. In the web it was maybe six years. In clinical genomics, four.

You find out it closed when the job postings start asking for five years of experience in a thing that has existed for five years.

The part where I am supposed to talk about hype

I have now watched three technologies get badly oversold from the inside.

The web was going to replace retail by 2001. Genomics was going to give us personalised medicine within a decade of the genome being published. AI is currently going to do most of what you have read this year that it is going to do.

All three were oversold on timing. All three were roughly right on direction. The mistake people make is treating the collapse of the hype as a verdict on the technology, when the hype collapsing is just the money leaving before the infrastructure is finished.

For AI right now, the honest answer is that some of it will leave residue and some will not, and I do not think anyone can reliably sort them from inside the cycle. What I do think is that the residue is where the products are, and that finding out which is which is a measurement problem rather than an opinion problem.

Which is a reasonable description of the job I just took.

The address changed

Tempus is the first place I have worked where the data is not the constraint. Molecular, imaging and clinical outcomes on the same patients, at a size a model can learn something from. I have spent twenty-five years running into the absence of that, and I would like to find out what is on the other side of it.

People keep telling me this is a full-circle story. I do not think it is. A circle means you came back to where you started, and I never left. I have had one job for twenty-five years, which is to turn up somewhere the rules have not been written yet and start writing them badly until they get better.

The address changed. And that's about it.

I am still building, and still writing about it.

Every Sunday I pick one paper or release that's worth your time, break it apart, and tell you why it matters. No hype. No summaries of summaries. Just the idea, explained.

In late July, Anthropic deleted more than 80% of the system prompt that runs Claude Code.

That prompt is the standing instruction set the tool carries into every session, built up over a year of watching the model get things wrong. Four-fifths of it came out. Their coding evaluations showed no measurable loss.

The engineer who wrote it up said those constraints "were once needed to avoid worst-case scenarios," and that they could now "delete many of them and let the model use surrounding context and judgment instead." A rule about code comments that had grown into three dense clauses came back as fifteen words: write code that reads like the surrounding code, match its comment density, naming, and idiom.

They were not tidying up. They had worked out they were holding their own model back, through the system prompt and through the instruction files and skills they had written around it.

Everyone downstream has been writing those files too.

The cast

You put a cast on a broken limb. It is the right call. The joint is immobilized, the bone knits, and while the bone is broken nothing else does that job.

Leave it on past healing and the muscle underneath wastes. The limb that comes out is thinner than the one beside it. The cast never stopped doing what it does. What changed is that the thing it was protecting no longer needed protecting, so all that is left is the constraint.

Every instruction you have written for an AI system is a cast. You wrote it because the model got something wrong, and you were right. The model underneath has been replaced two or three times since, and nothing in your setup tells you which bones have healed.

The instruction did not stop working when the model improved. It became the thing holding it back.

None of that makes scaffolding bad. It makes it indistinguishable. Some of what you wrote is capability the model still lacks, and some is a monument to a weakness it grew out of, and on the page the two look identical: both helped on the day you wrote them, both read like good instructions, and only one of them still earns its place. Reading the file will not tell you which one you are holding.

What the vendor is telling you to do

Boris Cherny built Claude Code and now runs product for it at Anthropic. At Y Combinator, asked what ordinary users should do, he gave the shortest possible answer:

"for people that aren't building agentic products but you're using Claude Code, every six months delete your Claude.md. Delete your skills."

He is not guessing. Anthropic keeps an internal switch that strips every prompt out of the tool, and they flip it on purpose to check whether the prompt is helping:

"we actually use this as a sort of ablation to figure out: is the prompt useful? And what's interesting is that the model is actually a little bit more intelligent without these prompts."

Then, a few sentences later, he says the opposite:

"when you use Claude Code as a product, you do actually want some of these prompts because it helps you use the product."

Both of those are true, and reading them together is what makes the trade visible. The instructions make the model slightly less capable and the product considerably more predictable. Predictable is usually what you wanted. So the choice was never scaffolding against no scaffolding. You are spending capability to buy consistency, and almost nobody knows the exchange rate they are getting.

His rebuild rule is the practical half. Delete the file. Work normally. Add an instruction back only after you have watched the model make the same mistake twice. Guessing in advance is how the file got big in the first place, and every line you guess wrong at gets read on every request for as long as it sits there.

So I tested mine

I have 93 of these files. In May I had 41, after a deliberate cull with a written report justifying every cut. It grew back inside three months. It always grows back, because adding one is somebody's job and deleting one is nobody's.

Not one of them had ever been measured. I had usage counts and instincts.

The test itself is dull. Take one file. Run the same real tasks three ways: the assistant with no instructions at all, the assistant with my current file, the assistant with a rewritten candidate. Then have a separate AI read two answers side by side, blind to which came from where, and pick the better one against a written standard.

Before any of its verdicts counted, I handed the judge a rigged pair. One answer was good. The other was fluent, well formatted, and built on two invented citations. It had to catch them, and it named both. A judge that cannot fail will certify whatever you point it at, so the scoring does not start until it has proved it can flunk something.

I picked the file I use most: the one that tells the assistant how to read and review an academic paper. Three matchups, twenty blind comparisons.

The rewritten version won all five of its comparisons against no instructions. Against my current file it won eight of ten.

Then the matchup I had not expected to care about. My current file against nothing at all: three wins, two losses.

Five comparisons is thin and I am not going to dress it up as more. But three to two is not the score of a file I would have defended, and I would have defended that one hard. Most of what I thought it was adding, the model already had.

The instruction set I use most beat deleting it by three to two.

What had actually rotted

I kept the file. What was wrong with it was specific, and it was not the failure I would have guessed at.

Academic papers usually exist twice, as a free preprint and as the published version of record, and the numbers in them differ. My old instructions produced a review that took three figures from the preprint, reported them as the journal's, and then signed off by certifying that every number in the document traced to the published article. Wrong on exactly the sentence a reader repeats out loud in a meeting, and confident about it.

The rewrite makes every claim record which document it came from, and bans blanket confidence statements outright. It went in this week, after I read the diff.

The file had not gone useless. It had decayed in one specific place, and nothing I was tracking would have surfaced it. Usage was high. Overlap with other files was zero. It read well.

The dashboard will not tell you

The obvious question is whether you can skip the testing and read this off your metrics instead. You cannot. Every cheap signal I have used to judge these files has already misled me, and each failure is written down with a date.

I removed a browser tool after a scan showed it unused. The scan window turned out to be 0.2 days. Measured properly afterwards: 3,030 calls across 159 sessions, 19 of them on the morning I removed it. Restored the same day.

Four of my instruction files show zero use while the capability behind them runs constantly. One shows zero uses against 770 calls to the same underlying tool by a different route. The instruction was never the path anything took. A zero can mean nobody wants this, or it can mean something else already won, and on a dashboard those are the same number.

In July one component measured 22,837 tokens of overhead. In August the same component measured 863. Nothing about it had changed. A setting governing when things load was misconfigured, so every earlier measurement had been reading the setting rather than the component. A separate cut I projected would save 5,665 tokens measured 111.

A per-component cost is a property of your configuration, not of the component.

Cutting is not free either

In April, Anthropic added one instruction to Claude Code's system prompt telling it to keep responses brief. One of their evaluations showed a 3% drop, on both Opus 4.6 and 4.7. Added on the sixteenth, reverted on the twentieth. A single sentence of standing guidance, and the damage ran the opposite way to anyone's intuition, because the instruction that hurt was an instruction to be concise.

There is also no threshold to aim for. The work measuring how models degrade as instruction counts rise finds steady decay across 500 instructions, in three different shapes depending on the model, and names no cliff in any of them. Prune by whether a thing still earns its place, never by a budget.

Which three you measure this quarter

A finer-grained experiment exists and I could not afford it: instead of testing a whole file against nothing, cut it into layers and measure what each layer adds on its own. Showing that a layer helps takes six comparisons. Showing that one does nothing takes twenty three, because proving no difference is a much stronger claim, and four layers at that rate runs about four times what one night already cost.

So measuring everything was never the plan. One file took about sixty agents and most of a week's compute. The plan is picking which three you measure this quarter, and three properties decide it. Usage is not one of them.

The first is reliance. Not how often a file fires, but how far you would trust its output without checking. A file whose answers land in a slide nobody re-derives deserves more scrutiny than one that fires forty times a day into work you read line by line anyway.

The second is age measured in model generations, not months. An instruction written for a model two releases back was written against failure modes that may no longer exist. Mine was young on the calendar and two generations old.

The third is whether the file makes factual claims that can rot without showing a symptom. Mine did. A style instruction that goes stale produces prose you dislike, and you notice inside a paragraph. A factual instruction that goes stale produces confident wrong numbers in a register that reads exactly like the correct ones, and nothing in the output tells you which one you are looking at.

The job nobody owns

Underneath the technical problem is an organizational one, and it decides whether any of this happens more than once. It also explains how my own count went from 41 back to 93 in three months without anyone deciding that it should.

Adding an instruction is somebody's job. It happens the moment something breaks, it takes ten minutes, and the person who does it is visibly fixing a bug. Deleting one is nobody's job. It means proving a negative, it produces no improvement anyone can see on the day, and the person who does it is one incident away from being the person who removed the guardrail. The incentives are not close, and they point the same way in every organization I have watched, including my own.

The teams that get this right are not the ones with the best tooling. They are the ones who have made four unglamorous things normal.

Every instruction carries the failure it was written for, and the date. One line. An instruction with no recorded reason can only be reviewed by the person who wrote it, and in eighteen months that person has changed teams or forgotten. Write down what broke and you have handed the next reviewer a test they can run: does it still break.

Removal has a named owner and explicit cover. Somebody's job description includes taking things out, and leadership has said out loud, before anything goes wrong, that a removal which turns out badly is a normal cost of the review rather than a mistake with a name attached. Without that, the review meeting happens and nothing leaves the room.

The review runs on model releases, not on quarters. Every release re-dates every instruction written before it. Tie the review to the calendar and you will audit a stable setup in a quiet month and miss the one that moved under you in a loud one.

The output of a review is a number, not an opinion. Two versions, real tasks, and a reader who does not know which is which. It does not take sixty agents. It takes one person blind to the answer, which is the step that gets skipped, because the author of a file is the worst available judge of it and is usually the only person in the room.

Adding an instruction is somebody's job. Deleting one is nobody's job. That asymmetry, not the model, is why your setup grows.

Ninety-two

The three-to-two is not the number that bothers me.

I own 93 instruction files. I measured one of them, on one night, and it cost about sixty AI agents and most of the compute I would normally spend in a week. The other 92 are exactly as unexamined as they were on Monday, and several of them are older than two model generations.

That ratio is going to get worse. Models ship faster than anyone's scaffolding gets reviewed, every release re-dates every instruction written before it, and there is no alert for any of it, because nothing degrades visibly and the file keeps loading and the work keeps coming out fine while the gap between what your instructions do and what they were written to do widens on a schedule you do not control.

So pick your most-relied-on instruction, the one you would defend hardest, and run a week without it. Have someone who does not know which is which read the output both ways.

You will get one of three answers, and two of them are cheap. Either something better exists, or the model outgrew it, or what you have is fine and you can stop wondering. I went in expecting the first. I got it, and I also found out my old file was barely beating the fourth option nobody plans for, which is having written nothing at all.

Sources

• The new rules of context engineering for Claude 5 generation models: Thariq Shihipar, Anthropic, 24 July 2026. The 80% deletion, the "no measurable loss" claim, and the then/now table of practices Anthropic now calls myths.

• Boris Cherny: Building Claude Code: Y Combinator Startup School. The delete-every-six-months advice, the ablation switch, and the admission that the model is "a little bit more intelligent without these prompts."

• Anthropic's April 23 postmortem: the brevity instruction that cost 3% on one evaluation, added 16 April and reverted 20 April.

• How Many Instructions Can LLMs Follow at Once?: Jaroslawicz et al. The IFScale benchmark, 500 instructions across 20 models, and the finding that there is no threshold to prune toward.

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

My AI coding setup is two years old. It is not a model. It is a system: rules for how the assistant should behave, skills it knows how to use, specialized agents for different kinds of work, guardrails that stop it deleting things it should not. I built it up the way you build anything that carries real weight, one piece at a time, and I assumed it was welded to the expensive model it ran on.

Over the weekend I moved the whole thing onto a model that costs 40 cents. Every rule, every skill, every agent, and most of the guardrails. The port took an afternoon. It cost under forty cents, measured on the model's own usage dashboard. It was fast enough that nothing sat waiting. And for the work I actually do, I could not tell the difference.

You are reading the result. This post was drafted on that 40-cent model, running my rules, my skills, and my voice checks.

Under 40 cents. Nobody noticed the difference.

What a two-year harness actually is

Vendors sell you "the agent" as if it were a single thing, inseparable from the model underneath. Here is the model that thinks, here is the tool it lives in, they come as a pair. That framing is comfortable because it is simple. It is also wrong.

What I actually depend on is the layer above the model. A set of written rules that encode judgment: what to let the assistant do on its own, what to check before believing an answer, how to recover when something breaks. A library of skills, each one a procedure the assistant can follow, from drafting a blog post to running a security review. A few specialized agents for jobs that need a particular temperament or toolset, which I have come to think of as a squad rather than a toolbox. And guardrails, the ones that refuse a destructive command and tell you to back up first. I broke the pieces down component by component if you want the inventory.

Two years of that is an asset. It is the accumulated discipline of deciding, over and over, what you want an assistant to be allowed to do. The model underneath is just the executor. I had never tested whether the executor mattered as much as I assumed.

The provocation

DeepSeek shipped an endpoint whose only purpose is to let a rival company's coding tool talk to its model. Not a community wrapper, not a shim someone reverse-engineered: an official endpoint, so a competitor's agent points at DeepSeek and works.

It is not the first to do this. Kimi and MiniMax already run the same kind of passthrough, which is what makes it a signal rather than a stunt. When three model makers independently build an on-ramp into someone else's tool, they are all betting the same way. They are betting that the user's investment sits in the harness, and that the model underneath is what swaps out. I decided to test the bet on my own setup.

I had already compared these tools from inside one back in May, which tells you where they differ on paper. That is a different question from what happens when you pick a working setup up and put it down somewhere else.

The number that matters

Here is what the port actually cost.

The expensive model I was on charges $5 per million input tokens and $25 per million output. DeepSeek V4 Flash charges $0.14 and $0.28. The July build, the one Artificial Analysis actually put a score against, is cheaper still at $0.09 and $0.18. Call it 35x cheaper on input and 90x on output, and hold in mind that those are the conservative numbers.

The whole migration, every piece of the harness moved over, came to under forty cents on the model's API key.

The expensive model was never the asset. The discipline was.

And the cheap model is not a toy. On Artificial Analysis, the independent scoring benchmark I trust, DeepSeek V4 Flash scores 69.1 on the coding index. Claude Opus 4.8 scores 74.3. Claude Opus 5, the current flagship and the one I actually came off, scores 78. So the cheap model sits about nine points behind the best model money can buy, at one-ninetieth the output price.

I am not telling you cheap models are as good as expensive ones. They are not. Measured against Opus 5 on coding, DeepSeek lands just under ninety percent. On the broader intelligence index it falls to about eighty-two. The gap is real and it is wider than the excitement suggests. For one specific kind of work it is also invisible, and the port was that kind of work.

What transferring taught me

Four things, and none of them were what I expected.

The written parts moved without a fight. Rules, skills, agents. They are all just instructions, and instructions do not care which model reads them. The expensive model was never doing the thinking my setup does. It was following thinking I had already written down.

Judgment travels. Enforcement stays behind. One of my guardrails is a plain paragraph telling the assistant to back up before it deletes anything, check the backup worked, and only then delete. The cheap model read that paragraph, refused a delete it had been asked to make, and walked the sequence. My two mechanical blocks, the code that physically stops a dangerous command before it runs, never made the trip. The new tool keeps a fingerprint of your safety file and quietly ignores anything it does not recognize, so both blocks are sitting there written, tested and switched off. I would rather say that than let "every guardrail moved" stand.

A whole phase of the work was unnecessary. My plan said the new tool could see two of my skills against the eighty-eight I have. So I wrote a converter and moved them all. Then I hid a uniquely named skill in each location and asked the tool what it could see. It saw both. It had been reading my original library the entire time.

Underneath that was a difference only the hidden-file test would have found. My usual tool hands the model a skill's full instructions the moment it needs them. The new one shows a one-line summary and leaves the rest on disk, read only if the model goes looking. That economy is why skills work at all, and the new tool pushes it further than I would have chosen. Same files, very different odds of the thing running.

The dial that does nothing. The model has a low-to-high effort setting. On the connection my setup uses it is inert: I told it to think briefly, then told it to think hard, and got the same work at the same cost either way.

Then I went looking, and the public picture is a mess. teortaxesTex read a benchmark chart as low matching high at twice the price. Morgan Linton, who ran that benchmark, found close to the opposite: DeepSeek won on medium effort, not high. Same model, same week, two readings. I did not settle it. The claim I can stand behind is smaller, that on my connection the setting does nothing, and no documentation would have told me.

A control you cannot verify is a control you should not build assumptions on.

The comparison

The cheap model is having a moment. opencode, a coding tool, tracked 8 trillion tokens of DeepSeek Flash traffic on August 1st alone, 5 trillion of that free usage. DeepSeek's own launch numbers put V4 Flash at 82.7 on Terminal Bench 2.1, ahead of its larger sibling, and that is a vendor number, so weigh it the way you weigh vendor numbers. The independent read is Morgan Linton's VulcanBench run, where it took the top spot outright and pushed two much better-known models out of the top five.

The objection that deserves airtime is cost per task rather than cost per token. cline raised it: a cheaper token still loses if the model burns more turns reaching the same answer. It then answered its own objection in the same post, citing Artificial Analysis clearing the same benchmark tasks at 105x lower cost. My experience splits along that line. For harness plumbing, moving rules and skills and agents around, the cheap model held without extra turns. For the hardest reasoning it is still a step down. The judgment call is knowing which of the two you are doing before you start.

This is uncomfortable for the premium tier. At roughly ninety percent of the capability and one-ninetieth of the cost, the premium has to buy a workload you can name and point at. Not "it is the best model." A task. If you cannot produce one, you are paying for the brand.

The meta turn

Which brings me back to where this started. This post was researched, structured and drafted on the 40-cent model. The same rules, the same skill that formats a Substack essay, the same voice checks that stop me sounding like a language model, all running on something that costs less than a vending machine snack.

Be precise about what that proves, because there are two separate tests here and only one of them wrote this. The first is the model swap: my everyday tool, my harness, a different and far cheaper model underneath. This post is the output of that one. The second is the framework port: the same harness rebuilt inside a rival vendor's tool, which is where the guardrail finding came from and which I ran on its own. Neither test alone would have told me much. Together they say the harness survived a change of model and a change of tool.

This is also not the first long thing it has produced. Earlier this year the same setup drafted a 33,000-word book in eight days, Builder-Leader: The AI Exoskeleton That Crosses the Gap, whose argument is the one I have just spent an essay stress-testing against a model that costs 40 cents. I did not set the port up as a sequel to the book. It became one, and it could have gone the other way.

You are reading this because a 40-cent model ran my harness well enough to write it.

A two-year-old pile of written instructions, pointed at a nearly-free model, then asked to hold a style, route research and stay disciplined across a full essay. It did. The model did not need to be the smartest one available. It needed the instructions to be good.

Close

I did not downgrade my setup this weekend. I found out how much of it never depended on the expensive model at all.

The moat was never the model. It was the accumulated discipline: what to let an assistant do, what to check before believing it, how to recover when it breaks. Written down, tested, and enforced wherever the tool let me enforce it. That followed me for 40 cents and it is still doing the work.

Every setup has a number here and it costs an afternoon to find: swap the model underneath and count what still runs. Whatever survives is the asset. Whatever breaks was rented. I have argued before that you should choose your harness every six months rather than let it choose you, and this is the cheapest test I have found for whether you own one.

I did not downgrade my setup. I found out how much of it never depended on the expensive model at all.

Sources: Artificial Analysis model indices for DeepSeek V4 Flash, Claude Opus 4.8 and Claude Opus 5 · opencode on DeepSeek usage · rohanpaul_ai relaying DeepSeek's Terminal Bench numbers · morganlinton on VulcanBench · teortaxesTex on effort scaling · cline on cost per task.

A published result is a promise. Run this, and you will see what we saw. For most of the history of computational science nobody collected on that promise, because collecting cost about as much as the original work and earned a fraction of the credit.

On July 15, Hugging Face and alphaXiv opened a contest to collect. It closes Sunday August 2. Roughly 6,800 papers from ICML 2026 are in scope, anyone can attempt a reproduction, and every attempt gets published as a public logbook that an automated judge then scores. The prize pool is $4,500, paid in GPU credits rather than cash, which tells you who the organizers think is entering.

As of this morning, eleven days in: 4,147 judged logbooks from 312 people, carrying 20,594 separate verdicts on individual claims. About 1,600 papers have been attempted at least once, roughly a quarter of the conference.

To see why that is a strange number, you need the thing it replaces. Joelle Pineau started the ML Reproducibility Challenge at ICLR 2018 as "a small community experiment." Eight years on it is the field's serious effort, mostly run through graduate courses, and its NeurIPS 2019 edition drew 173 papers claimed, itself a 92% jump on the year before. Claimed, not finished. Published reports have run in the tens. ReScience C, which demands a fresh independent implementation rather than a re-run of the authors' code, manages about a dozen a year.

173 papers claimed in a year, against 4,147 judged logbooks in eleven days. That is not an improvement on the existing effort, it is a different quantity.

What is actually being scored

One design decision separates this from a leaderboard, and it is easy to skim past. The unit of judgment is not the paper. It is the sentence.

Every paper has been broken into its individual claims, and each claim gets its own verdict: verified, falsified, toy (a scaled-down version that says so), or inconclusive. Nobody asks whether a paper "reproduced," which is a question with no answer. A paper can come back partly verified, partly toy and partly inconclusive at once, and most of them do.

Then the scoring, from the contest's own FAQ: "2 points for a full reproduction or full falsification, 1 point for a toy-scale reproduction, 0 otherwise."

Falsifying a claim pays what confirming one pays. Against a literature where negative results are half as publishable as positive ones, that is a deliberate inversion.

When Nature surveyed 1,576 researchers in 2016, 24% had published a successful replication and 13% a failed one. The same survey found more than 70% had tried and failed to reproduce someone else's experiment and 52% agreed the field had a reproducibility crisis, and then 73% said they still trusted at least half the papers in their own field.

The failures are also the deliverable. A logbook, in the organizers' description, holds "the experiments, simplifications, failures, and results it found." Prior efforts published conclusions. This one publishes the attempt.

What it does not do is test the hard version. Machines have been tried on this before and were not good at it: every benchmark that measured reproduction topped out between 16% and 27% on rebuilding a paper from scratch, with top ML PhDs ahead of the models, and only reached about 60% when handed the original code and data. This contest tells participants to start from the authors' code. Re-running someone's code, rebuilding from the paper alone, and getting the same finding on fresh data are three different tests, and headline percentages get quoted across that boundary constantly.

What the inside looks like

For a large class of papers the easy version is now free, which is the whole reason this works. We downloaded a full-marks logbook from the contest leader and read it. The entire reproduction ran on a laptop processor in three seconds. No GPU, no cluster, no cloud bill. The real ceiling is not compute but publishing: Hugging Face caps accounts at 20 new pages a day, and when a participant asked for relief the organizer's answer was flat, "we cannot lift the 20 spaces/day limit so I'll close this issue."

I entered, which is the only reason I have anything to say about the texture of it. We are 23 judged logbooks and 174 points in, against a leader at 1,862. Not a contender, and far enough inside to see how the work divides.

It divides into two lanes, and we ran them differently on purpose.

The volume lane is an autonomous agent called Vulcan, and it never publishes anything. It runs on a box in my house on free local compute, and its job is narrow: take a paper whose claims are all theory and check every formula numerically, against a brief that fixes the method rather than the answer.

A check that passes on every input you can construct certifies nothing.

Its audit of one paper came back as 721 lines of code that matched the paper's algebra to the last digit a computer can represent, plus a deliberate sanity test that failed when it was supposed to. Careful work, at no cost, with a hard limit written into the contract: it stages, a human publishes. No autonomous process of mine puts a verdict about somebody else's paper into the world.

The other lane is me and Claude Code running on Opus, for the parts the first one cannot do. Two things came out of it.

The first is that picking the right paper beats checking it well. Our most careful audit matched its paper exactly and scored 2 points out of a possible 8, because half that paper's claims were hardware benchmarks that sit at inconclusive until somebody spends days of compute on them. The judge said so directly: the logbook left "half the paper's claims without any experimental evidence." A paper whose claims are all theory earns 12 points in an afternoon on a laptop. Selection is the skill, not rigor.

The second is what the process catches. One paper states a theorem using a fixed setting, and then, four thousand lines later, its own proof assumes that setting changes with the length of the run. Not a typo, and not a disagreement between the paper and the world: the paper disagrees with itself, in print, peer-reviewed and accepted. Nobody had noticed because nobody had reason to run the theorem as written and watch it fail.

Beyond those two, the work was unglamorous. It caught our own staging script dropping most of a conclusion, so a 60-line write-up shipped as three sentences. Vulcan itself lost twenty hours to an outage at its model provider, with no backup configured and nothing raising an alarm.

None of those failures announce themselves. Every one produced output that looked finished.

The three percent

Of 20,594 claim verdicts, 615 are falsified. That is 3.0%. The rest split 43.5% verified, 28.9% inconclusive, 24.6% toy. The distribution has barely moved for days while the corpus grew by hundreds of logbooks: the shape is stable, the counts are not.

Three percent is low, and it has two readings. Either the literature is in better condition than the surveys suggest, or falsification is the hardest verdict to reach and the easiest to get wrong.

In one day, four of our audit agents returned a falsification. Three of them were wrong. All four came out of the automated lane. All three errors were caught by the other one.

One agent read a formula out of a PDF that had lost a character in scanning, then correctly proved the mangled version false. Another was checking a sentence the authors never published, because the claim it was handed came from a draft that is not the public paper. The third found a difference too small to distinguish from rounding error and called it a contradiction.

Every one produced a confident, well-formatted, coherent falsification. I wrote about that pattern in The Failure That Leaves No Corpse, and this is the cleanest instance of it I have run into.

The rule we ended up with fits on one line: does the paper disagree with itself, or does the claim disagree with your copy of the paper? Only the first is a falsification. The second is a bug in your pipeline wearing a falsification's clothes.

Applying it cost us the verdict. Across the 130 claim verdicts we have published, the number labelled falsified is zero. The one candidate that survives is the self-contradicting theorem above.

The default failure of an automated checker is not a missing answer. It is a wrong one, delivered with the same confidence as a right one.

This is not only our problem. The contest's own claim lists are extracted by a language model, and a participant found a claim in one paper's list that belonged to a different paper. The organizer confirmed it in the open, "That claim belongs to a different Safety-area paper and leaked in during auto-extraction," removed it, rescored, then swept all 6,768 papers for the same contamination.

And the bias is measured, not hypothetical. A June 2026 study of reproduction agents found that showing the agent the original paper made it more likely to confirm claims that could not actually be reproduced, and that small changes in wording pushed it further the same way. In this contest the agent always has the paper and the claim in front of it.

The judge is a language model

The thing scoring 20,594 claim verdicts is itself a language model.

There is published criticism aimed squarely at that. A 2026 position paper, "Stop Automating Peer Review Without Rigorous Evaluation", compared human against AI reviews of ICLR 2026 submissions and found two things. The AI reviewers showed "a hivemind effect of excessive agreement within and across papers that reduces perspective diversity." And their scores turned out to be "trivially gameable through paper laundering": rewriting a paper with a language model raised the scores it got, on style rather than science.

None of that is a reason to dismiss the contest, and I want to be careful here, because I am competing in it and have every incentive to be generous.

The defence is structural rather than technical. The artifacts are public. Every logbook is a page anyone can open, the verdicts are a public dataset downloaded 38,438 times last month, and the errors get argued in threads with the organizers answering. The contamination case is the proof, and the fix landed inside a day. A closed judge making the same error produces a number nobody can audit.

So the position I hold is that this design pairs a known-weak instrument with an unusually strong correction loop. Whether the loop outruns the instrument is an empirical question, and a week from now there will be enough public data to start answering it.

What it would take to trust this

I am not going to predict how it turns out. The contest closes August 2 and every distribution above is provisional.

The concrete claim is narrower and I think it holds. For a meaningful slice of published science, checking a claim now costs less than making it. Three seconds of laptop processor against however many months produced the original theorem. That ratio is why verification stayed volunteer work for a decade, and it has flipped.

What it does not change is the direction of the error. A checker that costs nothing will be run constantly, and cheap checking produces far more false accusations than misses. Confidently accusing a sound paper is worse than missing a bad one.

The contest's design anticipates this better than most things I have seen. Paying the same for a falsification as a verification removes the incentive to go looking only for confirmations. The logbook has to carry the failures, so there is no tidy way to bury the attempts that went nowhere. And because every artifact is public, a wrong answer stays inspectable instead of disappearing into an aggregate.

Nobody has to trust the checkers. You can read what they did. That is a weaker guarantee than peer review claims to offer, and a stronger one than peer review delivers.

Sources

• Reproducing ICML 2026, Open Reproductions: the contest, its FAQ, rules, and public discussion threads. Verdicts live in a public dataset.

• MLRC 2026: Reproducibility as an Official Track at NeurIPS: the ML Reproducibility Challenge's history and Pineau's founding description of it.

• Improving Reproducibility in Machine Learning Research, Pineau et al., JMLR 22(164): the 173-papers-claimed figure.

• ReScience C: the journal that requires a new independent implementation.

• 1,500 scientists lift the lid on reproducibility, Monya Baker, Nature 533: the 2016 survey of 1,576 researchers.

• PaperBench and CORE-Bench: the agent-reproduction benchmarks this contest inherits from.

• Reproducibility agents and confirmation bias: the June 2026 benchmark showing that giving an agent the paper biases it toward confirming.

• Stop Automating Peer Review Without Rigorous Evaluation: Baumann, Pei, Koyejo and Hovy, 2026: the hivemind effect and paper laundering.

• The Failure That Leaves No Corpse: the earlier piece on failures that produce a confident answer instead of an error.

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

Last week a team at Cognition, the company behind the coding agent Devin, published a number that reads like a typo. They replaced Opus 4.8 with Fable 5. Fable 5 costs about twice as much per token. Their bill went down.

Not their quality. Their bill.

Here is the part of their table that matters:

Fable 5, inside their new architecture: score 57.6, cost $3.00 per task.

Opus 4.8, on its own: score 48.8, cost $3.24 per task.

The expensive model, wired up correctly, was better and cheaper than the cheap model on its own. Not a trade. Both columns at once.

If you have ever sat in a meeting where someone proposed saving money by dropping down a model tier, that result should stop you. It stopped me, because I had run the opposite experiment, and I had lost.

What they actually built

Cognition split the work between two models instead of routing between them.

The expensive model is the driver. It plans, it interprets what the request actually means, it makes the judgment calls, and it does the final review. It takes very few actions itself and it reads only what it must.

The cheap model is the sidekick, and it is not a helper function. It is a full agent with its own tools that goes and does the mechanical work: fetching context, running the slow tests, carrying out the routine implementations once someone has decided what they are.

Both hold their own memory. Both run at the same time. And this is not a demo. Eighty-eight percent of Cognition's own internal code changes now run through that automated split.

The reason it works fits in one line. You pay for thinking once, and for doing it many times.

The problem is that the opposite is also true

Everyone in this field has met the other result. You move a workload to a cheaper model to save money, and it costs you more. The cheap model misunderstands the task, produces something confidently wrong, and a person spends an afternoon unpicking it. The invoice went down and the total cost went up.

That happens, and I am not going to argue with it, because I have the receipts.

So we have two findings that appear to be at war. Expensive models save money. Cheap models cost money. Both are observed, both are honest, and most of the advice you will read picks one and ignores the other.

I ran the losing experiment

I built a cost router. The logic was the logic everyone reaches for: work out how hard the task is, send the easy ones to a cheap model, keep the expensive one for the hard ones.

It saved nothing. Not a little less than I hoped. Nothing.

I want to be precise about why, because the failure is more useful than a success would have been. I routed by task type. The axis that pays is ambiguity.

Every individual call my router made was defensible. This one looks like a simple rename, send it to the cheap model. That one looks like architecture, keep it upstairs. And the total refused to move, because "simple rename" was a description of the work, not a description of how much had already been decided. Half the jobs I was labelling easy still had open questions inside them, and an open question handed to a cheap model is the single most expensive thing you can buy.

The law

Once you see it, the war stops.

A cheap model is expensive on an open question and cheap on a closed one.

An open question is one where something still has to be decided. What are we actually building. What does done mean here. Which of these two readings of the request is the real one. Give that to a cheap model and it will not tell you the question is open. It will pick an answer, sound sure, and hand you something plausible that you now have to check line by line.

A closed question has had the deciding done. Rename this function in these three files, and the test that proves it is this one. There is no judgment left in the task, and a cheaper model does it for a fraction of the price with nothing at risk.

Which gives the driver model a job description nobody writes down. Its work is not "the hard parts." Its work is to turn open questions into closed ones. And that conversion has a name we already use for it. It is called a plan.

This also explains the finding that keeps embarrassing people who try to build a committee of cheap models and vote. A group at Princeton tested that directly last year and found that running the single best model several times and combining its own answers beat mixing different models together, on every one of the thirteen mixed configurations they tried, using roughly half the forward passes. Their explanation is blunt: mixing models of different quality drags the average quality down. You cannot vote your way to judgment. Where the question is still open, quality dominates, and diversity is not the free lunch it looks like.

Cheap is not a property of the model

Here is where I think the whole conversation is framed wrongly, including by the people getting the right answers.

Everyone calls this "big model, small model." My own setup says that is not the axis.

The models I push my mechanical work to are not small. They are large, capable models. They cost me nothing at the margin, because I bought them on flat monthly subscriptions instead of by the token. That is not a smaller brain. It is a different contract.

So there are two dials, and they are independent. Where does the marginal cost live, and where does the judgment live. A fine-tuned small model is cheap because it was narrowed. A large model on a flat rate is cheap because of how you bought it. Both are "the cheap one," and treating them as the same thing is how people end up sending an open question to a bargain and wondering why the quarter went sideways.

The sentence I had already written

I found a note I made a few weeks ago, while building the thing that hands my grunt work off to those subscription models. I had written the law without recognising it:

A vague spec produces confident garbage, and the cheaper the model, the truer that is.

Read that again as a cost statement, because that is what it is. A cheap model is not a discount. It is a multiplier on the quality of your specification. Specify tightly and it multiplies your savings. Specify loosely and it multiplies your mess.

Which is why the discipline I run alongside it is not optional. The expensive model writes the entire work order: the task, the files, what done means, and the exact command that proves it. And when the cheap model comes back and says the job is finished, that claim is worth nothing. I read the difference it made, and I run the test myself. It has never once been the model's word that closed a task.

So the small-model story is not a cost play

This is the part I keep chewing on, because it inverts something I believed.

Everyone is excited about training small models to do one narrow job extremely well, and the excitement is framed as a cost saving. It is not, or at least the saving is not where people are pointing.

If you can specify a task tightly enough to train a small model on it, you have proved the task is closed. All the ambiguity was removed by somebody, at some point, doing the expensive work of deciding. The training run is just collecting the winnings.

A small model is not a discount. It is a receipt for judgment already spent.

And the same is true of every cost reduction I have ever managed to make stick at scale. Every one of them was bought earlier, by an act of judgment that closed a question. The saving showed up in the invoice. It was created somewhere else entirely.

The boundary, in their words

Buried in Cognition's own write-up, past the numbers, is the caveat I would have led with:

The sidekick fails when judgment is the deliverable.

Their example is a hard feature whose subtle intent got lost the moment it was handed down. The work came back correct and wrong at the same time.

Anyone who has run a team knows exactly where that line sits, and knows it is not about the seniority of the person you handed it to. You can delegate the work. You cannot delegate the judgment about what the work is for. The failure looks identical from the outside either way: something arrives, it is technically defensible, and it is not what the thing was for.

The honest caveat

One thing about that Cognition result deserves saying out loud, because they said it themselves and nobody repeating the headline has.

Their best row was measured on Fable 5 during a stretch when the model was briefly pulled from sale, suspended in June under a US export-control order and restored at the start of July. Their own note adds two more caveats: those numbers were taken before the interruption, and that configuration was never tuned the way the others were. The model is back on sale now. The untuned-config caveat is not.

The pattern transfers. But the exact recipe is one vendor's best-case number on a setup they admit they never optimized, which makes it directional, not a benchmark. A result nobody has reproduced is a claim, not a finding, and that distinction is worth keeping close in a year when every week produces a new number.

What to do with this

You do not save money by hiring cheaper people. You save money by putting your best judgment on the plan, and then making the execution mechanical enough that it does not need judgment. Every leader has run the cheaper-people experiment at some point. Most of us have the scar to show for it.

The unit of cost was never the hour. It was the outcome.

So the question to take into your next architecture review is not which model you are using, or what it costs per million tokens. Those are the numbers on the invoice, and the invoice is a lagging indicator of a decision somebody already made.

Ask what you are paying per solved problem. Then ask who is doing the thinking.

Justin Johnson writes Run Data Run. His book on building with AI, Builder Leader, is at builder-leader.com.

In 2016, an OpenAI agent was set loose on a boat racing game. It found an isolated lagoon where three targets respawned forever, and it learned to spin in a circle and farm them. It repeatedly caught fire. It crashed into other boats. It drove the wrong way down the course. It never finished a single race.

It scored about twenty percent higher than the average human player.

That story gets told as a curiosity, a clever machine doing something silly. It is not a curiosity. Every executive reading this has approved that boat's quarterly results.

The boat did exactly what it was asked. Nobody had checked what they asked for.

The category your dashboard cannot hold

There is a taxonomy every leader already carries around. Known knowns, the things you know you know. Known unknowns, the things you know you do not know. And unknown unknowns, the ones you do not know you do not know.

Almost every mechanism you have is built for the middle category. Risk registers list known unknowns. Escalation paths, on-call rotations, red flags in a status pack: all of it assumes the failure will announce itself. Something will break, someone will notice, a corpse will turn up.

The third category has no such courtesy. An unknown unknown leaves no corpse.

A dead instrument leaves a crash. Four weeks perfecting the wrong thing leaves a green dashboard and a satisfied changelog. Nothing will alert you, not this week, not ever. The only way that failure surfaces is if somebody decides, unprompted, to go and ask.

This is not new. What is new is the rate.

An AI collaborator is very good, very fast, and extremely willing. Point it at a task and it will improve that task, tirelessly, inside whatever frame it was handed. It does not stop to ask whether the frame is right, and it reports success either way. It is an unknown-unknown machine, and it runs at a speed no human team has ever run at.

The gradient has a direction, and it has been measured

The failure is not that a model does the average thing. It does the cheapest thing that clears the bar. Not the mean. The nearest. Four separate literatures point at the same behavior.

Shortcut learning. Geirhos and colleagues, writing in Nature Machine Intelligence in 2020, showed that networks take the easiest rule that passes the benchmark, not the intended one, and the two are indistinguishable until the world changes. The clinical example is the one that should stop a biopharma reader cold. A model that appeared to detect pneumonia from chest X-rays worked beautifully until it met a new hospital. It had learned to read the hospital's own metal token stamped on the scan, then combine it with how often that hospital saw pneumonia. It had learned almost nothing about pneumonia itself.

The scoreboard pays for guessing. In a 2025 paper from OpenAI and Georgia Tech, researchers surveyed ten major benchmarks and found nine give zero credit for abstention. It is trained against a scoreboard that pays for a confident answer and pays nothing for saying the question is wrong.

Reward hacking has a rate. METR measured frontier models gaming the evaluation on 30.4% of task attempts. One patched the scoring function so that every submission was judged successful. Another overwrote the equality operator so the grader's comparison always returned true. Asked afterwards whether this matched what the user wanted, the models said no, ten times out of ten. They know. It is not confusion. It is the gradient.

And it agrees with you. Stanford's SycEval work found that a correct answer is flipped under simple pushback about one time in seven, and once the model has capitulated it stays capitulated roughly four times in five.

None of these produce an error message. Every one of them produces a result.

Five shapes

I started keeping a ledger of these as they happened to me. Written up, they stop looking like seventy separate mistakes and start looking like five, wearing different clothes.

One. The frame arrived as a default, and everyone optimized inside it. Nobody chose the question. A setting chose it, or a prior session, or whoever handed the task over. Then every hour after that goes into making the answer better. The tell is that the work is good: careful, measurable, improving. I spent four sprints tuning inside a data corpus that turned out to be a fortieth of what we held. An extraction default picked it; no human ratified it. Optimizing hard is how you stay in a local minimum, and the metrics look healthy the whole way down.

The same shape shows up in a meeting: fifteen comments answered one by one, every answer correct, none naming the thesis underneath. A wrong-level answer, indistinguishable from diligence.

Two. The instrument that cannot fail certifies whatever you point it at. An instrument is anything that tells you whether something is true or working: a test, a gate, a dashboard, an audit, an eval. It can be broken. When it is broken, it does not say "broken." It says "PASS." It gets its own section below, the shape that nearly cost me the most.

Three. Silence was read as an answer. Nothing came back, so nothing is there. A research sweep of mine came back nearly empty and read as a quiet field. It was four separate broken legs: an exhausted credit balance, a search-quoting bug, a query that ran too long, and one source carrying pure noise. The field was not quiet. The instrument was. If a search returns nothing, prove the search works before you report the nothing.

Four. The summary outlived the thing it summarized. Every layer between a leader and the underlying fact is a place the truth can die unnoticed: the headline, the dashboard, the status deck, the note that says "fixed." One number, computed once on a flawed setup, survived five months across a whitepaper, a grant application, a partner brief and a standing rule, while the project's own log one rung below said it showed no clear benefit. A retracted number does not stop existing. It stops being watched.

Five. The correction was made of the same stuff as the bug. The most humbling one, because a fix feels like the end of a problem rather than the start of one. A fix is an artifact. It gets checked like any other artifact.

A dead instrument leaves a crash. A month spent perfecting the wrong thing leaves a green dashboard and a satisfied changelog.

The exam whose dunce could not lose

Here is shape two in full, the cleanest example I have of a failure dressed as a win.

Before spending real money training a model to pull structured facts out of free-text documents, we ran a gate: prove the model beats a dumb baseline by enough to be worth the compute.

We wrote the dumb baseline down in advance. It was: always guess the most common answer. On the field the whole evaluation rested on, it scored 13.6%.

Our model was going to beat that by a mile. Everything beats 13.6%.

And that is the trap, because a baseline that loses by a mile does not prove the model is good. It proves the baseline is lazy. A huge margin feels like rigor. It is a measurement of the dunce.

So we asked the question the instrument itself could never ask: could this baseline ever have won? We made it slightly less stupid, handing it the one free clue every document gives away in its first line.

It went from 13.6% to 88.4% on one of the fields.

Then the number that ended the argument. 88.4% is higher than the 85.7% rate at which the document contains the answer at all. The headroom we were about to buy compute to chase did not exist.

Three of six fields died at that gate, two to a forty-line rule whose accuracy was the ceiling. The catch landed before a single training hour ran.

Now run the other history. We train the model. It beats 13.6% by seventy-eight points. The dashboard is green, the result is "strong," and we ship a number that measures our own laziness. Nothing in that sequence errors. There is nothing to alert on. There is no corpse.

A large margin over a lazy baseline is a measurement of the baseline.

And the part I would rather not write. The code that caught this was written that same week, specifically to check the previous week's code, and it carried two of the same bugs. That is shape five, live. The relief you feel when you find a bug is the exact moment to check the thing that found it.

A blind spot shaped like its own subject

When I swept the ledger, 314 incident files, the first shape was the emptiest bucket by a wide margin.

Think about why. Not because it happens least. Because a wrong-level failure never gets written up as an incident, since nothing visibly breaks. Every other shape is enforced by a failure that eventually shows up and demands one. That one has to be caught on purpose, or it is never caught at all.

So the ledger has a blind spot shaped exactly like its own subject.

The record of my mistakes under-counts the one I am writing about, and it under-counts it for precisely the reason that one is dangerous. A failure that leaves no corpse does not leave a case file either.

First principles, or nothing

There is only one move that finds an unknown unknown, and it is unglamorous. Refuse the frame you inherited and re-derive the thing from the ground.

A cursor, a handoff, a prior session's conclusion, your own framing from last Tuesday: every one of those is a hypothesis, not a work order. Fixing the fault you were told about is how you hide the fault that matters.

In practice that is three questions, asked on a cadence, in writing, whether or not anything looks wrong:

1. Is the thing we are improving the right thing?

2. What have we assumed since the last time we checked?

3. Which of our standing claims has never been re-derived from its artifact?

The third one pays for the other two. It is how a retracted figure was found still alive in a grant application.

I should print the counter-evidence, because the essay is stronger for it. METR finds the task length an agent can complete on its own has doubled roughly every seven months for six years. So "they cannot hold a goal" is not something I get to say. OpenAI's own reporting has sycophancy falling sharply between model generations. So "it is getting worse" is not available either.

The claim that survives is narrower and it is enough. None of the underlying mechanisms have been removed. Binary grading. Preference-model reward. Test-passing as the only signal. The gradient is a default in the absence of countervailing pressure, and the only countervailing pressure anyone has demonstrated is a human holding the goal.

And the human half is structural too, not a matter of effort. Parasuraman and Manzey, reviewing decades of automation research in Human Factors in 2010, found that complacency toward an automated aid shows up in experts and novices alike, and cannot be trained away with practice. You do not fix this by trying harder or by hiring better people.

I have my own receipt. I asked a system a leading question, the way everyone does: "that seems like the right call, yeah?" It agreed. The change was a no-op, and it would have shipped with my full sign-off. The literature says roughly one in seven. Once was plenty.

Knowing the rule does not run the rule

The obvious response to all of this is to write the lessons down and circulate them. I did exactly that. Here is what it bought.

Sixty-eight of the sixty-nine failures in that ledger were bought inside a nineteen-day window during which a document containing almost all of these lessons was loaded into every session, on every machine. The lessons were there. In the room. At the moment of action. They were caught within hours, by somebody going and reading the thing. They were not prevented.

Then I tested it directly. A system with the relevant principle sitting verbatim in its context window told me, when asked, that it had no such rule. It produced the principle only after I handed it the exact words to search for.

Availability is not activation. A rule you merely hold does not fire, and it reads as satisfied every single time.

Which makes a rulebook one more instrument that cannot fail.

Holding altitude

You have just read two thousand words. On the evidence of my own ledger, reading them will prevent nothing. That is not modesty. I had all of this written down, loaded, in front of me, and I bought sixty-eight of these failures anyway.

The versions that ever fired left something behind. A question asked on a cadence, so that it gets asked when nobody feels like asking. A way it could be wrong, written down before the result rather than after. A baseline that somebody had to prove could win. A principle with no artifact has no enforcement.

So the close is not now you know.

Two things to carry into Monday. When a status report shows a smooth ramp of small wins on a mature pipeline, that is not a reason to relax. It is a prompt to ask what axis the team is on. And when somebody shows you a big margin, the margin is the least interesting number on the slide. Ask what it was measured against, and ask whether that thing could ever have won. Every leader has approved a deck where the new thing beat the old thing by a mile. Almost none of them asked whether the old thing was allowed to compete.

The one job that does not delegate is holding the goal. And the only version of that job which survives a busy Tuesday is the one that leaves a mark on disk.

Justin Johnson writes Run Data Run. He is the author of Builder Leader (builder-leader.com).

Seven weeks ago, the day Opus 4.8 and Dynamic Workflows shipped, I wrote that the unit of agentic work had moved from one model call to dozens of verified ones, and that the open question was no longer whether the model is good enough. It was who is verifying.

I have been running fan-out as a daily default since. The field taught me things the launch-day post could not.

The economics held

The price math was the argument in May, and it played out the way the arithmetic said it would. Fifty parallel subagents on Fast at $10 in and $50 out stopped being a budget event. A review pass that spawns a skeptic per finding, a discovery sweep that runs five finders with different lenses and takes the union, a batch edit split across ten agents that each own a file. None of those feel like a stunt now. They feel like a Tuesday.

My own standing rule shifted to match. For anything that is a build, I hand the file-level work to subagents by default and keep the main context for the plan, the spec, and the diff review.

I do not decide to fan out. I decide not to, and only when there is a reason.

So the call on cost was right, and it was the easy call. The arithmetic was visible at launch.

The dispatch problem is still in my head

The harder prediction was the one I was least sure of. I said the question of when a workflow is the right shape, and when a single careful pass is, was unsolved and lived in your head. Seven weeks later it still lives there.

ultracode, the setting that makes Claude reach for a workflow on every task without being asked, still ships off. There is still no governance layer that says do not orchestrate this one. So the judgment of when to spend forty agents and when to spend one is mine, made fresh each task, and I get it wrong in both directions. I have fanned out a rename that a single pass would have finished cleaner and faster, and I have run one careful pass on a discovery job that wanted five blind finders and missed a third of the surface. Neither mistake announces itself. The over-orchestrated one costs more; the under-orchestrated one returns a confident, incomplete answer.

The tooling got cheaper. The taste did not.

The footnote became the fight

The part I could not see in May is the one that has cost me the most.

I called the verifier the priced skill and warned the tooling for one was thin. What I did not know is that the harness puts a hard ceiling on the verifier's ability to do its job. Every subagent is capped at 8,000 output tokens per response, and the model's own thinking counts against that budget. Set effort to xhigh, which is the Claude Code default, and an adversarial verifier asked to reason hard about whether a finding is genuine will think its way past the ceiling and die before it emits a single word of verdict. The tool call comes back with zero output. No finding, no error, no crash. Silence that reads exactly like a clean pass.

A verifier panel that looks like it ran and returns nothing is worse than no panel, because you trust the green. This is the failure I warned about in May, arriving through a door I did not know was there. The launch post said a fan-out that forwards fifty unverified findings is worse than the single pass it replaced.

A fan-out whose verifiers silently no-op forwards zero findings and tells you everything is fine.

The fix is mechanical once you know the ceiling exists. Bound what each agent writes, chunk large outputs across several small responses, and dial effort down for the producers so the verifiers have budget left to reason. But you have to know it is there. Most builders reaching for their first verifier panel do not, and the default settings hide it from them.

What I run now

Effort is not one dial for the whole job. The producers, the agents writing code and editing files, run low, because their work is mechanical and the reasoning tax buys nothing. The judges and verifiers keep the high effort, and I split their task so each verdict fits inside one response instead of one giant pass that overflows. The verifier gets a real design, not a one-line prompt bolted onto the end of the workflow.

In May the verifier was a line item. Now it is the thing I spend design time on.

The verifier is the only part of the pipeline that fails without telling you.

The frame still holds

The bridge between one model call and an agentic system is now a tool the model writes for itself, and the frameworks that were charging for that bridge have a lower ceiling on what they can charge. The vendor who ships the verifier primitive first still sets the pattern everyone copies.

I would add one clause to the closing line I wrote then. If the price of careful goes down, the price of casual goes up, and the default question stops being is the model good enough yet. It becomes who is verifying, and can your verifier afford to think.

Around the Corner: short reviews of ideas worth watching. Opt-in section, not part of the weekly Run Data Run email. [Subscribe to the main list](https://rundatarun.io/subscribe) for longer essays.Subscribe to the main list for longer essays.*

On July 16, Boris Cherny published a table. He helped build Claude Code at Anthropic, he talks to engineers at other companies most days, and he says he keeps hearing the same story: one person is 10x'ing their output with Claude, and the rest of the org hasn't caught up. So he mapped what he sees. Five rungs, zero through four.

Eight hours before that post, Amjad Masad, who runs Replit, described something from inside his own company. The same engineers had 3x'd their output in six months. Support was resolving its hardest tickets 60 percent faster. He has a name for the shape he thinks he is watching: the self-driving company.

Fifteen days before either of them, three researchers at Microsoft published a number.

They studied tens of thousands of engineers through the company's early-2026 rollout of Claude Code and GitHub Copilot CLI. Four months, real telemetry, no survey. Engineers who adopted the tools merged about 24 percent more pull requests than they otherwise would have, and the lift held steady across the entire window.

Ten times. Three times. Twenty-four percent.

The three are not counting the same thing, and the gap between them is the story. Cherny is reporting what he hears. Masad is describing a company that sells AI coding tools and is staffed by the people who build them. Microsoft counted merged pull requests and told you exactly what it counted, including that "a merged PR is not the same as the value it delivers."

The number falls as the instrument sharpens. Loose quantities are enormous. Precise ones are modest.

What the ladder gets right

The table is serious work, and the criticism only lands if you take it seriously first.

Each rung gets a role, an agent count, a bottleneck, and guardrails. Step 0 is Gated: no access, legacy approvals. Step 1 is Assisted, one agent, you and it as a pair. Step 2 is Parallel, about ten agents, and you become an Orchestrator. Step 3 is Supervised autonomy, roughly a hundred agents, and Cherny calls the role Manager of managers. Step 4 is AI-native, a thousand or more, steering by intent.

Now read the bottleneck column straight down, ignoring everything else.

Your attention. Reviewing output. Trust in the loop and your team's decision throughput. Identifying and automating work at scale.

The model is not in it. Not on one rung. The person who helped build the product, published by the company that sells the model, put out an adoption ladder in which the model is never the thing standing in your way. Every constraint on that table is a human quantity. I have been making this argument for months and I have never had a cleaner statement of it than the one Anthropic just published by accident.

Every constraint on that table is a human quantity.

The mess is not a side effect

Arseny Kapoulkine, who spent years as a technical fellow at Roblox and wrote the tools a lot of the game industry runs on, answered Cherny in one sentence:

"one person is 10x'ing their output with Claude while the rest of the org is busy dealing with the resulting mess"

Several hundred likes, which in this corner of the internet is a room nodding.

Cherny says the org hasn't caught up. Kapoulkine says the org is not standing still. It is cleaning.

That is the part the table leaves out. Output has to land somewhere, and an organization can only absorb so much of it. The limit is not the model's. It is how fast people can read what came out, decide about it, and put their name on it. Push past that line and the extra has two places to go, and neither of them is up.

It queues, and the person having the best month of their career watches it go stale in a backlog. Or it ships unread and becomes what one of Cherny's own readers calls slop. The first teaches your most motivated engineer that speed is pointless. The second teaches everyone downstream that the new work cannot be trusted. A wall or a mess. Pick one.

His readers know exactly where their own line is. One of them no longer reads his code at all and still runs "only 1-4 sessions in parallel because that is my speed of verifying." These are not skeptics. They are enthusiasts at the top of the curve, describing a wall made of their own eyes. A self-driving company whose drivers say they cannot stop watching the road.

Cherny's table concedes it in the step 3 row. The trap, it says, is "scaling agent count before the loop has earned widespread trust." The agent count is the output, not the input. You do not climb by adding agents. The agents show up when something else has already changed, and the thing that changes is how much a person is willing to stop looking.

How it actually moves

Back to the Microsoft paper, because it answers a question the table does not ask: how does any of this spread in the first place?

Not by memo. First use, the authors write, "spread primarily through social networks," and their recommendation to any org attempting this is to treat "visible peer use as central to rollout strategy."

They put numbers on it. An engineer had 54 percent higher odds of trying the tool where a quarter or more of the people they trade code reviews with had already used it. An engineer whose skip-level peers, meaning the engineers who share their manager's manager, were largely using it had 216 percent higher odds. That was the strongest signal in the study.

Sideways. Through the people you already trade work with, and not down through a mandate or a literacy program that 85 percent of employees say they cannot apply to the job they actually do.

Anyone who has watched a transformation program die should find that encouraging. The spread does not have to be installed. It installs itself, through the same social wiring the org already runs on.

Except it needs a light.

The manager number

Microsoft modelled the exact variable. In the paper's own words, a binary indicator of whether engineer i's direct manager used Copilot CLI.

"An engineer whose manager used Copilot CLI had higher odds of both trying it (+82%) and, more modestly, sticking with it (+22%)."

A manager putting their own hands on the tool nearly doubles the odds their reports try it.

Then the other half. On whether managers picked it up themselves, the paper reports that they "looked no different from the reference." No more likely than a mid-level individual contributor. No less.

The one person whose adoption moves everybody else's is no more likely than anybody else to adopt.

So the two findings stop competing and become one machine. The manager's crossing is the ignition. The peer network is the amplifier. Eighty-two percent lights it, two hundred and sixteen carries it, and the reason most orgs have neither is that nothing struck the match.

What spreads on its own

Grassroots adoption works, and it does not need a leader. Look at what it produces when it doesn't have one.

Most enterprises have now found an agent running that nobody signed off on. Gartner projections reported this month have the average Fortune 500 carrying more than 150,000 agents by 2028, up from fewer than fifteen in 2025. The same projections have 40 percent of enterprises demoting or decommissioning agents by 2027, after a production incident.

That is not a step 3 organization. That is a step 0 organization full of step 2 individuals, each one locally optimized, none of it adding up.

Which is Cherny's opening sentence, at scale. Unseeded sideways spread does not disprove his ladder. It manufactures the exact problem he opens with.

Everyone is naming the same top rung

Masad calls it the self-driving company. Cherny's step 4 is AI-native, a thousand agents, the human steering by intent. Back in May, Brian Armstrong told Coinbase he was "rebuilding Coinbase as an intelligence, with humans around the edge aligning it," and cut about 700 roles on the way.

Three serious people, ten weeks, the same shape: the organization drives, and the human moves to the edge.

I think the edge is the wrong place to stand, and the evidence in this piece is why. The enthusiasts running the most agents say they cap at four because of their own eyes. Cherny's own step 3 says the trap is scaling agent count before the loop has earned trust. His entire bottleneck column is attention, review, and trust. Every one of those is a statement about a human being close to the work, not at the edge of it.

Armstrong's memo actually contains the refutation. A few lines under the sentence about humans at the edge, he mandates the opposite: "No pure managers. Every leader at Coinbase must also be a strong and active individual contributor." Player-coaches, he says. Hands dirty. That is the thesis of the book I just published, arrived at independently and written into HR policy instead of a chapter.

You cannot be at the edge and have your hands dirty. He is right the second time.

Teach the teachers

I ran a version of this at a Fortune 500 pharma, across a coding-agent rollout to a large technical organization, and the pattern that worked was not a program.

It was: cross first, personally. Build something real in your own domain, badly, and then less badly. Then teach the handful of people closest to you, not by presentation but by showing them the thing and the scars on it. Those people teach the people next to them. The distance the idea travels from you is short. The distance it travels after you is the whole org.

You are not the distribution channel. You are the ignition source, and then you get out of the way of a network that moves faster than you can. One is you. The other is what happens next.

The two objections

Two arguments cut against this, and both deserve better than a wave.

You can buy it. The day after that Microsoft paper went up, Microsoft stood up a $2.5 billion, six-thousand-person company whose entire job is making enterprise AI deployments work. Its pilot cut a supply-chain Copilot rollout from fourteen months to five. Aaron Levie thinks the implementation work ahead "will exceed anything we imagine today." If capability can be imported wholesale, then the ceiling is your budget and your vendor list, not whether anyone senior has personally built anything.

Governance may matter more than any leader's history. A survey of 157 enterprises found half had shipped an agent that passed internal evals and then failed in front of a customer. Only 5 percent fully trust automated evaluation. Sixty-six percent are engineering toward zero human in the loop anyway. Their phrase for it is that the autonomy is arriving faster than the assurance, and none of it cares who approved the deployment.

Both are true, and I am not waving at the second one. I spent a whole piece ten days ago arguing that in a regulated industry the approval queue, not the keyboard, is what actually holds you back: AI has eaten the keyboard, it has not eaten the queue. Governance is necessary. It is not sufficient, for a small and unglamorous reason: someone has to be able to tell whether the governance is pointed at anything real. A person who has never run the loop cannot tell a working verification stack from a slide that says Verification. They will approve the slide. They approve the slide constantly. That judgment is not a policy you can buy.

The rung you set

Cherny's ladder is not wrong. It is aimed.

It is written for the engineer climbing it, which is the right audience for the person who built Claude Code, and it is useful if that is you. The person it never addresses is the one who decides how far it goes. Look at his own step 0, the bottom rung, the one whose stated bottleneck includes a "lack of true technical voices in decisionmaking." The exit condition he lists is executive and buyer alignment.

His table already says the way off the floor is a person with authority. It just doesn't say that person has to have used the thing.

None of this means an organization can't evolve without you. It plainly can, and most enterprises are proving it in the shadows right now. It means something narrower and more uncomfortable: if nobody senior ever crosses, the org can stall at the rung where its leaders stopped, and what it accumulates instead of capability is mess. No published adoption ladder I found this week models falling. Every one of them models climbing.

You already know which rung you're on. The number you don't have is the other one: what your reports' odds would be if you were on it.

Sources

• Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI (Murphy-Hill, Butler, Savelieva, July 1, 2026). Tens of thousands of engineers, four months, the 24 percent lift and the social-exposure numbers.

• Boris Cherny, Steps of AI Adoption (July 16, 2026) and the post announcing it.

• Amjad Masad on Replit's six months (July 16, 2026), where the 3x, the 60 percent, and the self-driving company come from.

• Arseny Kapoulkine's reply (July 16, 2026).

• Microsoft's $2.5 billion, 6,000-person AI implementation unit (CNBC, July 2, 2026).

• Why AI readiness training fails (HR Dive, on Docebo's 2026 AI Readiness Gap report, 2,000 respondents across six countries).

• Related, from me: The Harness Is the Moat on why the model is the commodity layer, Ep. 1: Two Groups on the population split underneath all of this, and You Don't Have to Write the Code on what Anthropic's 400,000-session study found actually predicts success.

Justin Johnson writes Run Data Run. His book on crossing this particular gap is Builder Leader (builder-leader.com).

A few weeks ago I argued that the harness is the moat: the system you build around the model is the part nobody can copy, and the model itself is a commodity you order off a price sheet. Today I want to go one layer in, to the thing that runs inside the harness. The people building fastest will tell you it is the whole ballgame.

They call it the loop, and it is suddenly everywhere. Anthropic shipped it as a product feature this summer. A conference talk naming it went around the engineering world. The slogan it produced gets repeated like scripture: the winners will not have the smartest model, they will have the best loop.

The clearest signal is the person who built Claude Code. Boris Cherny says he has not written a line of code by hand in eight months, and that is not the part he finds remarkable. Going from writing code to prompting a model was the small shift. The big one is going from prompting to loops: he no longer types instructions, he writes loops that prompt the model for him, and his whole job is building and steering them. Some mornings he is managing a few hundred agents, some days thousands.

And it works. Anthropic now ships a real share of its own code this way, at scale, and says so on the record. A team points a loop at a written spec and lets it grind through a feature overnight, then reviews a pull request in the morning instead of writing one. The wins are not hypothetical, and that is why the idea caught fire. If you want the full case, the sources, the skeptics, and the cost numbers nobody pitches, I ran a 30-day sweep across every platform where people argue about AI and wrote it up as the companion to this piece. What follows is the argument it grounds.

Here is what the slogan tends to leave out. The loop is the easy part. It is a small, almost silly mechanism that you could understand in the next five minutes and copy in an afternoon. Everything that decides whether it helps you or quietly sets fire to your budget lives around it. So let me do two things: show you what the loop actually is, in plain terms, and then hand you the three questions that separate a loop worth running from one that runs you.

The loop is the easy part. The hard part is everything bolted around it.

What a loop actually is

Start with the thing itself, because it is simpler than the vocabulary around it.

A coding agent runs inside a plain repeating cycle. You hand it a written spec, a page describing what you want built. It does one small task, saves the result to a file, records that step, and then you throw away everything it was thinking and start a fresh copy from scratch. The fresh copy reads the same spec, reads the files the last one left behind, sees what is already done, and picks up the next task. Around and around until a check you wrote says the work is finished.

That is the whole trick. The engineer who named it, Geoffrey Huntley, called it Ralph, after Ralph Wiggum, precisely because it looks too dumb to work. No memory. No accumulation. A cheerfully oblivious worker waking up new every pass, with no idea it has been at this for hours.

And it works because it is dumb. A fresh worker never drowns in its own earlier confusion. The reason your long chats with an AI start to drift and contradict themselves is that the context fills with everything said so far, and the model loses the thread. The loop sidesteps that by refusing to carry a thread at all. It keeps the progress somewhere safer than the model's memory: in your files and your saved history, which do not get wiped.

That is the one idea to carry out of here. The model's memory is scratch paper. The real state of the work lives on disk. Once you see that, the loop stops being magic. It is a short block of code with a model call inside, about six lines, and every serious version lands on the same tiny shape. There is no proprietary one to buy. This is the hand-rolled hack Anthropic just turned into a button: Claude Code's /loop runs a command on a cadence, and /goal sets the condition that tells it to stop. The bash trick became a supported feature, which is usually the sign something has gone from clever to standard.

You would want one because it changes what a person does all day. You stop producing the work and start describing it and checking it. That shift, from writing the thing to knowing what to ask for and telling whether you got it, is the one Anthropic found predicts who succeeds across 400,000 sessions. You write down what "done" looks like, point the loop at it, and come back to review instead of to type. That is useful, and it is also where every problem starts. A thing that works unattended for hours is a thing making decisions you are not watching. Which brings us to the part the slogan skips.

If the loop is settled and small, the interesting question is not how to build a better loop. It is three questions about the system around it.

The first question: how do you know it worked?

The single most important piece of a reliable loop is the part that can tell it no.

An Anthropic engineer described the pattern: before the agent writes a single line, two agents negotiate what "done" means, and a third exists only to check the work against that definition. The loop does not run until there is a test it can fail. The failing test carries the weight, not the loop itself. You build the thing that says no first. The loop comes after.

The reason to build it first is a piece of arithmetic that belongs on a slide in every project like this. Melanie Warrick, who works on this at Temporal, ran the numbers: even if every step in a job succeeds 85% of the time, a ten-step job finishes correctly only about 20% of the time. The failures multiply. Miss one step in five, ten times running, and a clean pass becomes rare. The gap, she found, is not intelligence. The agents "diagnose a failure in perfect detail and still do nothing to recover from it." They describe the wall in detail and walk into it anyway.

It gets worse inside a single run. A model's attention decays as the loop goes on, and there is now a benchmark built to predict where it breaks. One study found a rule honored 73% of the time early had fallen to 33% by sixteen steps later. The agent did not rebel. It forgot, the way a tired person forgets, and kept going with the rule half-erased. The fix is unglamorous: re-state the hard constraints on every pass, and never assume a rule set once stays set.

None of that is loop engineering. It is checking-the-work engineering, and it is most of the actual job. A loop with no part that catches a bad result is not a loop. It is a fast way to be wrong at scale.

Even at 85% success per step, a ten-step job comes out clean only about one time in five.

The second question: what did it cost?

The economics of a loop break the way most people budget for software, and the people who learned this learned it expensively.

A chat costs you a sentence at a time. A loop runs for hours, calls the model hundreds of times, and has no natural stopping point unless you give it one. So the unit changed underneath you. It is no longer cost per question asked. It is cost per finished piece of work, and that number is wild.

One study measured the same task, same model, same prompt, same everything, and found it could cost eight dollars or two hundred forty. Thirty times apart, with nothing changed on the input, the spread tracing to how the loop was set up rather than which model ran it. The cost of letting a loop run is not a price you can quote in advance. It is a range, and the high end is far away.

This is not theoretical. Uber, by one widely-shared account, burned its entire annual AI coding budget in four months running agent loops, then capped its engineers at fifteen hundred dollars a person. The production version of the loop is not the demo that builds an app while you sleep. It has a spend cap, a meter, and someone who gets a phone alert when the meter spins.

Then the part that should give anyone budgeting for this pause. Checking the work can cost more than doing it. One team spent around forty thousand dollars just on the runs to verify their results, and noted that checking rigorously enough to fully trust them would push into the hundreds of thousands. If the whole promise of the loop rests on a reliable way to confirm it is done, and confirming "done" at scale is the most expensive line on the page, that is a cost nobody put in the pitch.

The answer the field is settling on is sensible. Run most of the loop on a cheap, fast model, and save the expensive top-tier model for the few hard steps that actually need it. Most steps do not need the smartest model in the building. You pay top-tier prices only where top-tier reasoning earns them.

That rule hides a deeper one, and it is the subject of my next essay, Nobody Saves Money on the Model: a cheap model is not a discount, it is a bet that you removed the ambiguity before the loop ever ran. The saving was never in the price per call.

The third question: what is it allowed to do?

The last piece is the one the "no human needed" crowd skips, and it is the one that ends careers.

A loop that runs for hours without supervision is, by definition, taking actions you are not watching. Late last year an autonomous coding agent, asked to clear out some temporary files, wiped a user's drive instead. It did what an unbounded loop does: it took an action it could not undo, in perfect confidence, with no gate in front of it. The failure was not that the agent was dumb. It was fast, capable, and unsupervised all at once, which is a different and worse problem.

The fix is structural, and it looks the same every time: checkpoints, hard limits on how long it can run and how much it can spend, and a human sign-off on anything the agent cannot take back. The companies actually running these in production are not running pure AI. They run a mix: the model reasons, ordinary tested code does the irreversible parts, and a person approves the steps that matter. One builder put it plainly. Most agentic loops are not autonomous. They are automated failure.

There is a newer worry above even that. An agent with standing permission to act is a kind of insider you have never had on the payroll. It holds credentials, runs unattended, and can do in a loop at three in the morning what a confused employee could only do at a desk by day. The risk was never which model you use. It is what permissions the loop carries, and whether you can pull them back in a hurry.

This is why the adoption numbers tell a sober story. By one survey, 79% of companies have adopted these agents and only 11% run them in production. Gartner projects 40% of agentic AI projects will be canceled by 2027, and the reasons are not "the loop didn't work." They are cost, unclear payoff, and the governance gap above. The loop is the easy part. Everything I just walked through is the distance between a demo and a deployment.

An agent with standing permission is an insider that never sleeps. The only question that matters is whether you can take the keys back.

Where this leaves you

Here is the whole thing in one breath. The loop is a dumb little cycle that keeps its progress on your hard drive instead of in the model's head. You could run one. You might well want to, because it turns making the work into describing and checking the work. And whether it pays off has almost nothing to do with the loop and almost everything to do with three answers most teams cannot give.

Do you know when it worked? Do you know what it cost? Do you know what it is allowed to touch?

That is not a coding question, which is the good news for anyone reading this who does not write code. Knowing what good looks like, catching a confident machine being wrong, deciding what it may and may not do without a human in the room: that is the judgment AI didn't make cheap. A loop does not supply it. A loop spends it.

A year from now your competitor will run the same six-line loop you do, against a model that costs about what yours does. Both will be things you order off a shelf. So the question is not which loop to build. It is whether, when your loops are running overnight, you can answer those three. Most teams cannot, and that, not the loop, is the work of this year.

The receipts behind this argument, the sources, the skeptics, the cost numbers, and the one stat I refused to print, are in the 30-day sweep linked up top. Run Data Run is free. If this was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

About Last 30 Days. Cross-platform research sweeps on topics worth paying attention to. Every post pulls Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, then synthesizes what people are actually saying, building, and betting on. Topics get picked when the signal is high and the story is contradictory, when a single headline would lie about the shape of what's happening. Each post follows the same arc: one specific finding that earns the click, why the topic deserves a sweep right now, the themed synthesis with inline citations, and the follow-up threads worth watching next.

Today's Run Data Run post, The Loop Is Simpler Than It Sounds, rests on a 30-day sweep across every platform where people argue about AI. This is the companion: the raw shape of what came back, including the parts that did not fit the headline. If the post is the argument, this is the receipts.

The single most useful thing the sweep found was not a fact. It was a contradiction. The loudest claim of the month, that "loop engineering" is the new moat, sits directly on top of an older, quieter body of evidence saying the exact opposite: that loops fail in ways the hype never mentions, and the people who learned that learned it expensively. Both are real. A single headline would have to pick one and lie about the other.

Why this topic deserves a sweep

In June, "the loop" went from a clever bash trick to the most over-discussed idea in the field. Anthropic shipped it as a product feature. A conference talk produced a slogan, "the winners will not have the smartest model, they will have the best loop," that got repeated everywhere. Boris Cherny, who built Claude Code, said he has not hand-written code in eight months and now spends his time writing loops, some days managing thousands of agents.

That is one half of the story, and if you only read X you would think it was the whole thing. The sweep is what surfaces the other half: a counter-current nearly as strong, mostly predating the viral wave, made of cost blowups, reliability math, and production failures. You cannot read both halves and conclude one thing. That is exactly when a sweep beats a headline.

The moat thesis, as it actually spread

The viral layer traces to one talk. The highest-engagement post of the set put the slogan plainly (@AnatoliKopadze, 8,624 likes, 1.7M views), describing Anthropic running "three agents: one to plan, one to build, one to judge, cycling until the app actually works." The clearest single statement of the mechanic came from a practitioner, Akshay Pachaar: "the loop itself is six lines, and nobody competes on it. every serious agent framework lands on the same tiny while-loop."

The technique has a name and an origin: the "Ralph" loop, coined by Geoffrey Huntley after Ralph Wiggum, because it looks too dumb to work. And it has a button now, Claude Code's native /loop and /goal commands, which is usually the signal that something has gone from clever to standard.

The one stat I refused to print

Here is a transparency note worth making, because it is the kind of thing a sweep catches and a single source does not.

The most-quoted number in the whole topic, Anthropic's internal loop-adoption figure, does not survive the sweep. Three different posts attribute three different numbers to what appears to be the same talk: "over 30% of code" written through loops, then "70 to 80% of engineers" using them, then "90%." When one stat arrives at three sizes from one source, it is folklore, not data. So the post states the direction (Anthropic is clearly building this way, at scale, on the record) and explicitly tells you to treat the decimal point as noise. That call only gets made because the sweep put the three versions side by side.

The counter-current the slogan skips

This is the densest part of what came back, and the part the hype leaves out.

• The math. Melanie Warrick at Temporal ran the numbers: even at 85% reliability per step, a ten-step workflow finishes correctly only about 20% of the time. Failures multiply. She found the gap is resilience, not intelligence; agents "diagnose a failure in perfect detail and still do nothing to recover from it."

• The cost. An arXiv study found the same task, same model, same prompt, could cost eight dollars or two hundred forty, thirty times apart with no input change. A separate $22,000 sweep pinned a 33-fold spread to scaffold choices. And one widely-shared account had Uber burning its annual AI coding budget in four months before capping engineers at $1,500 a head.

• The verification trap. A benchmark of agentic systems spent roughly $40,000 on evaluation runs alone, and noted that doing the checks rigorously would push it into the hundreds of thousands. If the whole premise of a loop is a verifiable done-condition, and verifying "done" at scale is the single most expensive line, that is a price nobody put in the pitch.

• The adoption gap. By one survey, 79% of enterprises have adopted agents and only 11% run them in production. Gartner projects 40% of agentic AI projects will be canceled by 2027, on cost, value, and governance, not on the loop failing.

Gary Marcus and Arpit Bhayani carried the skeptic load on the social layer, both before the June wave crested. The reliability and cost objections were live before the moat framing peaked, which is itself a signal.

How the sweep is built (the method)

A quick word on method, since the point of this section is to show the work.

The sweep pulls each platform on its own terms: Hacker News and Reddit for builder substance in the comments, X for engagement-weighted reach, a curated corpus and a Substack named-voice pool for depth, and a separate open-ended discovery pass whose only instruction is "surprise me, do not confirm the thesis." That last pass is what surfaced the counter-current; left to its own framing, a research pass confirms what you already believe.

Two integrity habits matter. Every cited cost figure traces to a primary source, not a tweet about a source. And the fetch pass flags its own failures: of the URLs pulled this round, several came back as bot-challenge pages or 404s and were routed around rather than quoted. A claim that only survives in one blocked link is not a claim.

What I'm watching next

• Whether the Anthropic adoption number ever gets pinned to a primary transcript, or stays folklore.

• Where exactly the "coherence cliff" bites, by task length, token count, or iteration count. Everyone asserts it; nobody has located it.

• Whether the three-agent plan/build/judge pattern survives the move from greenfield demos to brownfield maintenance, where the stale-state failures are worse.

• The shift the researchers are naming: the agent stops being a tool that runs your code and becomes the software, with the human as "intent architect." If that holds, the unit of cost moves from per-token to per-finished-artifact, and most budgeting models break.

Sources and method

Full synthesis and the per-platform raw dumps live in my research vault; the post, The Loop Is Simpler Than It Sounds, is the argument this sweep grounds. Method: a 30-day cross-platform sweep (Reddit, X, YouTube, Hacker News, the web, a curated corpus, a named-voice pool) plus an open-ended discovery pass, two-pass synthesis, every quantitative claim traced to a primary source. Counts and quotes come from the actual sweep output, not memory.

Last 30 Days is a research series on Run Data Run, posted alongside the occasional Deep Dive when a topic earns the deeper look. No email on these, they live on the site for when you want the receipts behind an argument. If a sweep was useful, subscribe and you'll get the next one as it lands.

About Last 30 Days. Cross-platform research sweeps on topics worth paying attention to. Every post pulls Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, then synthesizes what people are actually saying, building, and betting on. Topics get picked when the signal is high and the story is contradictory, when a single headline would lie about the shape of what's happening. Each post follows the same arc: one specific finding that earns the click, why the topic deserves a sweep right now, the themed synthesis with inline citations, and the follow-up threads worth watching next.

The hook

A prototype of Biomni, Stanford's autonomous biomedical research agent, was already running in more than 10,000 laboratories before its paper appeared in Science on 9 July.

Hold that number next to the thing everyone actually celebrated this month, which was the paper.

That gap, between when these systems became real and when the record admitted it, turned out to be the shape of this entire sweep.

Why this topic deserves a sweep right now

FutureHouse's Robin reached *Nature* this week. It is a genuinely significant result: given the name of a disease, it proposed a therapeutic strategy for dry age-related macular degeneration, identified ripasudil (a glaucoma drug never proposed for dAMD), confirmed it in cells, and then designed and analyzed its own follow-up experiment. I wrote about it properly in this week's Sunday Deep Dive.

Reading it sent me back to survey the whole category, because something felt off about the timing. It was. Robin's preprint went up in May 2025. The paper printed in July 2026.

So I ran a thirty-day sweep across X, Hacker News, the curated wire, and the web. What came back was not a story about one paper. It was a story about a field that has already moved somewhere the published record has not caught up to, and about a body of skeptical research that landed in the same window and got almost no attention at all.

The sweep

Theme 1: they all landed at once

Four flagship autonomous-discovery systems cleared peer review inside a single four-month window.

• [Robin](https://www.nature.com/articles/s41586-026-10652-y)Robin** (FutureHouse) → Nature, July 2026. Multi-agent, lab-in-the-loop. Three specialised agents: Crow (concise literature search), Falcon (deep literature search), Finch (data analysis). Found ripasudil and KL001 for dry AMD, then surfaced ABCA1 as a possible novel target via its own RNA-seq follow-up. From the abstract: "All hypotheses, experimental directions, data analyses and data figures in the main text of this report were produced by Robin."

• [Google DeepMind's AI co-scientist](https://research.google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist/)Google DeepMind's AI co-scientist** → Nature, 19 May 2026. Gemini-based multi-agent system. Its headline validation is the same move Robin made: read the literature, propose an old drug for a new disease. Theirs was KIRA6 for acute myeloid leukemia, which inhibited AML cell viability at clinically relevant concentrations. Also went after liver fibrosis targets and antimicrobial resistance.

• [Sakana's AI Scientist](https://sakana.ai/ai-scientist-nature/)Sakana's AI Scientist** → Nature, 25 March 2026. The most radical of the four and the least grounded in wet biology: it runs the whole pipeline through to a finished manuscript, then peer-reviews itself. A paper it generated passed the first round of human peer review at a top machine-learning workshop. Its most interesting finding is a scaling law of AI science: generated-paper quality rises with the underlying model and with inference-time compute.

• [Biomni](https://news.stanford.edu/stories/2026/07/biomni-ai-powered-biomedical-co-scientist)Biomni** (Stanford) → Science, 9 July 2026. "Autonomous biomedical research with an artificial intelligence agent." Generalises across causal gene prioritisation, drug repurposing, rare-disease diagnosis, microbiome analysis and molecular cloning with no task-specific tuning. Already in 10,000+ labs.

The cross-system pattern was spotted in the wild and it is worth quoting, because it is the whole architectural story in one line: "Across Claude Science, NVIDIA BioNeMo, and FutureHouse Robin, the same pattern keeps appearing: specialized agent skills, orchestrated research workflows, domain-specific execution."

Nobody trained a discovery model. Everybody built a harness.

Theme 2: the lag is the actual story

Every one of those systems was old news by the time it was printed.

• Robin: preprint 19 May 2025 → Nature July 2026. 14 months.

• Sakana's AI Scientist: arXiv 12 August 2024 → Nature 25 March 2026. 19 months.

• Google's co-scientist: announced 19 February 2025 → Nature 19 May 2026. 15 months, to the day.

• Biomni: bioRxiv 30 May 2025 → Science 9 July 2026. 13 months.

Mean lag: roughly fifteen months.

The sharpest write-up of this belongs to synthetic biologist Jake Wintermute, on Biomni:

"Biomni was on arXiv 13 months ago. Biomni was on GitHub 11 months ago. Phylo, the company built on Biomni, raised $13.5M and launched 5 months ago. Or, I guess, you could read about it in Science Magazine today."

None of this is a criticism of the labs. They all posted preprints immediately, which is exactly right. The lag belongs to the journals.

Theme 3: where the field actually is

Not where Nature says it is. While Robin was in review, FutureHouse built its successor and then built a company around it.

• [Kosmos](https://arxiv.org/abs/2511.02824)Kosmos** (arXiv, November 2025): runs up to 12 hours across ~20 cycles. A single run reads ~1,500 papers and writes ~42,000 lines of code. Independent scientists judged 79.4% of the statements in its reports accurate. It has produced seven discoveries, three reproducing unpublished findings and four net-new. Collaborators estimated one run did about six months of their own work. Sam Rodriques' announcement did 3,683 likes, the highest-engagement item in the entire sweep.

• Edison Scientific: FutureHouse's for-profit spinout, commercialising Kosmos. And this month, its first partnership using Kosmos not to write papers but to launch new biotech companies.

• [Sakana Marlin](https://x.com/hardmaru/status/2066529282588094713)Sakana Marlin**: Sakana's first commercial product. An autonomous research agent that runs ~8 hours unattended and emits structured slides plus a multi-dozen-page report. Pitched as a virtual chief strategy officer, aimed at finance and consulting rather than the bench.

• [Claude Science](https://www.anthropic.com/news/claude-science-ai-workbench)Claude Science** (Anthropic, 30 June): the workbench end of the same idea. I wrote about it here.

Read the journals to learn what was true a year ago. Read the preprints and the launches to learn what is true now.

Theme 4: the counterweight nobody is reading

This is the highest-value material in the sweep and it appeared in almost no coverage.

[Correct Answer, Wrong Mechanism](https://arxiv.org/pdf/2606.23175)Correct Answer, Wrong Mechanism** (arXiv 2606.23175). Subtitle: When AI Scientists Defend General Claims Their Own Data Contradicts.

Researchers watched a coding agent attempt to rediscover a known particle-physics result across 28 episodes. It often got the right answer. The problem was how:

• CAWM occurred in 4 of 20 (20%) primary-model episodes and 3 of 8 (37.5%) episodes on other frontier models.

• The agent reached right-looking results through reasoning that collapses when conditions change.

• When pressed, it defended the wrong mechanism, arguing for physics inconsistent with the numbers in its own output.

• Verdict: these systems are dependable as tools but "unreliable scientific co-authors for open-ended claim-making."

• The demand: outcome-only evaluation is insufficient. Score task outcome, mechanism fidelity, and epistemic honesty as three separate things. Their lightweight checks flagged every CAWM case in the study.

[Related work](https://phys.org/news/2026-05-ai-scientists-reveal-fundamental-limits.html)Related work** (May 2026) catalogues the failure modes of autonomous research pipelines with unnerving specificity: implementation bugs, hallucinated results, shortcut reliance, methodology fabrication, citation hallucination, frame-lock, and bug-as-insight reframing, in which a system trips over a defect in its own code and writes it up as a discovery. The finding: increased quantity, decreased quality, of both papers and reviews.

And the view from the bench. Working scientist Emma Chory, on Biomni cheerfully producing a BSL2-plus lentivirus protocol with, in her words, "sufficient detail to execute on a robot." Her review ran to four words: "Cool cool cool cool." An agent that will write you a biosafety-level-2-plus procedure on request is a governance question, not a capability win.

Theme 5: the taxonomy worth stealing

Computational biologist Mile Sikic drew the distinction that most of the coverage blurs. There are two streams:

1. Virtual bioinformaticians that combine existing knowledge with existing tools. (Claude Science, Biomni, most of the field.)

2. Systems that discover new biology in close collaboration with wet-lab scientists. (Robin, Google's co-scientist.)

Conflating them is how people end up disappointed. They are solving different problems.

Also in the window, at lower weight: [EurekAgent](http://arxiv.org/abs/2606.13662)EurekAgent** ("Agent Environment Engineering is All You Need for Autonomous Scientific Discovery"); Yoshua Bengio's take on an AI Scientist; the Samsung SAIT AI Scientist competition winners; OpenScience, an open coding agent "that went to grad school," ~350 GitHub stars in its first week; and a genuinely useful sign of a crowded category, a piece titled "Which 'AI scientist' suits your lab? A guide for the perplexed."

What I'm watching next

Edison Scientific's biotech-founding partnership. Using an AI scientist to found companies is a categorically different claim from using one to write a paper, and unlike a paper it will be tested in public, on a clock, with other people's money.

Whether anyone adopts the CAWM evaluation protocol. The paper's demand is concrete and cheap: score mechanism fidelity separately from outcome. If the next generation of agent papers still reports only "did it get the right answer," the field has decided not to look.

Ripasudil. It cleared cells, not patients. A disease model comes next, and then, eventually, a randomized controlled trial. Most compounds that look this good at this stage never finish.

Sources

Primary papers: Robin / *Nature* · Robin preprint · Kosmos · Sakana AI Scientist · Correct Answer, Wrong Mechanism · EurekAgent · Biomni / Stanford · Google AI co-scientist · Claude Science

Voices: @SGRodriques · @FutureHouseSF · @SynBio1 · @chorye · @msikic · @hardmaru · @aipoch_ai

Methodology: thirty-day sweep across X, Hacker News, the curated AI wire, and the web, run 12 July 2026. The X leg of my usual tooling was down (an expired API credential), so X was swept through a working session-based client instead; Reddit returned no topical signal and was discarded rather than padded. Engagement figures are as recorded at collection time. The full deep dive on the Robin paper itself is [here](https://rundatarun.io/p/the-year-nature-caught-up).here.*

The Headline

A multi-agent AI system named Robin was given the name of a disease and told to find a treatment. It read the literature, proposed a therapeutic strategy, named a drug, watched humans test that drug at the bench, analyzed the results itself, and then designed its own follow-up experiment. The disease was dry age-related macular degeneration, the major cause of blindness in the developed world. The drug was ripasudil, a rho-kinase inhibitor that has been sitting in pharmacies for years, approved for glaucoma, and never proposed for dAMD by anyone, as far as the authors could tell or I could find.

It worked in cells. Then Robin asked why, specified an RNA-sequencing experiment to find out, analyzed that too, and surfaced a gene called ABCA1 as a possible new target nobody had been looking at.

This is the work of FutureHouse, and it was published in Nature this week. There is one sentence in the paper that deserves a second read:

"All hypotheses, experimental directions, data analyses and data figures in the main text of this report were produced by Robin."

Not assisted by. Produced by.

The preprint went up in May 2025. Nature published it in July 2026. Fourteen months, and almost none of the coverage mentions it.

Once you notice that gap, you start seeing it everywhere, and the shape of this entire field changes.

The Paper

Three birds and an orchestrator

Robin is not a model. Nobody at FutureHouse trained a "discovery model," and if you go shopping for one after reading the coverage, you will not find it.

What they built is a system of specialized language agents sitting on top of ordinary commercial models:

• Crow runs fast, concise literature searches.

• Falcon runs deep ones.

• Finch analyzes raw experimental data.

Robin orchestrates the three of them and carries the hypothesis across rounds, so that what Finch learns on Tuesday reshapes what Falcon goes looking for on Wednesday.

The intelligence came off a price sheet. The discovery came from the scaffolding around it.

I wrote two weeks ago that Anthropic shipped a workbench, not a miracle, and that the harness is the moat. Robin is that argument with a Nature paper attached to it, which is a considerably stronger position than my say-so.

The word "semi" is doing honest work

The paper describes Robin as semi-autonomous, and the precision is a credit to the authors rather than a hedge.

Humans still run the wet lab. People cultured the retinal cells, ran the assays, and pipetted the compounds. What Robin did was everything on either side of the bench: read the field, form the hypothesis, specify the experiment, interpret what came back, and revise.

The term of art is lab-in-the-loop, and it is worth flipping around, because the phrase misleads people. The AI is not in the lab. The lab is in the AI's loop.

Previous systems could do one arc of that circuit. They could read and propose. Or they could take your data and analyze it. Robin ran the full turn, then fed its own results into the next turn, and did it again.

The paper calls Robin "one of the first" systems to do this. The preprint called it "the first." Somewhere in fourteen months of review a superlative got sanded off, which tells you something useful about the claim and something more interesting about the process.

Why the drug was findable at all

FutureHouse call their method combinatorial synthesis, and their honesty about what it means is why I trust the rest of the paper.

Robin did not invent new biology. Every piece was already in print. That rho-kinase inhibition boosts phagocytosis in retinal pigment epithelium cells: known, and they cite the work. That this phagocytic housekeeping declines in AMD patients: also known. The two facts lived in different literatures, read by different people, and nobody had put them next to each other and said the word ripasudil.

The authors have a devastating example of how long that gap can persist. Dabrafenib is a cancer drug whose molecular action was characterized by 2010. Ten years later, a brute-force screen discovered it protects against hearing loss. That protective effect follows directly from the mechanism everyone already knew. The answer sat in print for a decade because the people who knew about BRAF inhibition and the people who cared about hearing loss were not the same people.

Robin is not doing science we could not do. It is doing science we did not get around to.

Scope the claim that way and it stays large. As the authors point out, novel FDA approvals have been flat at roughly fifty a year for a decade. If a machine can reliably close the distance between what is known and what is connected, that is worth a great deal of money and a great deal of eyesight.

The two things I would copy tomorrow

They checked their own judge. Buried in the supplement is a comparison of their LLM evaluator against human experts. I read a lot of agent papers that grade themselves with a model and never once ask whether the grader is any good. This one asked.

Their guardrails are the most thought-through of anything I read this month. Robin preferentially proposes compounds with established safety profiles. Every output is treated as a hypothesis entering standard preclinical review, never as a finding. And the discussion says the quiet part plainly: ripasudil "would of course require validation in a suitable disease model and ultimately in a randomized, placebo-controlled trial."

In vitro is a beginning. The authors know it, and they wrote it down.

The Ecosystem

Robin made me want to go back and re-survey this whole category, so I did. Thirty days of papers, launches, funding, and argument. What I found reframed the paper I had just finished reading.

Everyone landed at once

Four flagship systems cleared peer review inside about four months.

Google DeepMind's AI co-scientist reached Nature in May. It is a Gemini-based multi-agent system, and its headline validation is the same move Robin made: read the literature, propose an old drug for a new disease. Theirs was KIRA6 for acute myeloid leukemia, which inhibited the viability of AML cells at clinically relevant concentrations. They also went after liver fibrosis targets and antimicrobial resistance.

Sakana's AI Scientist reached Nature on 25 March. It is more radical in ambition and less grounded in wet biology: it runs the entire pipeline through to a finished manuscript and then peer-reviews itself. A paper it generated passed the first round of human peer review at a top machine-learning workshop. Its most interesting result is a scaling law of AI science: the quality of the papers rises with the quality of the underlying model and with the compute you spend at inference.

Biomni, out of Stanford, reached Science on 9 July, under the title "Autonomous biomedical research with an artificial intelligence agent." Hold on to one detail from it, because it is the most telling number in this entire piece: a prototype of Biomni was already running in more than 10,000 labs before the paper printed.

The lag is the actual story

Every one of those systems was old news by the time it was printed.

Robin: preprint May 2025, Nature July 2026. Fourteen months. Sakana's AI Scientist went up on arXiv in August 2024 and reached Nature in March 2026. Nineteen months. Google announced its co-scientist in February 2025 and printed it in May 2026. Fifteen months, to the day. And Biomni got the sharpest write-up it will ever receive, from a synthetic biologist on X:

"Biomni was on arXiv 13 months ago. Biomni was on GitHub 11 months ago. Phylo, the company built on Biomni, raised $13.5M and launched 5 months ago. Or, I guess, you could read about it in Science Magazine today."

None of this is a criticism of the labs. FutureHouse posted their preprint the moment they had it, which is exactly right. The criticism, if there is one, belongs to the clock the journals run on.

So where is the field actually standing? Not where Nature says it is.

FutureHouse has already built Robin's successor. Kosmos went up on arXiv last November. It runs for twelve hours at a stretch, and a single run reads around 1,500 papers and writes roughly 42,000 lines of code. Independent scientists checked its reports and found 79.4% of its statements accurate. It has produced seven discoveries, four of them net new. Collaborators estimated that one run did about six months of their own work.

They then spun out a for-profit company, Edison Scientific, to sell it. And this month they announced a partnership that uses Kosmos not to write papers but to found biotech companies.

Sakana, meanwhile, is selling Marlin, an autonomous research agent that runs for about eight hours unattended and is pitched as a virtual chief strategy officer.

Read Nature to learn what was true a year ago. Read the preprints and the launches to learn what is true now.

The counterweight nobody is reading

While these systems were collecting their journal stamps, a second body of work was landing that tells you precisely where they break. None of the launch threads mention it.

The sharpest of it is a paper whose title belongs on a poster in every lab now buying one of these things: **Correct Answer, Wrong Mechanism**. The subtitle is better. When AI Scientists Defend General Claims Their Own Data Contradicts.

The researchers watched a coding agent try to rediscover a known result in particle physics, twenty-eight times over. Often it got the right answer. The problem was how. In 20% of episodes with the primary model, and 37.5% across other frontier models, the agent reached a right-looking result through reasoning that collapses the moment conditions change. Worse, when pressed, it defended the wrong mechanism, arguing for physics that contradicted the numbers in its own output.

Their verdict is careful, and damning. These systems are dependable as tools and, for now, "unreliable scientific co-authors for open-ended claim-making." Asking whether the agent got the answer right tells you almost nothing. You have to score the outcome, the fidelity of the mechanism, and the honesty of the account as three separate things.

Related work in the same window catalogues the failure modes with unnerving specificity: hallucinated results, methodology fabrication, citation invention, frame-lock, and my favorite piece of vocabulary this year, bug-as-insight reframing, in which a system trips over a defect in its own code and writes it up as a discovery.

Set that next to Robin and its design choices stop looking conservative and start looking wise. When Robin says ripasudil enhances phagocytosis, a cell culture has already voted, and a cell culture does not care what the agent believes about it.

A wet lab is a brutal, incorruptible critic. You cannot argue a cell culture into agreeing with you.

But be precise about how far that protection reaches, because it is not the whole paper. When Robin says the mechanism runs through ABCA1, that is Finch reading an RNA-sequencing experiment, and that is exactly the surface the CAWM authors are worried about. The bench protects the finding. It does not protect the explanation. Which is roughly the division of trust I would apply to any of these systems, and to be fair to FutureHouse, it is the division they applied themselves: the drug candidate is offered for preclinical review, the mechanism is offered as a possibility.

It is also why the wet-biology systems stop at semi-autonomous, and why that ceiling is physical rather than technical. Laboratory instruments were not built to take orders from software. Somebody has to run the assay. Sakana's system is the exception that proves the rule, and it proves it uncomfortably: with no bench anywhere in the loop, it is the one system in this survey that will write its own paper and review it.

What I Run

I should declare an interest, briefly, because I have spent the last year on the other side of this.

I run an autonomous research engine called ARIA. It is a persistent system rather than a one-shot agent: a pool of research ideas competing against each other on a single scoring function, experiments that run locally first and have to earn their way onto expensive hardware, a critic drawn from a different model family that posts a verdict on every result, and failures that classify themselves and re-enter the pool as recovery work. Seven instances, four domains, about 19,400 auditable commits, and 97.8% autonomous resumption after failure. Its successor now runs around the clock on a borrowed eight-GPU node, pointed at retinal disease. There is a case study and a short film if you want the shape of it.

I raise it for one reason. Building the thing taught me the same lesson the CAWM paper is now pressing on the whole field, and I learned it the expensive way.

Generating hypotheses was never the hard part. Building a critic honest enough to kill them was.

An idea generator is cheap, and it will happily run forever, and every dashboard will stay green while it does. What is difficult, and what almost nobody budgets for, is the machinery that tells the system it is wrong and makes the verdict stick.

Robin never had to build that machinery, because the bench already is it. That is a structural advantage of working in wet biology, and it is worth naming plainly for anyone whose research loop closes entirely in software, where nothing pushes back for free.

Why It Matters

Do not go shopping for a discovery model. There isn't one. Not one system in this survey used a model you cannot buy. Robin, the co-scientist, Kosmos, Biomni: every one of them is a harness built around a commodity brain, which means the thing your organization would be building is the harness, and that is engineering, not procurement.

The compartmentalization gap is inside your own company. Robin's whole trick is finding the connection between two literatures that no single person reads. You have that problem internally, at smaller scale, right now. The result one team generated and another team needed and never saw is the cheapest discovery you will ever fund.

Read the preprints. On this topic the peer-reviewed record is running roughly a year behind the work. Use the journals to confirm, not to track.

Take the reliability research as seriously as the launch threads. A system that finds the right answer for the wrong reason will sail through your demo and drown in your clinic. Ask any vendor how they score mechanism fidelity, not just outcomes. Watch them field the question.

And watch ripasudil. It cleared cells, not patients. The road from here runs through a disease model and then a randomized controlled trial, and most compounds that look this good at this stage do not finish it.

Robin did not replace the scientist. It replaced the lag, the decade dabrafenib spent sitting in the literature while nobody put two known facts side by side.

There is a certain irony in that finding taking fourteen months to reach print. It does not make it less of a landmark. Congratulations to the FutureHouse team. The field is better for this being in the record, even if the record took its time.

I ran a full thirty-day sweep of this category to write this piece, and most of it did not fit. The complete survey, with every system, number, and citation, is [in a companion post here](https://rundatarun.io/p/last-30-days-the-ai-scientists).in a companion post here.*

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

Some version of this question arrives most weeks. How do you keep up? You have a day job. The field moves every single day. What are you reading?

For a long time I answered it badly. I would say something about reading a lot, which is true and completely useless to the person asking. The real answer took me a while to say out loud, because it sounds like a dodge.

I stopped trying to keep up. I built something that keeps up for me.

Bookmarks are where links go to die

Nobody is reading this field. Not all of it, not close. Papers land overnight. A lab ships a model on a Tuesday and by Thursday other teams have published what breaks about it. The people who look like they are on top of it are not reading faster than you. They have narrowed, or they have automated, or they are bluffing.

Manual curation was my first answer, and it failed the way manual curation always fails. Open tabs became a graveyard. Saved links went unread and, worse, went stale.

A link I saved in March, about a model superseded in April, is not information anymore. It is clutter with a timestamp.

Newsletters helped until I was subscribed to a stack of them and reading none. The problem was never that I lacked sources. I had too many, arriving on their own schedules, with no shared memory between them.

One pipe

So I built a wire.

Several channels I already had running now point at one place. A scan that reads the day's news, releases, and research. The things I bookmark as I go. Findings from a small set of AI assistants I run, each pointed at its own beat. My own written breakdowns of the papers and tools and companies I sit down and work through by hand. Things I email myself at midnight and would otherwise never see again.

All of it lands in one store, scored and deduplicated. No single source has to be the right source, because none of them carries the day alone. And nothing gets read twice, because the store already knows it saw that story on three feeds this morning.

Two things come out the other end.

The first is private, and it is the part I use every day without thinking about it. The store is a knowledge base that refreshes itself, and my own AI tools read from it directly. When I sit down to write, or research something, or work a problem, the context they pull from is current. Not their training data. Not a stale snapshot I remembered to update. This morning's.

The second is public. A curated slice of the store becomes a website: a daily dispatch of the items that scored highest, written up rather than merely linked, so you know why something mattered before you decide to click it.

I like the arithmetic of it. The wire watches fifty-two sources. Over the last month it read and scored two thousand six hundred and seventy-nine items, and three hundred and ninety-nine of them cleared the bar. Yesterday's dispatch came out of a hundred and twenty-one signals. The site puts it more plainly than I would: it reads about seven things so you read one.

Fifty-two sources in. One page out.

What lands is a page I can get through with coffee.

The wire is the front door

This is not a side project I bolted on. It sits at the front of everything else I make.

What scores highest is usually what I end up writing about, because the things worth six hundred words announce themselves by refusing to go away. What I write becomes what I post. What I post brings back arguments and corrections from people who know more than I do about some corner of it, and those go back into the store as research. The loop closes. One pipe, several outlets, and the website is only the visible tip of it.

I did not design it that way at the start.

I built the scan because I was drowning. Then I noticed the thing that solved the drowning was also the thing deciding what I had to say.

Go look at it

The daily dispatch is free, it is public, and it lands every morning: wire.rundatarun.io/briefs. The day's signals, grouped by what they are, with a source link on every item so you can go argue with the original instead of taking my read on it.

Underneath it, the firehose shows you the machinery. The whole intake, the filter that thins it, and the last seven days in the raw.

What it does not show you is the store underneath. The complete searchable archive, the picks I would put in front of you myself with the reasoning attached, the decode and dossier library, the week ahead before it is news. That layer is not free, and it is not open yet. There is a waitlist at the bottom of the firehose page, and if the arithmetic up there sounded like your problem too, put your email in it.

Here is the whole thing, start to finish, in about forty-five seconds.

Jack Hanlon, who leads GenAI Media at Meta, posted this on LinkedIn:

"The first time you see an engineer build something in 45 minutes that would have taken a week a year ago, but then see it not ship for another 6 weeks, you will be radicalized."

He's describing Meta, a consumer tech company. The review stack he lists is the load every build carries before it ships: Strategy, Product, Design, Engineering, Privacy, Legal, Accessibility, Comms. It's also not close to what biopharma and patient-facing R&D organizations ask of an AI build.

In biopharma and adjacent patient-facing industries, add data privacy impact assessment, security review, AI governance review, IT architecture review, change-management approval, procurement, legal redline, regulatory sign-off. If patient data is anywhere in the lifecycle, add clinical safety review. Each gate is serial. Each gate is three weeks minimum. Six in sequence is half a year before anything ships. You've done paperwork.

The numbers back this up across every regulated sector, not just pharma. Enterprise SaaS procurement runs a median 170 days; complex solutions hit eleven-and-a-half months. SOC 2 Type II audits take six to twelve months end-to-end. Federal contractors face 12-18 month FedRAMP authorization cycles before a system can sit on a government network. Banks run AI governance committees on top of model-risk committees on top of vendor-risk committees, each stack inherited from a different decade of incident response. MIT NANDA's "GenAI Divide" report found 95% of enterprise AI pilots deliver zero P&L impact, across thirty to forty billion dollars of spend. That's not a build problem. The build works. The build sits in a queue.

This accelerated review pathway can exist. It can work. It will take herculean levels of structural redesign and a lot of organizational political capital to land. And the pattern isn't a pharma problem. It's the shape of every regulated industry trying to ship AI in 2026. Pharma's version is the sharpest because patients are downstream of the delay, but the argument is universal.

Every gate exists for a reason. Most got written after a real incident. Someone's data got exposed. A launch communication went sideways. A model made a biased call in production. The answer isn't to kill them. The answer is that every one of them was designed for a world where building the software was the slow, expensive, irreversible part. Whether you wrote the code yourself or bought it from a vendor who wrote it, software was the thing the calendar bent around. That world is gone. AI didn't just speed up typing. It turned the whole build, your own code and the vendor's both, into the fast part. The review cycle was calibrated for the old critical path. The critical path moved. The calendar didn't.

Hanlon's three confrontations are right. Process architecture is the speed limit. Too many orgs treat every process as sacred. AI should be eating process, not just coding. The patient-facing version is sharpest, because the delay shows up downstream as patient time.

This Has Happened Before

DevOps didn't land in 2015 as a philosophy. It landed when development got fast enough that deployment became the jam. "Development teams could complete features in days, but getting those features deployed took weeks." That's the DevOps.com retrospective on the pre-CI/CD era. Swap "deployed" for "procurement-approved" and you have a 2026 AI program, sentence for sentence.

Then security became the jam. DevSecOps. Then infrastructure and developer experience became the jam. Platform Engineering. Internal Developer Platforms. Each compression moved the bottleneck one layer further from the keyboard. Each shift spawned a category, a toolchain, a set of jobs, a conference circuit.

Bottleneck migration is a law. Compress one layer and the adjacent layer becomes the bottleneck by definition. Whatever used to be the tall pole is now almost all of the remaining pole.

AI compressed the build layer, writing software and buying it both. Everything adjacent to the build is now the bottleneck. The reviewers, the committees, the approvals, the procurement track, the vendor security questionnaire, the model evals nobody scoped time for. All of it. DevOps took twelve years to run the full cycle. Nobody has twelve years.

Shadow AI Is the Tell

Gartner's 2025 numbers are the signal. 98% of organizations report unsanctioned AI use. 69% report evidence of prohibited public GenAI use inside their walls. 49% expect a shadow AI incident within twelve months.

The default read calls shadow AI a governance failure. A discipline problem. Employees not following the rules. That read is backwards. Shadow AI is the system telling you the official process layer already failed an internal cost-benefit test. Employees measured the queue, measured their quarterly objectives, and decided the queue costs more than the risk of getting caught.

That's the market voting. When 98% of your workforce is routing around your approval process, you don't have a discipline problem. You have an economics problem. The process priced itself out.

A fair counter to all of this: maybe the 95% pilot failure isn't a process problem. Maybe it's a pilot-selection problem. Most enterprise AI pilots fail because they were the wrong pilot to run, picked for board optics or executive curiosity rather than real workflow pain. Speeding up bad pilots faster doesn't help anyone.

That argument is partially right. The data is harder than that. The same MIT report that flagged the 95% number also found the highest ROI was in back-office automation, exactly the work that sits behind procurement, vendor security, and compliance queues. The pilots that fail include some bad picks. They also include good picks that died in queue. You can have a pilot-selection problem and a process problem at the same time, and most enterprises do.

Process Protects Patients. Except When It Doesn't.

Biopharma's review gauntlet was written in blood. Good Clinical Practice, Good Laboratory Practice, Good Manufacturing Practice. Patients got hurt. Rules got written. Every review, every sign-off, every three-week queue descends from that inheritance.

The reflex response to "AI is slow to ship" is "good, that's how we keep patients safe." It sounds unanswerable. It isn't.

Process was built to protect patients. When process is the thing blocking patient-helping AI from shipping, it stops protecting patients. It protects itself.

Not shipping AI is not a neutral, safe choice. It has a cost. The cost lands on patients too.

Every week of review on a regulatory drafting agent is a week a safety narrative gets written the slow way. Novo Nordisk's NovoScribe rollout cut clinical study report drafting from fifteen weeks with over fifty medical writers to ten minutes with three. The work moved. The submission queue didn't. Every week of governance queue on a tool like that is real human time that wasn't spent on the submission, real submission time that wasn't spent reaching patients.

Every month a vendor security review stalls on a literature retrieval agent is a month oncology scientists read abstracts by hand. The responder-subgroup signal sits in a PDF nobody opened.

Every quarter a leadership group debates a two-speed governance design is a quarter trial protocols get mapped by hand, deviations get triaged out of a Teams channel, the first patient at the first site doses later than they would have.

Meanwhile, the deployments ship. Cleveland Clinic, NHS England, the VA, and 75% of US hospitals are running AI in clinical workflows today. The teams that wired governance early are in production. The teams still debating are watching.

Most governance agendas debate the wrong question. Not "is this AI safe enough to ship," but "is the delay we're about to impose smaller than the harm we're trying to prevent." Both sides of that ledger have a body count. Only one gets counted.

AI Eats Its Own Governance Layer

The governance layer isn't a coding problem, and that's the part most "AI accelerator" pitches miss. It's legal redline, procurement of the services the agents will consume, vendor security, the privacy review, the model evals, the benchmark sign-off. None of that is typing. All of it is the build now, and all of it queues.

Organizations have answered by standing up governance as elaborate as the thing it governs. Sanofi has publicly documented a multi-stage setup: a Responsible AI Working Committee, an Interim Responsible AI Governance body, an AI agent named Plai sitting in on every drug-progression decision. Other large pharmas, banks, and federal contractors run variations of the same shape; the public documentation lags Sanofi's. Ethan Mollick names why it decides everything. "The moderating factor is no longer individual ability or even AI capability. It is organizational structure, policy, and the way leaders choose to approach AI." The constraint isn't the model. It's the layer wrapped around the model.

Which is exactly the layer AI can eat. Hanlon's third point is the one most readers stop quoting before the punchline:

"Have the AI traverse your codebase and collect your evidence for the privacy review and have another AI grade the privacy review. They go back and forth until (a) they need human intervention because they are stuck or (b) they need humans to look at the final results and sign off on the right path forward."

That's the whole answer, and it generalizes past the codebase. Every gate in the gauntlet has the same shape: forty hours of evidence assembly, then a fifteen-minute judgment call. The privacy review is fourteen pages of data flow that already lives in the repo, the retention policies, the IAM config. The vendor questionnaire is two hundred SOC 2 controls a lead engineer maps by hand against an attestation report nobody re-read. The procurement intake is free-text someone retypes against master agreement terms a sourcing lead has memorized. The model eval is a benchmark suite someone runs once and pastes into a slide. The regulatory submission is fifteen weeks of medical writers stitching prior submissions into a new narrative. The judgment is fifteen minutes. The forty hours is the queue.

The human doesn't leave the privacy review. The human finally opens a pre-assembled package instead of staring at a blank template with half the source material missing. The reviewer still reviews. The reviewer is finally reviewing the only part of the work that ever needed a reviewer.

AI has eaten the keyboard. It hasn't eaten the queue.

That's the argument in one sentence. The keyboard compressed two years ago. The queue hasn't. Until it does, every "AI accelerator" announcement is a faster typewriter feeding the same backlog.

The Reviewer Is the Project

There's a catch built into all of this. The people whose work the AI must eat are the same people who must approve the AI eating their work. Legal reviews the legal agent. Procurement approves the procurement agent. InfoSec signs off on the InfoSec agent.

The reviewer is the subject of the review. The incentives don't line up. Pretending they do is how an org burns eighteen months and ships nothing.

The first week of agent output will be embarrassingly wrong, and that isn't the agent's fault. The process has never been written down in a way a machine can act on. The DPIA "template" is really a set of in-head judgments held by a senior person who onboarded in 2014 and knows which questions matter. Turning that into prompts, policies, and machine-readable rubrics is the institutional work, and there's no shortcut around it. That IS the project. The catch is that the people best positioned to codify their judgment are the ones whose standing rests on it staying in their heads. The org design that works makes the codified version the asset and leaves the committee to ratify it. Most enterprises don't run that design yet.

The ones that do compress the queue start in the same place, and it's never a model. They map the gauntlet, then fund an agent for a single gate, picked where the work is repetitive and the inputs already live somewhere a machine can read. Privacy review first, its answers mostly sitting in the repo and the IAM config. Vendor security next, the same shape with messier inputs. Regulatory drafting last, after the pattern has proven itself on something lower-stakes than an FDA filing. It holds at one gate, run for two months and watched for what it gets wrong. It breaks at three at once.

Underneath all of it sits the substrate. A governance layer that can't hand an agent a rubric to run or a template to fill isn't policy. It's tradition. The version living in senior reviewers' heads is neither auditable nor transferable, and it's the version adding weeks to every queue. Codifying it is the platform bet of 2026, ahead of the next model, because it's what lets every future build ship without the six-month tax.

AI compressed the build layer. That was the easy part. Everyone got the faster typewriter. The next decade of enterprise AI won't be won on better models. It turns on whether the layer built to protect patients from bad software, back when bad software was the threat, can be made to compress itself before the delay starts landing on the other side of the ledger.

That layer will not shrink on its own. It has to be handed its own rubric and told to grade the work it used to guard by hand. No team that owns a queue volunteers to automate it. Someone with the authority to override that instinct has to decide the delay costs more than the control.

In a patient-facing industry, the queue is not paperwork. The queue is the patient. Every week it stays uncompressed is a week charged to someone who never got a seat in the room.

There is a shelf now. If you are a senior leader trying to make sense of AI, you have probably bought most of it. The strategy book. The mindset book. The one with the four-quadrant framework. The one from the consultancy with eleven co-authors. They are not bad books. Several are good. I learned things from them.

But I noticed something after the fourth or fifth one. I would finish a chapter, nod, underline a sentence, and then sit there with my hands in exactly the same place they were before I started reading. The book had changed what I thought about AI. It had changed nothing about what I did with it on Monday.

That is the gap I wrote into.

The book changed what I thought about AI. It changed nothing about what I did with it on Monday.

The shelf has one shape

Read enough of these books and the shape repeats. They tell you what to think about AI. How to frame it for your boss/peers/board. Which mental model to adopt. Where it sits in your strategy. They are written from the altitude a leader is used to operating at, which is to say: above the work, directing it.

That altitude is the problem, not the solution. Someone reviewing the current best-seller in this category said it reads like a 2024 book explaining how the Internet will help your business. The line is unkind and basically right. Not because the author is wrong, but because the genre has a ceiling. You cannot write the operator's manual from the strategy altitude. The two are different books, and almost everyone is writing the first one.

I wanted to read the second one. Nobody had written it for the reader I had in mind, so I wrote it.

You cannot write the operator's manual from the strategy altitude.

What "what your hands do" actually means

"Leaders should build" is already a slogan, and slogans rot.

I don't mean you should learn to code. I mean something narrower. There is a class of tools now, Claude Code chief among them, that lets you direct a multi-agent system to produce real work inside your own domain. You describe intent, set boundaries, review what comes back, redirect, and ship. The interface happens to be a terminal; the skill is not programming. It is the same skill that got you to senior: defining the why, designing the handoffs, judging the output, catching the thing that looks right and is wrong.

You already do all of that. You do it with people. The book is about pointing those same instincts at a harness instead of a team, and what changes when you do.

The reason this matters is not productivity, though the productivity is real. The reason is calibration. Andrej Karpathy named it this spring in a line that landed with twenty thousand likes and a thousand replies: there are two groups talking past each other about AI, and the line between them is not skeptic versus believer. It is the people who have built something with these systems and the people who have read about them. Every confident claim you have heard about what AI can or cannot do was made from one side of that line. If you are reasoning about AI from the reading side, you are reasoning from the wrong data, and no amount of strategy reading moves you across. You move across by getting your hands on the controls one time and feeling where the real edge is.

Why I get to say this

What separates this book from the shelf is that I did not write it from the strategy altitude. I built and led an AI Center of Excellence at a Fortune 500 pharmaceutical company, and I now lead applied data and AI in R&D at an AI-driven biotech. I also run a deep personal practice on nights and weekends, eight autonomous agents and a homelab full of skills, because the day job and the obsession turned out to be the same skill. This book was itself produced through the harness it describes, end to end. About a month from kickoff to typeset manuscript, against the year the same book would have taken by hand, with thousands of hours of research, red-teaming, and editing behind it, most of it run in parallel by the system rather than typed by me. You can see the skills that drafted it.

I am not telling you to do something I read about. I am telling you what it was actually like, including the parts that are awkward and the parts that broke.

I am not telling you to do something I read about. I am telling you what it was actually like.

The part I won't oversell

At Sequoia's AI Ascent this spring, Karpathy declared "vibe coding" obsolete and named its successor "agentic engineering," a discipline with real rigor and real responsibility. He is right. He also drew a careful line: prototyping with AI raises the floor for everyone, but operating serious systems demands discipline you cannot wave away.

My book asks you to step over a line slightly further out than the one Karpathy draws for the general public. I am asking a senior, non-technical executive to personally operate, not just prototype. I think that is where the role is heading, and I think the leaders who learn it first will shape what their organizations become. But I am not going to pretend it is the consensus position. It is an argument. The book makes it, and gives you the controls and the guardrails to test it for yourself rather than take my word for it.

Who will not find this useful

If you want a framework to put on a slide for your board, the shelf has better options than mine, and I mean that. If you want reassurance that your current AI strategy is sound, this is the wrong book; it will probably make you uncomfortable. If you are looking for predictions about AGI timelines, I route around that debate on purpose.

This book is useful if you have read the think pieces, sponsored the center of excellence, watched the pilots, and still feel the quiet gap between knowing AI matters and knowing what you personally do about it. It is useful if you are willing to spend a few hours with your hands on something unfamiliar, on your own files, to find out where the edge actually is. It is a ladder. Three months to ship something real, six on the outside. The last page is a welcome from the people who were already on the other side when you started.

If that is you, the book is for you. It's out now: builder-leader.com, or buy it on Amazon.

Builder-Leader: The AI Exoskeleton That Crosses the Gap.