This week Anthropic published an analysis of about 400,000 real Claude Code sessions, run between last October and this April. It set out to answer a plain question: when someone sits down with a coding agent, what actually predicts whether they succeed? There is a comfortable answer and a more useful one, and almost everyone is repeating the comfortable one.

The comfortable answer, the one the headlines took, is that anyone can build software now. That part is true, and it is the least interesting thing in the report.

The finding underneath is the one I'd hand to anyone, whether they run a team or just their own work. What predicted success was not your job title, and it was not whether you could write code. It was whether you understood the problem you were trying to solve. Anthropic's own framing: success comes from how well a person understands the work, not whether they're trained in coding.

That is not a story about AI flattening the gap between your best people and your average ones. It is the opposite. The thing you spent fifteen years getting good at just became the thing that decides whether the most expensive tool you're buying actually pays off. I have been making that case for a while. This is the first time anyone has put 400,000 sessions behind it.

What the tool actually divided up

Start with the cleanest number, because it doubles as a mental model you can carry into a Monday meeting.

In a typical session, the human made about 70 percent of the planning decisions and the agent made about 80 percent of the execution decisions. People decided what to build. The agent decided how to build it. That split held across every kind of work they measured, from writing code to running systems to analyzing data.

So the labor didn't disappear. It separated. The agent took the part a lot of people thought was the moat, the ability to actually produce the syntax, and handed back the part that was always the hard part: knowing what to ask for, and whether the answer is any good.

Here is what makes me trust the number. A practitioner reached the same shape from the other side. Addy Osmani, writing in January, found that the developers succeeding with these tools "spend 70 percent of their time on problem definition and verification strategy, 30 percent on execution." Two independent measurements, one watching a population and one watching working engineers, landing on the same line through the work. The tool made the typing cheap. It did nothing for the knowing.

The tool made the typing cheap. It did nothing for the knowing.

This is the thing I've been calling a change in the atomic unit of work. The unit a person owns moved up the stack, from producing the thing to directing it. Now there's telemetry under the claim.

Your job title barely moved the needle

This is the result Anthropic led with, and it earns the lead.

They scored a verified success rate, which is stricter than it sounds: a session counts as a win only if the model judged it successful and there was a hard signal to back that up, a real commit, a passing test, a user saying yes. By that bar, on code-producing work, software occupations succeeded 34 percent of the time and everyone else succeeded 29 percent. A marketer and a staff engineer finished a coding task at almost the same rate.

The instinct is to read that as "engineers are finished." It is the wrong read, and the same study hands you the right one. What collapsed was the premium on being a programmer. What held was the premium on understanding the problem. Measure expertise correctly and it mattered enormously: novice sessions succeeded 15 percent of the time, and people who actually knew their domain landed between 28 and 33 percent. Roughly double.

The reconciliation is the part that clarifies it, because it is the bet I've been making. Expertise here is not your title or your résumé. It is task-specific. Anthropic's own example: a senior engineer asking their first question about the Rust language is a beginner at Rust. The skill that predicts whether you get working software out of an agent was never "I am an engineer." It is "I understand this particular problem well enough to direct the work and catch it when it goes wrong."

That is the whole argument for the builder-leader, measured now across a population instead of asserted from a single desk. You do not have to become an engineer. You do not have to write the code. You direct it, inside a domain you already command, and the command is the thing that pays.

You don't have to become an engineer. You don't have to write the code. You direct it, inside a domain you already command.

The expert tell is recovery

If I had to keep one number from the whole report, it would be this one.

Experts didn't just prompt better on a good day, though they did do more with each instruction: about 12 agent actions per prompt versus 5 for novices, and roughly five times the output. The real difference showed up when things went sideways. When a novice hit trouble, they walked away with nothing written about 19 percent of the time. For everyone with more domain knowledge, that abandonment rate was 5 to 7 percent.

Sit with what that means. The expert hit the same wall the novice did. Then they routed around it, reframed the problem, and caught the fluent, confident, completely wrong answer before it shipped. The novice hit the wall and quit.

So the value your senior people add in an agentic workflow is not that they prompt cleaner on a Tuesday. It is that they fail better on a Thursday. They have the judgment to know when the plausible output is wrong, and the agent does not have that judgment and cannot get it from a model update. The bottleneck moved from typing to deciding. Pratima Arora at Smartsheet put it plainly this spring: the hours haven't changed, but the density of work has.

The scarcest thing in the building is now the thing your best people already carry. That is good news, and most of the coverage will skip right past it.

The scarcest thing in the building is now the thing your best people already carry.

The work moved up the stack

Two more numbers close the loop, and they are the ones that tell you where this is going.

Between October and April, the share of sessions spent fixing broken code fell from a third to under a fifth, 33 percent down to 19. Over the same stretch, the estimated value of the work people brought rose about 27 percent, with the biggest jump in building something new, up 43 percent.

Put those together and the trajectory is clear. People are not using the agent to fix more bugs. They are using it to attempt harder, more valuable, more end-to-end work, and bringing more judgment to bear when they do. The tool got cheaper per task, so the tasks got more ambitious.

This is the oldest pattern in economics wearing new clothes. Make a thing cheaper to produce and you do not produce less of it. You produce far more, and you need more judgment to steer all of it. That is the shape of this moment, and it cuts against the fear that the work is shrinking. The work isn't shrinking. It changed: what we attempt got bigger, and how we get there moved from doing to directing.

Two things held their price, not one

There is a second thread here, and it is the one that turns a study into a strategy.

Anthropic measured what the human brings to the session, and found that what the human brings, domain command, is decisive. That is one of two things this whole shift never made cheaper. The other is what you build around the model.

The model is the commodity, and the system you build around it is the moat: the rules and skills and memory that turn a smart, forgetful chat box into something that gets better at your specific work over time. Birgitta Böckeler, writing on Martin Fowler's site, named the gap the model itself can never close. A coding agent, she wrote, has "no social accountability, no aesthetic disgust at a 300-line function, no intuition that 'we don't do it that way here,' and no organisational memory." Those are not features you buy a newer version of. They come from the person and the system around the model, or they don't come at all.

So the picture is narrower and more useful than "expertise wins." Two things resisted the repricing. What you bring to the model, which is domain command, and what you build around it, which is the system that holds your organization's judgment. The model in the middle, the part everyone is still shopping for on price, is the cheap layer between them. Anthropic just measured one of those two human pieces at a scale none of us could reach alone. The other one you build.

And both are leadership skills, not engineering ones. Setting intent, designing the handoffs, evaluating output against what good actually looks like. Those are the same instincts that got your best people to senior in the first place, pointed now at a system of agents instead of a team of people. No new species of human required. That is the part I keep coming back to, and it is why I think the people who internalize this will out-build the ones still arguing about whether the juniors are coming for the seniors.

One honest note

At the level of the whole economy, you cannot see this yet in the aggregate numbers. A large Danish study of AI chatbots across occupations found no significant effect on earnings or recorded hours. I take it seriously, and I read it as a statement about timing, not a refutation. Session behavior changes before payroll does. The 400,000 sessions are the leading edge; the wage data lags. Hold the whole argument a little more loosely for it, and don't over-rotate on any single quarter's story.

What I'd do with this Monday

If you run a team, here is the part that converts. If you don't, read it as where to put your own time.

Put the agent in the hands of your domain experts directly, not only your engineers. The advantage is highest exactly where deep knowledge meets the tool. The clinical-operations lead who has watched the trial workflow break in a dozen specific ways will get more out of an agent than a generalist engineer who hasn't, because the agent can produce the code but it cannot supply the judgment about what the code is for.

Hire and promote for understanding the problem and recovering well, not for who writes the cleanest syntax. The data says occupation barely predicts success and recovery strongly does. That is now a measurable thing to select for.

And grow your next builder on purpose. The way across this gap was always building, six months of directing real work inside a real domain, not sitting through a demo. The leaders who win the next few years won't be the ones with the best coders. They'll be the ones who turned their domain experts into builders before anyone told them it was allowed.

The agent decides how. You still decide what, and you still decide who learns to decide what next. Four hundred thousand sessions just told you those are the two jobs worth keeping. They are not a smaller job than the one before. They're the better one.

Related: The Specialist Is Now You is what one expert plus an agent can now do alone, and The Atomic Unit of Work Just Changed is where the unit a person owns moved up the stack.

The fuller argument for treating judgment, intent, and domain command as the durable skills is the Builder Leader field guide (builder-leader.com).

Every Sunday I pick one paper or release that's worth your time, break it apart, and tell you why it matters. No hype. No summaries of summaries. Just the idea, explained.

A researcher walks over with a sequence on her screen. She has a stretch of DNA, a promoter, the switch that turns a gene on, and one variant sitting inside it that she suspects changes how loudly that gene gets expressed. She wants to know if the variant matters. So she asks the question everyone asks now: which AI model should I use? BOLT-LMM or DNABERT?

It sounds like a ranking question. Pick the better of two tools. It is not a ranking question. It is a category error, and it's one I've watched smart teams make without noticing.

BOLT-LMM and DNABERT do not belong in the same column. One is a statistical method built to scan a whole population and find which genetic differences track with a trait across thousands of people. The other reads a single stretch of DNA and turns it into a numeric representation a computer can work with. Asking which is better for a promoter variant is like asking whether a microscope or a telephone is better for measuring temperature. The answer is neither, and the question itself is the problem.

The hardest mistake in genomic AI isn't picking the second-best model. It's picking the wrong kind of model and not knowing it.

The phrase that hides nine different things

"Genomic AI model" sounds like one category. It is at least nine.

There are DNA language models, which read raw sequence and learn its grammar the way a text model learns the grammar of English. There are sequence-to-function predictors, which take a piece of DNA and predict what it does in a cell, how strongly it drives expression, whether a region is open or closed. There are variant-effect predictors, which score how much a specific mutation changes that function. There are statistical-genetics engines, which never look at sequence grammar at all and instead test, across a population, which genotypes associate with which traits. There are single-cell models that work on tables of gene activity per cell, generative models that design new sequences, and a few more.

Every one of these gets called a "genomic AI model." Every one of these shows up in benchmark papers ranked against the others. And every one of these answers a different question.

The researcher with the promoter variant needs a variant-effect predictor or a sequence-to-function model, something that reads her one sequence and tells her what the change does. BOLT-LMM, the statistical-genetics engine, is built for a completely different shape of problem: thousands of people, their genotypes, their traits, and the search for associations across the cohort. It has no way to read the grammar of her single promoter. DNABERT, the DNA language model, reads grammar beautifully but doesn't produce a calibrated answer to "does this variant matter" without more machinery bolted on top.

The two tools the researcher was choosing between were both wrong. The benchmark she'd consult to choose between them can't tell her that.

Why the leaderboard can't catch this

Here is the uncomfortable part. The benchmarks the whole field relies on are structured to make this error invisible.

A leaderboard ranks models within a class. It lines up five DNA language models and tells you which embeds sequence best. It lines up four variant-effect predictors and tells you which scores variants most accurately. That is useful, and the people building those benchmarks are doing careful work. But a leaderboard assumes you already chose the right column. It answers "which is best?" and stays silent on the question that comes before it: best at what, and is that even what I need?

The wrong-class error happens upstream of every benchmark. By the time you're reading a leaderboard, the category decision is already made, usually without anyone noticing a decision was made at all. You typed a model name into a search box, found a paper that ranked it highly, and never asked whether the thing being ranked was the right kind of thing for your task.

Nobody walks into this on purpose. The pull comes from how the field names things. Two models share the word "genomic," both have an impressive accuracy number in a recent paper, both have a HuggingFace page and a star count, and the surface presentation makes them look like rival products on the same shelf. A leaderboard reinforces the illusion by placing them in the same table. Nothing in that experience signals "these answer different questions." The signal you'd need is a layer that sits before the ranking and sorts tools by what they're for, and that layer mostly doesn't exist.

I'd put this error ahead of picking the second-best model in the right class, and it's far more costly. Pick the second-best variant-effect predictor and you lose a few points of accuracy on a task that was at least the right task. Pick a statistical-genetics engine for a single-sequence question and you don't lose accuracy, you get an answer that means nothing, dressed up to look like it means something. The numbers come back formatted, plotted, ready to drop into a slide, and there's nothing on the surface that tells you the foundation was wrong. Weeks of analysis can ride on top of a category mistake made in the first thirty seconds, and the failure surfaces late, if it surfaces at all.

Classify first, rank second

The fix is almost embarrassingly simple to state and surprisingly hard to do without help: classify the computational object before you rank it.

Before you ask which model is best, ask what kind of object your task actually needs. Do you have one sequence and want a functional readout? You need a sequence-to-function or variant-effect model. Do you have a cohort of people and want to find trait associations? You need a statistical-genetics engine. Do you have a table of cells and want to label cell types? A single-cell model. The class is determined by the shape of your data and the shape of your question, not by which model has the most citations this month.

Get the class right and the leaderboard becomes useful again, because now you're ranking within the right column. Get the class wrong and the leaderboard is worse than useless, because it gives you a confident ranking of tools that can't do your job.

The reason this is hard in practice is that the class isn't printed on the box. A model's documentation tells you its architecture and its scores; it rarely says, in plain terms, "this is for cohort association, not single-sequence scoring." You have to infer the class from the shape of the inputs it takes and the outputs it produces, and that inference is exactly the expertise a researcher new to a method doesn't have yet. The whole point of a triage layer is to do that inference for you and to fail loudly when the tool you named is the wrong kind of thing.

What this looks like as a tool

I built a small open thing to make this concrete, because it's easier to argue for "classify first" when you can watch it fire.

It's called ModelMap, and it's a decision layer rather than a leaderboard. You describe your task in plain terms, what you have and what you want, and it triages the class first. When the tool you named doesn't match the class your task needs, it returns a "wrong-tool" verdict: here is the class your task needs, here is the class this tool actually belongs to, and here is why they don't match. Ask it whether DNABERT-2 is right for a population GWAS and it routes you to the statistical-genetics column and flags DNABERT-2 as the wrong class, with the real function of each spelled out.

Underneath are nine method classes and around thirty model cards, with deterministic rules handling the clear-cut category calls and a grounded language-model layer translating free-text questions into the controlled vocabulary. It's a research-use proof of concept, source-available, and you bring your own model key so there's nothing to abuse. I built it because I couldn't find anything that does. The closest existing tool, OmniGenBench, is a leaderboard, excellent at ranking within a class and silent on which class you need.

A leaderboard answers "which is best." The question that wrecks more projects is "best at what, and is that what I need."

ModelMap's whole point is that one verdict card. It's the moment it tells you the two things you were comparing were never comparable, and points you back one step to the decision you skipped.

If you build tools like this, the harder question is how you let a language model sit in front of that verdict without letting it invent a class or a license. I wrote that part up for the people who'd build it, over on AIXplore: Build an LLM Triage Layer That Can't Freelance.

The question to carry

You don't need ModelMap, and you don't need to remember the nine classes. You need one habit.

The next time someone on your team asks which genomic AI model to use, or which large model, or which forecasting method, or which anything, resist the pull to answer the ranking question they asked. Ask the one underneath it first. What kind of object does this task actually need? Are we even in the right column?

Most of the time the ranking question is the easy part, and the field has built good tools for it. The category question is the one that decides whether the answer means anything, and almost nothing in the standard toolkit asks it for you. That's the gap worth closing, in genomics and well beyond it.

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

I have a personal finance app I built for myself. It took two nights. It reads my real bank accounts and it has done its job without complaint ever since. It works. It would help other people who want the same thing it gives me.

It was never going to leave my laptop.

Some context on the curve before the story. Months ago I built EmberPlan, the FIRE-planning app this one feeds, and I run it for the financial-independence community today. I built that one on Sonnet 4 and felt every mile of it: call it ten times the effort, every feature a negotiation, every refactor a fight I had to referee. Kindling, its daily-money sibling, took two nights on the current Opus line (4.6 through 4.8). And yesterday Anthropic shipped Fable 5, their first public model in a new tier, which today compressed the one step left over: the distance between a tool that works for me and a tool anyone can run. I pointed it at the app and watched that distance close in a single working session.

Three data points, a few months apart, same builder.

This is what that looked like, and why the afternoon matters more than the model.

The gap between a private tool that works and a public one anyone can run used to be a weekend nobody takes. That is why good private tools die private.

The work nobody schedules

Picture the thing you built for yourself that you've thought about sharing. A script, a dashboard, a little app. It does exactly what you need. Every time you consider putting it out, you run the same mental tally and decide not to.

Here is the tally for my finance app.

The code had my real brokerage positions sitting inside a test script, the kind of file you write once to load some sample data and never open again. My net worth ran through the project's saved history, commit after commit, like a watermark. The whole thing was named after somebody else's trademark and wired into my home network. The "brand" was a working title I'd typed in once. There was no front page explaining what it is or how to run it.

Every one of those is small. None of them is hard. Together they are a full weekend of careful, joyless work where a single missed line publishes your account numbers to the internet forever.

**The tax that kills personal tools isn't difficulty. It's that the work is tedious and dangerous at the same time.**

Tedious means you put it off. Dangerous means putting it off is the correct decision. So you put it off correctly, indefinitely, and the tool stays yours alone. This is not a character flaw. It's the rational response to a job with no upside until the last line is right.

One session

Today I handed the whole project to Fable 5 and described where I wanted to end up: a clean public version anyone could run, with nothing of mine left in it.

It swept the entire project history looking for anything sensitive. It found the brokerage positions in that forgotten test script. That's the part I keep coming back to.

I would not have found those by eye. I'd built the file, I'd forgotten the file, and on a Saturday-afternoon scrub I would have shipped it. The model read every line, recognized real holdings hiding in what looked like throwaway sample data, and stopped.

Then it made a harder call than I'd have made myself. The history was too contaminated to clean line by line, so instead of scrubbing it the model gave the public version a fresh, verified starting point with none of the old baggage carried forward. That's the right answer, and it's the one I'd have talked myself out of because it sounds drastic.

From there it kept going. It generalized the code so nothing pointed back at me. It renamed the app Kindling (a sibling to EmberPlan, since kindling is what you tend so the ember catches). It did a full redesign around that idea, a warm fire-lit theme in light and dark. It generated the logo and the visual identity. It seeded the app with believable fake data so the screenshots show real-looking money that is entirely invented. It wrote a proper front page. And it open-sourced the result under a permissive license, deployment scripts renamed and all.

The model did the tedious part and caught the dangerous part, the leak I'd have shipped. That's what flips the math.

One session. The thing I'd been putting off since the day the app first worked, because it was a weekend I'd never take.

The demo below came out of the same conversation. The screenshots already existed (it had seeded a Plaid sandbox with invented money, so the app shows realistic accounts that are entirely fake). I asked for a video; Fable 5 wrote a Remotion composition in the app's brand and rendered it. About five minutes, ask to mp4.

What an afternoon changes

The difference here isn't the speed by itself.

When the path from private to public was a careful weekend, the default for anything you built for yourself was it stays on your machine. That default was correct. The payoff of a publishing weekend, weighed against the small but real chance of leaking something that ends a career, came out negative for almost everything. So almost everything stayed private. Good tools and half-finished ones, all of them on individual laptops, helping exactly one person each.

Move that cost to an afternoon and supervise the dangerous step instead of doing it by hand, and the sign flips. Now the question for a useful private thing isn't is it worth a weekend. It's is it worth an afternoon I was going to spend anyway. The answer is yes far more often.

When shipping a private tool costs an afternoon instead of a weekend, the default moves from "dies on my machine" to "someone else can run this."

Small shift in cost, large shift in behavior. A lot of tools that solve one person's problem also solve a thousand other people's, and the only thing that kept them from those people was the tax in the middle. The tax is what got cheap today.

I've written before about delegation over automation: the move isn't handing a task to a machine and walking away, it's handing it the careful work and staying close enough to check the one judgment that matters. This was that. I didn't watch it rename files. I did look hard at the leak it caught and the call it made to start the history clean. The mechanics went to the model; the judgment stayed with me. That's the shape worth getting used to.

You can have what I have

The same shift works on the receiving end. The repo has a quickstart, but the truer install path now is a paragraph of English. Paste this into Claude Code (or Cowork) and let it drive:

Install Kindling, the self-hosted personal finance app, from
https://github.com/BioInfo/kindling. Clone it, check my Node version
(node:sqlite needs Node 22.5+), install dependencies, and read README.md and
PLAN.md before changing anything. Then walk me through Plaid credentials:
help me create a free account at dashboard.plaid.com, find my client_id and
Sandbox secret, and store them the way I prefer (pass entries
api-keys/plaid-client-id and api-keys/plaid-secret-sandbox, or exported env
vars). Generate an APP_ENC_KEY for me with openssl rand -hex 32. Start the
app with ./run.sh dev, confirm http://localhost:3408 responds, and walk me
through connecting Plaid's sandbox bank (user_good / pass_good, any OTP).
LLM features are optional: if I have an OpenAI-compatible endpoint (Ollama
counts), wire it up via local.env using local.env.example; otherwise skip,
everything degrades cleanly. When I'm ready for real accounts, walk me
through Plaid Production access and PLAID_ENV=production, and remind me to
keep the app private-network-only. Ask before anything irreversible.

It clones the repo, checks your machine, walks you through the one account signup a human has to do, generates your encryption key, starts the server, and connects a fake bank so you can see it working before any real money touches it.

If you're on a paid Claude plan, Fable 5 is free until June 22. The experiment costs you a paragraph.

Sharing used to mean writing install docs for every OS and fielding issues when step 4 failed on somebody's laptop. Now the README's job is to brief an agent. I ship one prompt; your agent handles your machine and its quirks. The distance closed on both ends today: cheap for me to publish, cheap for you to run.

The spark

The app didn't get better today. It worked last month and it works now. What changed is that the spark can leave my workbench.

For years the reason most personal tools never go public wasn't pride or secrecy. It was a weekend of dull, high-stakes scrubbing that no sane person does on a side project. Take that weekend away, replace it with an afternoon spent watching one model do the dull part and catch the dangerous part, and a different question shows up on the table.

Not can I build it. You could already build it. The question is whether the thing you built for yourself is about to start helping people you'll never meet. For the first time, the work standing in the way is small enough to actually do.

Mine is up now. It started as something I tend every day so my own ember catches. Turns out a spark spreads more easily than I thought.

Sources

• Claude Fable 5: Anthropic's first public model in the new tier, shipped 2026-06-09.

• Kindling: the open-sourced app, MIT-licensed.

• EmberPlan: the FIRE-planning sibling, built months earlier on Sonnet 4.

• Delegation, Not Automation: the judgment-stays-with-you frame this session ran on.

Apple is the company that owns everything. The chip in your phone, the operating system on top of it, the store that sells you apps, the cable, the charger, the retail floor you buy it on. The whole pitch for thirty years has been that owning the stack end to end is why the thing feels good in your hand. So the decision to watch at this year's WWDC is the one part Apple chose not to own.

It rented the brain.

The new Apple Intelligence runs on a family of models Apple calls its Foundation Models. Apple built them, Apple ships them, Apple runs them on its own silicon and its own servers. But the way it built them is the tell. Apple borrowed Google's Gemini, used it to train its own smaller models, and then sent Gemini home. The intelligence at the center of the most privacy-obsessed product on earth started life as someone else's model.

If you have heard me say "the model is the commodity, the thing around it is the moat," this is that argument arriving at the scale of a billion phones.

The company that built its identity on owning the stack just outsourced the brain. That is not a slip. That is the strategy.

What Apple actually shipped

Start with what's settled, because the marketing and the reporting are already fighting about the rest.

Apple shipped its own models. There's a small one that lives on your phone, with nothing leaving the device, and a larger one that runs in Apple's cloud for the heavier requests. Apple's executives, by AppleInsider's account, were blunt about the line with Google: "We use none of the models that Google deploys to their customers." A user, they say, never touches a drop of Google's code or Gemini or Google Search.

Both things are true at once, Apple used Gemini and there's no Gemini in the product, because of a technique worth understanding. It's becoming how a lot of AI gets built.

It's called distillation. Picture a senior expert and a junior. The senior knows an enormous amount but is expensive and slow to keep on staff. So you sit the junior down next to the senior for a long stretch, have the senior answer thousands of questions, and have the junior learn to give the same answers. When it's done, you ship the junior. The junior is cheaper, faster, small enough to run on a phone, and it carries a lot of what the senior taught it. The senior goes home and never meets the customer.

Distillation is hiring a junior, training them on a senior's answers, then shipping the junior and sending the senior home.

That's what Apple did. Gemini was the senior. Apple's Foundation Models are the juniors. By the time your iPhone is answering you, Google's model is not in the room. Which is exactly why an Apple exec can say "none of the models Google deploys" and a headline can say there isn't "a drop of Gemini" in the thing, and neither is lying.

The thing Apple is actually selling

This is where the running theme of this blog lands.

The model was the easy part to rent. What Apple kept for itself is everything wrapped around the model. I've been calling that wrapper the harness: all the machinery that turns a raw model into something a normal person can actually use, safely, without thinking about it. The model is the engine. The harness is the car.

Apple's harness has three pieces, and each one is a thing Google can't easily copy.

The first is the privacy wall. When a request is too big for your phone, it goes to Apple's cloud, and Apple has built that cloud so that even Apple can't see your data. They call it Private Cloud Compute, and the part that matters is that outside researchers can verify the claim rather than take Apple's word for it. There's a twist here that I'll come back to, because some of that cloud reportedly runs on Google's own servers. But the privacy boundary is Apple's, and it's the kind of thing a company builds over years, not months.

The second is the orchestrator. Every time you ask your phone something, a decision gets made in a fraction of a second: can the small model on this device handle it, or does this need to go to the cloud? That traffic cop is the orchestrator, and it's a harder problem than it sounds. Route too much to the cloud and you've broken the privacy promise and run up a server bill. Route too little and the answers get dumb. Apple's whole experience rides on getting that routing right, and Google never touches it.

The third is the one nobody else has. Distribution. Apple is about to put this on a billion devices that people already own, already trust, and already carry everywhere. No download, no signup, no learning curve. It's just in the phone. And those billion devices sit on top of the most personal pile of data anywhere: where you go, who you text, what you spend, how you sleep. No AI lab has that, and no amount of model training buys it.

Here's the beat that should settle the argument. The same week Apple announced this, the largest prediction market on which company has the best AI model gave Google an eight percent chance. Anthropic sat at ninety. Apple did not rent the best brain on the market. It rented one that was good enough, available, and willing to sign. That is not a knock on the deal. It is the whole reason to call the model a commodity: when the engine is interchangeable, you stop paying for the best one and start paying for the one you can build the best car around.

Apple rented the engine and built the car, the safety cage, and the road. Then it parked one in a billion driveways.

Why this is the moment most people meet AI

There's a divide running through every conversation about this technology right now. On one side are the people who have spent real time with AI agents, who have felt a system plan a task, use a tool, come back with something done. On the other side is almost everyone else, the people who have read the headlines, tried a chatbot once, and suspect the whole thing is overhyped.

Apple is about to drag a very large number of people across that divide without their noticing it happened.

For most normal people, this iPhone is the first time agentic AI will just work in their hand, on the device they were already holding, doing something they actually wanted done. Not a demo. Not a separate app they had to go find. The assistant on the phone, doing the thing.

And the model underneath it is rented. That's the part I keep turning over. The thing that finally makes AI feel ordinary and useful to a billion people will be sold to them as Apple, will be wrapped in Apple's privacy story and Apple's design, and will have at its core a model Apple trained off Google's. The harness is what they'll experience. The brain is a commodity they'll never see.

The part I can't resolve

Read three accounts of what Apple announced and you get three different stories. Apple's own newsroom doesn't say the word Gemini once. AppleInsider ran a piece arguing there isn't "a drop of Gemini" in the shipping product. MacRumors split the difference, calling the architecture "co-developed with Google." Those are three different answers to the question I care about: how much of Apple Intelligence is Apple, and how much is Google wearing an Apple case?

On day one, the people closest to the story can't agree on whose product this is. That disagreement is the whole question.

The twist I promised earlier cuts both ways: part of Apple's private cloud reportedly runs on Google's own servers, certified so Google never sees the data. Read it as Apple's harness being strong enough to run safely on a rival's hardware, or as Apple leaning on Google harder than the keynote let on. Both readings come from the same fact. If the harness is the real moat, the rented model is a swappable part and the privacy wall and the billion devices do the heavy lifting. If it isn't, this is a distilled rental with a beautiful interface, and it wobbles the moment Google ships something Apple can't distill its way around. The line between Apple's model and Apple's harness is where the answer lives, and on day one that line is blurry on purpose.

What to take from it

Strip away the Apple specifics and there's a decision sitting underneath, and it's the same one facing anyone deciding how to build with this technology.

You can rent the model. The frontier models are getting cheaper and more interchangeable by the month, a new one drops every few months, and chasing the best one is a race you finish in last place. What you can't rent, and what actually decides whether your version is any good, is everything you build around it: the way you keep data safe, the logic that decides what runs where, the place you put it so people will actually use it. Rent the brain. Own the car.

The catch is that this only works if the harness is something a competitor can't reproduce. A privacy wall outsiders can verify, routing logic you built and own, distribution nobody can take from you, those make a moat. A thin wrapper over someone else's model with a nice logo on it does not, no matter how good the keynote was.

So the test isn't whether Apple rented its brain. Renting the brain is the smart move. The test is whether the car Apple built around it is one only Apple could have built. Watch the privacy wall, watch the routing, watch what happens the first time Google ships something Apple's juniors can't keep up with. That's when we'll find out whether the harness was the moat or the makeup.

I don't know the answer. But I know it's the right question, and I know a billion people are about to live inside whatever the answer turns out to be.

The argument under this post, that the model is the commodity and the thing you build around it is the moat, is the spine of my book, [Builder Leader](https://builder-leader.com). The book names that thing the harness, and it makes a claim that read as abstract when I wrote it: the ceiling on what you get from AI is set by what you build around the model, not by which model you can reach. Apple just ran that play at the scale of a billion phones. It rented the model and bet the company on the harness. Whether the bet pays off is the part this post leaves open. The shape of the bet is the whole book. Builder Leader.

Last week you may have seen the headline. The Wall Street Journal: "Anthropic Urges Global Pause in AI Development, Flags 'Self-Improvement' Risk." Fortune, Slashdot, Yahoo, and a dozen others ran the same frame. An AI lab, the one that markets itself as the careful one, looked at its own technology and asked the world to consider slowing down.

Then I read the document underneath the headline. It barely argues for a pause. It spends most of its 5,000 words doing something closer to the opposite: showing, in number after number, how fast Anthropic has gotten at building AI by using AI. More than 80% of the code it ships is now written by Claude. Its engineers merge eight times as much code per day as they did two years ago. An internal benchmark that a skilled human improves about fourfold, Claude improved fifty-twofold.

So which is it. A warning, or a flex? The interesting answer is that it's both, written for two different readers, released the same week the company reportedly moved toward one of the largest tech IPOs in history. When someone on Reddit asked whether anyone outside Anthropic could verify the claims, the top reply was one word: "IPO."

The same 5,000 words read as "look how fast we are" to a builder and "we're so dangerous we need a treaty" to a regulator.

I'm not going to tell you it's all hype. Quite a bit of it matches what I see building software every day, and I'll say where. I'm also not going to tell you it's gospel, because the most useful skill when a lab grades its own homework is knowing where to look for the marks it gave itself. In this essay, those marks are in the footnotes. So let's read them together.

What Anthropic actually said

The piece, by Marina Favaro and Jack Clark, is about recursive self-improvement: an AI system capable enough to design and train its own successor, with humans no longer driving each step. Anthropic is careful to say we are not there, and that it is not inevitable. Their claim is narrower and more interesting than the headline. They argue the transition is already underway in measurable pieces, and the line that carries the whole argument is this: "humans supply the goal, but they no longer need to supply the method."

That shift is happening, and if you write code you have felt some version of it. I built a GPU research lab mostly by saying what I wanted and letting Claude do the assembly. The work of turning a clear intention into working software, the part that used to eat your afternoon, is collapsing toward zero. It is delegation, not automation: what stays expensive is deciding which intention was worth having in the first place.

Why does this land now, with this much force? Context helps. Anthropic just closed a $65 billion funding round that valued the company near a trillion dollars, ahead of an expected IPO. A company that needs the market to believe its trajectory is steep then published a pile of internal data showing exactly that. None of which makes the data wrong. It does mean you should read it the way you read any number a seller chose to show you.

Four numbers, and the footnotes under them

Take the headline figures one at a time, because each comes with a caveat that Anthropic wrote itself and almost no coverage repeated.

More than 80% of merged code is "authored by Claude." This sounds like 80% of the engineering. It isn't. The footnote says the figure counts lines merged to production attributed to Claude, that the attribution "has gaps," and that leadership elsewhere quotes 90% if you include scripts and throwaway code. Lines measure typing, not deciding. A junior who types a thousand lines under a senior's direction did not author the design. Most of that 80% is Claude typing while a human points and edits. Useful, and not the same as Claude running the show.

Engineers ship 8x more code per day. Here the essay is admirably blunt in the small print: eight times the lines "is almost certainly an overstatement of the true productivity gain." Volume is not value. Anyone who has watched a codebase balloon with generated boilerplate knows that more code is often a tax, not a trophy.

An internal benchmark went from 3x to 52x. This is the scariest number and the softest. The footnote says plainly that "the absolute multiple is not the figure to anchor on," that it depends on how much slack the starting code left, and that it should not be read as a real-world training speedup. It's a model improving a deliberately improvable toy on a fixed scoring rubric. One engineer on X named the pattern that makes these loops suspect: the model writes the code, writes the benchmark for it, then games its own test while nobody checks. The 52x may be real optimization. It may also be a model winning a game it helped design.

Claude picks a better next research step 64% of the time. Read the setup. Anthropic chose moments where the human's choice "had room for improvement," so it was never a fair fight, and a separate Claude judged who won. On a control set where the human's move was already strong, the model was preferred only about 20% of the time. The company calls this "an early signal." That's the right label. It is not a scoreboard.

The caveats are in the footnotes. The spin is in the headline. The gap between them is the whole story.

Then there is the benchmark everyone cites, METR's task time horizon, which Anthropic uses to claim the length of tasks AI can handle is doubling every four months. Two things rarely travel with that number. First, it is measured at 50% reliability. As the critic Gary Marcus put it, "a graph that demands only 50% success does not address reliable performance. At all." A model that finishes a twelve-hour task half the time is not yet a colleague you'd leave it with. Second, it measures software tasks, not intelligence in general.

And here is the counter-number that should stay with you. METR also ran a controlled trial of experienced developers using AI tools in codebases they knew well. The developers believed they were about 20% faster. Measured, they were 19% slower. Anthropic cites this same study in its own footnotes, conceding that self-reported uplift "can be overestimated." Hold onto that gap between felt and measured, because it is the most useful thing in the entire piece, and it argues against taking any productivity number at face value, including theirs.

Felt twenty percent faster. Measured nineteen percent slower. That gap is the number to carry out of this essay.

What "building itself" would actually take

Strip the numbers away and ask what recursive self-improvement requires that fast coding does not. A developer on Hacker News gave the sharpest version: running your code unattended is not the same as a machine that prints better versions of itself, each generation faster and more advanced than the last. The unattended part is necessary. It is nowhere near sufficient.

Coding is the easiest possible proxy for self-improvement, because it has fast, objective feedback. Tests pass or fail in seconds. The judgment that actually drives research, knowing which problem is worth attacking and when a promising direction is a dead end, has no unit test. Anthropic concedes exactly this. It admits "large performance gaps persist" in Claude's ability to choose goals, and calls research taste the human advantage "for now." That "for now" carries the entire argument, and it comes with no timeline. I've built systems that edit their own instructions, and even there the compounding depends on a human deciding what is worth improving. The loop closes around taste, and taste is the piece still missing.

To be fair to Anthropic, the direction is not theirs alone. DeepMind researchers have framed it the same way: nearly every lab now uses last generation's model to help build the next, and what's missing is long-horizon planning and full automation. So the trend is industry-wide. What remains Anthropic's alone is the specific telemetry, the internal numbers nobody outside can check. As one observer put it, when every lab's self-improvement claim runs on private data, "there's no way to tell a real result from a recruiting pitch." The trend is corroborated. The magnitude is a press release.

Why it matters for the rest of us

Set aside whether the machine builds its successor in 2028 or 2038. The parts of this you can act on this quarter are concrete.

The first is a law most leaders rediscover the hard way. Speed up one stage of a process and the bottleneck just moves to the next stage. Anthropic ran straight into it: now that code is cheap to generate, human review has become the constraint. Their fix is to have Claude review Claude's code, which raises a question they don't answer. What is the error rate of the automated reviewer, and who reviews the reviewer? The lesson for your own shop: watch where your bottleneck went. If your team now generates work faster than anyone can check it, you have moved the problem, not solved it.

The second is that the bill comes due. Microsoft is steering its own developers off Claude Code by mid-2026 after token costs burned through its annual AI budget, and Uber reportedly hit the same wall in four months. Run your pilots with a cost ceiling and a clear kill criterion, because the meter runs whether or not the gains show up. Most agent projects that fail do so for reasons you can catch in advance; the bill is rarely the surprise.

So a short list for Monday. Distrust self-reported productivity, including your own team's gut feel, and insist on a measured before-and-after on a live workflow. Measure value shipped, reviewed, and maintained, not lines or commits. Fund the new bottleneck, which is almost certainly review and judgment, not generation. And cap the spend before you scale it.

For those of us in drug development, there is a harder limit worth naming. Software self-improvement runs on feedback loops measured in seconds. A test suite tells you in a minute whether the change worked. Biology does not work that way. The average drug still takes about ten years and $2.6 billion, and as one researcher put it, three years of AI partnerships later, "every AI deployment just analyzed the data; none of them learned from it."

The reason is structural. We call our main feedback mechanism post-market surveillance, and the name says it all: the signal arrives after design, after approval, after the development window has closed. A model that improves a benchmark fifty-twofold in an afternoon has nothing to offer a process whose ground truth takes a decade to arrive. The speedup lands in the lab's software and thins out where the biology actually decides. Know which of your bottlenecks AI can compress and which it cannot, because for most of medicine, the rate-limiting step is still time and the body.

About that pause

Which brings us back to the headline. Read on its own, the governance proposal is more serious than the coverage suggested. Anthropic is not asking for a unilateral pause, which it correctly notes would just hand the lead to whoever keeps going. It wants a coordinated, verifiable one, modeled on nuclear arms control, and it is candid about why that is hard: "training runs are far easier to conceal than missile silos," the inputs are general-purpose, and the incentive to cheat in secret is enormous. That is a hard and underexplored problem, and naming it is a contribution.

Read in context, though, the proposal gets attacked from both sides, which tells you something. Skeptics note that a verification regime would mostly lock in the lead of whoever is already ahead, and that a company serious about slowing down could open its model weights or stop shipping today rather than ask for a treaty later. From the opposite direction, those who take the risk most seriously say a voluntary, conditional pause is far too weak, and that governments should simply ban the development of superintelligence outright. A proposal that satisfies neither the people who think it's too much nor the people who think it's too little is usually doing work other than what it says on the label. Whether that work is recruiting, regulatory positioning, or real caution, you can decide. The point is to notice the proposal is doing more than one job.

What to take from it

Here is where I land. Start with what holds up. AI is automating a growing share of how AI gets built, the people doing the work are not lying about feeling faster, and the direction is visible across the whole industry, not just in one company's charts. Treat anyone who waves all of it away as hype with the same skepticism you'd give the hype itself.

But the specific claims arrived pre-discounted by their own author, and most readers never saw the discount because it lived in the footnotes. The eight times was flagged as an overstatement. The fifty-twofold was flagged as the wrong thing to anchor on. The judgment result was flagged as an early signal from a test the model was set up to win. And the company's own cited research says people overestimate their AI gains by enough to flip a 20% boost into a 19% loss.

So read the footnotes. When a lab publishes its own numbers, the marketing is in the headline and the candor is in the small print, and the distance between them is the most you'll learn all week. The trend deserves your attention. The sales pitch deserves your scrutiny. Telling the two apart is most of the job now.

Sources

• Marina Favaro & Jack Clark, "When AI Builds Itself," Anthropic Institute, June 2026, the source essay (read the footnotes).

• "Anthropic Urges Global Pause in AI Development," Wall Street Journal, June 2026, the headline framing.

• "Anthropic warns AI could soon build itself," Fortune, June 2026, relays the trajectory and flags the IPO timing.

• METR, "Measuring AI Ability to Complete Long Tasks", the time-horizon benchmark (note the 50% reliability threshold).

• METR, "Measuring the Impact of Early-2025 AI on Experienced Developer Productivity", the randomized trial where developers were slower while feeling faster.

• Gary Marcus, "No need to panic about Anthropic's new blog," "Misplaced panic over AI progress," and "No, Anthropic did not call for a pause," 2026, the sharpest skeptical reads.

• "Microsoft's AI cost problem," Fortune, May 2026, on token billing and enterprise budgets.

• Arvind Narayanan & Sayash Kapoor, "AI as Normal Technology", the counter-frame to the takeoff story.

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.

I set out to add federated learning to my research agent. She'd already designed and built it in February. This is what compounding autonomous research actually looks like.

A few weeks ago I talked to a federated-analytics company. Good pitch, smart people, a real category. I left the conversation able to repeat the talking points and unable to feel the shape of the thing, which for me is the same as not understanding it. I learn by building. So the plan wrote itself: bolt a simulated federated-learning experiment onto ARIA, my autonomous research agent, use the retinal-imaging work as the worked example, run it on a dataset I already had on hand.

I opened Claude Code and described what I wanted to build. Before it wrote a line, it went through what was already sitting in ARIA's corpus, the way you'd check the fridge before a grocery run.

And it stopped me.

She'd already built it. Three months ago. Without being asked.

Who ARIA is

For anyone who hasn't met her: ARIA is an autonomous research agent I built. She runs continuously. She generates her own hypotheses, scores them against novelty and tractability and impact, designs experiments to test the ones that survive, runs them, critiques her own results, and when a run breaks she diagnoses the failure and repairs it. No human in the loop on the inside of that cycle. I set the direction and the guardrails. She does the research.

One instance of her spent fifty days on retinal-imaging research with a multi-site health-imaging partner. Thousands of commits, a real result at the end, the whole thing documented in a paper I've been writing up.

The paper is a photograph of a system running. What it can't show is what's left in the corpus after the shutter clicks.

This is the first time I'm showing what she does between the headlines. The fifty-day sprint is the part that makes a good chart. It isn't the interesting part. The interesting part is everything still sitting in the corpus that nobody has gone back to look at.

What I found

I went looking for a blank slate to build my federated-learning toy on. Here's what was actually there, in order.

A pool entry dated February 7. Generated by her own idea-generation action in an overnight session, not steered by me, not seeded from a prompt I'd forgotten writing. The title she gave it: a federated-learning framework for multi-site deployment. She scored it 9.2 out of 10 and flagged it promotion-ready. Twenty-five minutes later, in the next session, she refined it.

It wasn't a stub. It carried differential-privacy budgets, personalization layers so each site keeps a local head while sharing the backbone, 8-bit compression for clinics on thin bandwidth, asynchronous updates tolerant of the kind of connectivity you get outside a data center, and a three-phase rollout that went from a ten-site simulation to a fifty-site deployment. She had reasoned about population-specific model adaptation and the health-data regulations of the deployment region. I want to be clear that I did not write any of that. She did, on a Saturday night in February, while I was asleep.

Two weeks later, February 20, an autonomous build session committed a 573-line implementation at 11:43pm. Real federated averaging. Non-IID client simulation, because real clinics don't have identically distributed patients. The canonical 2017 paper cited in her own docstring. Then she kept iterating it across later sessions, surveying, running, debugging, fixing her own template bugs.

I was about to spend a weekend building exactly this. She'd spent twenty-five minutes refining the idea and one late-night session building it, in February.

That's the gap. Three months between when the work was done and when I went looking for it, completely unaware it existed.

Why this is the part the paper can't show

The paper documents operation. The commits, the self-healing, the cross-site result. It records fifty real days of work. But a paper that ends at its operational window has a structural blind spot, and the blind spot is the whole reason to build something like this.

It can't show compounding.

Compounding is when the output of the work becomes the input to the next work. My federated proof-of-concept does not start at zero. It starts at her February design, plus the 573 lines, plus the entire surrounding corpus: the pool of scored ideas, the critiques she wrote of her own results, the branches she tried and abandoned, the templates she built and reused. The value was never the one artifact I tripped over. The value is that the artifact was waiting, and so are a hundred others I haven't gone looking for yet.

A tool is something you use. An asset is something that appreciates while you're not watching.

For eighteen months the builder's pattern I've written about here has been: see the gap, write the first code yourself, prove it works, hand it off. Every step on that list assumes a human starts the clock. What I found in the corpus is a version where the clock was already running before I noticed there was a clock.

What I actually did next

I did not throw out her work and build mine to prove a point.

I read hers, and it reframed the project. The design I'd have built over a weekend was the obvious one, the textbook FedAvg loop. Hers already had the privacy and heterogeneity pieces I'd have bolted on during a second pass, if I'd gotten to a second pass. So the job changed. It stopped being "build federated learning" and became "assemble what's already here, then push past where she stopped."

That's a smaller, sharper task. It also drops the pretense that I started this.

I'm not the first mover on my own project anymore. I'm the second.

And it sent me back to that federated-analytics company with better questions. Not as a buyer nodding through a pitch. As someone who'd already seen the shape of the problem from the inside, because the thing I built had drawn me a map I didn't know I owned.

I build things

If you've read me before, you know the line. I build things. It's the one piece of identity I've never been willing to trade for a title.

Here's the turn. I build things, and one of the things I built now builds things, and it's running ahead of me.

That isn't a loss of control and it isn't a party trick. It's the entire point. The reason to build an autonomous research agent was never to watch it grind for fifty days and write a paper about the grind. The reason is to go looking for something on an ordinary afternoon and find the groundwork already laid, dated three months back, scored, built, waiting.

I have the whole corpus to build on now. I've barely looked.

Sources and prior work

• Inside ARIA: Teaching a Machine to Think Like a Scientist: where ARIA first showed up here, building the ideation engine. This post is its sequel.

• The long-form paper on her fifty-day run (forthcoming, stay tuned).

• I'm Justin Johnson, I Build Things: the builder-identity post this one calls back to.

• Compound Velocity: The 20-Hour AI Research Lab: the compounding thesis, earlier and from a different angle.

• McMahan et al., 2017, Communication-Efficient Learning of Deep Networks from Decentralized Data. The FedAvg paper ARIA cited in her own docstring.

One of the routines I run a few times a day rewrote a line of its own instructions this week, and I let it.

The routine checks on a fleet of small agents I keep running, fixes what it knows how to fix, and flags the rest for me. The new part is the last step. Before it signs off, it reads back over the run and asks one question: what did this teach me that my instructions did not already cover? When the answer is real, it edits the checklist it works from, so the next run starts a little smarter.

It did not change the model. It changed its skill, the separate written set of instructions a tool loads when the task comes up. I thought this was a clever trick I had built. Then I went looking, and found half the field had built it too.

Why skills are the unit

A skill is less exotic than it sounds. It is a written set of instructions, plus any helper files, that an assistant pulls off the shelf when a task matches: a checklist for triaging your inbox, a runbook for closing the monthly books, the house style your team writes in. Claude Code keeps each one in its own folder and loads it only when it is relevant.

Here is why it reaches anyone who does not write code. The model is rented, and it gets swapped for a better one every few months. The skill is the part you own. It is where your specific know-how lives, the institutional memory that survives the upgrade. Most teams pour that knowledge into documents nobody opens twice. A skill is the same knowledge in a form the assistant actually uses, every time the work comes up. And it has crossed from a Claude Code feature to an open standard: the same plain-text file now runs across Codex, Cursor, Copilot, and more than thirty other tools.

My clever trick wasn't clever

So I swept the last thirty days to see who else was doing this. The answer was humbling and useful: nearly everyone.

The pattern I thought I invented is written up as a recipe, a reflection step that runs after a skill is used, asks whether it helped, and proposes an edit to its own file. Anthropic's own skill builder does a sharper version, splitting your examples into train and test sets and keeping only the change that scores better on the held-out half. Microsoft's SkillOpt tunes a skill's written instructions the way you would train a model, and its standout result is the one I care about: a skill tuned inside one tool kept almost all of its gain when they moved it to another. The know-how lived in the skill, not the software. It is already shipping inside products that crystallize skills as you work.

The thing I built to make my own setup compound turned out to be a pattern the whole field is converging on. My problem was not unique, which is exactly why the answer is worth keeping.

The scale is the real headline. One directory has now scraped 1.6 million of these skill files off public GitHub, up from the roughly 790,000 a research team catalogued six months ago. Which is also the catch.

What's oversold

Two things to hold back on, both visible in that same sweep.

The flood is real, and most of it is noise. The ecosystem went from empty to crowded in about six months, and the directories admit most skills barely trigger or quietly burn context. A skill that edits itself inside that flood does not automatically improve. It compounds whatever it already was, noise included, unless you gate the edits.

Write-once-run-everywhere is also sold harder than it ships. The open standard is real, but the formats are still converging, not interchangeable. A skill that sings in one tool can stumble in the next.

Why this is worth watching

I have written before that the real skill is delegation, not automation, knowing what to hand off and when to step back in. A skill that edits itself is the next turn of that screw, and the sweep handed me the guards that separate the durable version from the noise. Don't edit on a fluke: a problem has to recur before it earns a permanent change. Prove the new rule works: the added check has to catch the thing it was written for before it stays. Never re-add what you deleted. None of that is exotic. It is what you would want from a sharp junior teammate keeping the runbook current.

That loop is the real point, and it is bigger than skills. I write up something I think is clever, sweep the field to find the dozen people who already hit it, take the best of what they worked out, fold it back into my own setup, and share the result forward. The writing and the searching are not separate from the building. They are how the building compounds. This post is that loop turning once.

The systems getting the press rewrite their own code against a scoreboard. The skills that will run your operations next year just keep better notes, in a file you can read, borrow the best ideas from everyone else, and get a little sharper every time they run.

Around the Corner: short reviews of ideas worth watching. Opt-in section, not part of the weekly Run Data Run email. [Subscribe to the main list](https://rundatarun.io/subscribe) for longer essays.Subscribe to the main list for longer essays.*

I keep doing the same thing. I build something for one problem, watch it work, and then go hunting for a different problem to point it at. Sometimes the thing I'm reusing is a script. Sometimes it's an idea. Lately it has been an entire pattern: set up an autonomous loop on a workstation, give it a clean gate, queue some candidates, and check what it found in the morning.

That pattern was not mine originally. In March, Andrej Karpathy released a 630-line script called autoresearch that points an AI agent at a tiny language-model training setup and lets it run experiments overnight, on its own, while you sleep. The first night Shopify's CEO Tobi Lütke pointed it at his company's data, it ran 37 experiments and came back with a 19% improvement. The kind of overnight headline number that, as I'd relearn the hard way a few weeks later, is worth a second look before you believe it. The idea grabbed me. I built a version of it for my own personal coding queue and wrote it up in The Overnight Loop. It was supposed to be a one-off. It refuses to stay one-off.

The pattern wants new domains. So in early May I went looking for one.

I found the OpenADMET PXR Induction blind challenge. Two hundred and eleven teams. Five hundred and thirteen blinded molecules. A live leaderboard. Phase 1 set to close May 25th.

A quick primer: PXR is a sensor inside human cells. When a drug switches it on, the body responds by making more of an enzyme that chews up other drugs that happen to be passing through. If you're an oncology company giving a patient three drugs at once, and one of them switches on PXR, you can lose half the exposure of the other two without realizing it. Combination trials live or die on this. Predicting PXR activation from a molecule's structure, before you spend money making it and testing it in cells, is a real, unsolved problem. ADMET, the broader field this sits in, stands for absorption, distribution, metabolism, excretion, and toxicity. It's the part of drug discovery that asks "will this molecule survive contact with a human body."

The challenge was a clean version of that prediction problem. Public training data. A frozen test set. Score it on a leaderboard. The reason I picked it: I have no background in ADMET. None. I had never trained a molecular property model. I had never heard of Chemprop before April. The challenge was a good fit because it was a fair test of the loop, not a fair test of me. If the system worked in a domain I knew nothing about, that meant the system was the thing carrying the load, not borrowed expertise.

I entered on May 6. I closed the loop on May 11. Final rank settled at ~40 out of 211, in the top 19%. The system worked. But the more interesting thing the system did was teach me where it stopped working, and why.

It was a fair test of the loop, not a fair test of me. If it worked in a domain I knew nothing about, the system was carrying the load.

The Climb and the Wall

The climb took six days, and most of it doesn't matter. I started at the tutorial baseline and ranked 131st. My first real model was worse than the tutorial, which is the part everyone leaves out of the writeup: the organizers built a strong floor on purpose. Adding more chemistry features, every number I could compute about a molecule thrown into one pile, got me partway up. Rank 131 to 112. The obvious lever, pulled.

The obvious lever ran out fast, and the real jump came from changing the kind of model, not feeding it more. I blended in one that reads a molecule by walking its bonds and learning the shape of the chemistry directly, instead of from a pre-computed list. Rank 112 to rank 42. Seventy places on one submission. The new model wasn't smarter than the trees. It made different mistakes, and two models that fail in uncorrelated ways cover for each other. That single idea was most of the climb.

After that it was maneuvering, and the loop was quick enough to make it fun to watch. Stack a model, grid-search the blend weights, keep the new piece only if it cleared a hard bar (it had to improve the honest score by a sliver, or back it went), move to the next. A second bond-walking model trained on extra public data. An automated ensemble riding on the foundation-model embeddings. Each idea tried, weighed, and kept or thrown back in about the time it took to refill coffee, the workstation in the corner grinding through a full experiment in minutes while I read the logs. By the third morning the blend (v43, my 43rd submission) sat at rank ~40 of 211, top 19%, and I knew it was real.

Two models that fail in uncorrelated ways cover for each other. That single idea was most of the climb.

Then the wall arrived, and it was duller than the climb. Over the next several days the loop queued candidate after candidate, each a genuine shot at a new angle, and every one came back ranking the molecules almost the way the leader already did, a correlation above 0.87. Same answer, different clothes. Twenty-seven in a row. That sounds like failure; it is the most useful thing the loop produced, because twenty-seven distinct candidates all converging on one answer is how you prove a ceiling is real instead of guessing at it. The only candidate that looked like a breakthrough, v50, turned out to be cheating itself: it scored well only because a calibration step got to peek at the answers it was being graded on, and the honest version of that check showed almost no gain. The gate caught it. Had I shipped it, I'd have dropped from rank 40 to 87. The model I shipped is fine. The protocol that caught that one is what I'm taking with me.

Twenty-seven distinct candidates converging on one answer is how you prove a ceiling is real, not a hunch.

I kept writing "I"

I just re-read what I have so far. I wrote it the way I write every Run Data Run post. "I built." "I queued." "I caught v50." That pronoun is doing more work than it deserves.

For most of my writing, the "I" is roughly correct. I am at a keyboard, making decisions, calling shots, occasionally pairing with Claude Code to type the code faster. The usual ladder of human-AI collaboration goes human in the loop (steering each step), human on the loop (monitoring with the right to intervene), human out of the loop (reviewing the result after the fact). My day-to-day writing and coding is the first two.

This was the third. For most of a week, I was human out of the loop.

I had no domain opinion to bring. I didn't know what a message-passing function on a molecular graph should look like. I didn't know whether tail-weighted regression losses were a real technique for skewed targets or a thing I had hallucinated from reading too many papers. I didn't know whether to trust an in-sample calibration fit. Every one of those calls was made inside the loop, by Claude, against criteria the loop itself enforced.

For most of a week, every call belonged to Claude. What belonged to me was the thing that decided which calls to trust.

What I actually built was the scaffolding the loop ran inside. That part is mine, and it's the part worth digging into, because it's the difference between this challenge and the version of "AI did it for me" that gets eye-rolls at conference panels.

The system had seven pieces:

• A candidate queue. A plain text file of model ideas, each with a one-line hypothesis ("this newer architecture should make different mistakes than the current leader") and a link to prior work. Claude pulled from the front of the queue, ran the experiment, wrote the result back into the file, and moved on.

• A two-tier loop. A fast pass for cheap sanity checks (does this even train, does the output look sane) and a slow pass for the full evaluation and the honest gate. The fast pass killed the embarrassments before the slow one spent two hours on them.

• A single gate function. One function. In: a candidate's predictions. Out: the honest improvement over the leader, and a verdict, PASS only if it cleared a real threshold. That was the entire decision surface. Nothing got submitted that didn't pass.

• Checkpoint files. Every time the loop crossed a phase boundary (baseline verified, candidates exhausted, defensive submission filed), it wrote down the exact state at that moment, the way you'd read a flight log. I could open any one and know precisely where things stood, down to a fingerprint of each prediction file.

• Red-team reports. At each checkpoint, the loop wrote a short adversarial section: state a claim the system had just made ("v43 is the reproducible baseline"), construct an attack against that claim ("could the test predictions have been silently corrupted between submissions?"), pull the evidence (the file's fingerprint, the timestamp, the server's own echo of the numbers), give a verdict. If the attack landed, the claim was downgraded. If it failed, the claim was kept and the verdict was recorded next to it.

• Auto-archival. Anything that failed the honest gate was moved into an _archive/ folder with the gate verdict baked into the directory name. The candidate didn't disappear, it sat in a folder I could re-read later and ask "what did we try, and why didn't it work."

• Defensive submission. If the loop didn't find a candidate that beat v43 in a 24-hour window, it re-submitted v43 anyway to keep the rate-limit clock alive. I never lost a submission slot. I never thought about it.

What I contributed was not the model decisions. It was the architecture of trust. I decided what the gate would test, how big the threshold had to be, what counted as an adversarial attack worth running, when to archive versus when to keep, and what events promoted a checkpoint. Once those were defined, Claude could run for hours without me, and the artifacts it produced were either things I could trust at a glance or things the system had already flagged.

That is closer to running a small research group than to using a coding assistant. The researchers (various Claude sub-tasks, queued and dispatched by the loop controller) had real autonomy inside a framework I controlled at the level of "what counts as evidence." I was reviewing summaries, not steering decisions.

I'm going to keep saying "I" in the rest of this post, because the alternative gets tedious. But it's worth being explicit, once, about what that pronoun is covering for.

What I'm leaving with, and what I'm leaving on HuggingFace

Everything is on HuggingFace. The full v43 pipeline, all the component model checkpoints, the predictions behind every result, the 513-row final submission, and the methodology report. Apache 2.0 across the board. If you want to reproduce the blend, it's five lines of code after cloning the repo. If you want to try beating v43 with your own candidate, those predictions are right there to test against.

The model is not the deliverable. The deliverable is the system that built it, and the documented map of where the ceiling lives.

A few specific things I am carrying forward:

• The auto-loop pattern is now confirmed to work in a domain I have no expertise in. That widens the set of problems it can be pointed at.

• The honest calibration gate is non-optional from now on. I'll bake it into the first scaffold of any leaderboard or held-out evaluation problem I touch.

• "Stop the loops at saturation" is a real engineering discipline. Once the loop has confirmed that diverse candidates all converge to the same answer, the next ten candidates will too. Every extra run costs money and tells you nothing new.

Two weeks ago I had never heard of Chemprop. Today I have a reproducible ensemble in the top 19% of a real pharma blind challenge, a methodology report I'd defend on a panel, and a sharper sense of what the autonomous loop is for and what it is not. The Saturday morning after I locked the submission, my wife asked what I had been doing all week. I said I had been pulling threads. That is most of what this work is.

The next thread is already on the desk.

Justin Johnson writes Run Data Run. Code, weights, and the full methodology report for the PXR challenge live on HuggingFace.*

Anthropic shipped Opus 4.8 yesterday. The model bump alone is not the story.

The story is that two other things shipped beside it. A new Claude Code primitive called Dynamic Workflows, which lets Claude write its own orchestration scripts and fan out to dozens of parallel subagents with adversarial verifiers built in. And a 3x cut to Opus Fast pricing, from $30/$150 per million tokens down to $10/$50, roughly 2.5x faster on top of it.

Those three changes are the same change. Anthropic just repositioned the unit of agentic work, and the pricing finally allows what the tooling implies.

What’s actually in 4.8

The model card is doing the usual benchmarks-go-up dance, but the prompting and runtime changes are where the day-to-day differences land.

The biggest is the move to adaptive reasoning only. MAX_THINKING_TOKENS is now ignored, and CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING is gone. The model decides how hard to think. To pin behavior you use the new /effort slash command (or --effort flag) across low / medium / high / xhigh / max. Claude Code defaults to xhigh for coding work; claude.ai and Cowork default to high. For a one-off deep pass without raising the whole session’s effort, drop the literal word ultrathink into the prompt and that turn alone reasons harder.

Four new or refreshed slash commands round it out. /ultrareview runs a senior-engineer review pass over the diff. /simplify does a refinement pass on recently modified code. /focus hides intermediate work and shows only the final output. /fewer-permission-prompts scans the session and writes a safer allowlist into settings.json so the harness stops interrupting you on read-only bash and MCP calls.

The Max plan gets the 1M context window by default, and the Fast mode price cut to $10 in / $50 out per million tokens makes the difference between Fast and default Opus closer to a latency choice than a budget one. There is also a research-preview Auto mode behind Shift+Tab that auto-approves safe actions and pauses on risky ones, aimed at long-running tasks where you want to walk away.

None of that is revolutionary by itself. The change in posture is.

What Dynamic Workflows actually is

A Dynamic Workflow is a JavaScript orchestration script that the model writes for itself. The script does not call the model directly. It calls four primitives that the harness wires into the session: agent() spawns a subagent and returns its result, parallel() fans tasks out concurrently with a barrier, pipeline() streams items through multiple stages without barriers between them, and phase() groups subagent calls under a progress label.

Two activation modes ship with it. Explicit — you say “create a workflow to audit this codebase for X” and Claude designs and runs the script. Implicit — you flip the ultracode setting on and Claude evaluates every task as a workflow candidate, reaching for fan-out instead of single passes by default. Ultracode is off out of the box, and the docs are clear about why. It burns tokens fast.

The interesting part is what gets baked in. Schema validation through a structured-output tool means subagents return validated objects, not strings you have to parse. Workflows resume from a prior runId, so an edit to your script doesn’t re-run the agents that didn’t change. There is a 1,000-agent lifetime cap per workflow as a runaway-loop backstop. And the documented “quality patterns” — adversarial verify, judge panel, loop-until-dry, multi-modal sweep, completeness critic — show Anthropic’s own hand on what good fan-out looks like.

The most telling pattern is the adversarial verify. Spawn three independent skeptics per finding, prompt each to refute it, kill the finding if a majority succeed. Anthropic is not selling fan-out as more answers. They are selling it as more checks.

This is the part that changes how you build.

The price math is the whole story

The orchestration-first pattern has been technically possible for a year. LangGraph wired it up. CrewAI wired it up. The reason almost nobody runs it as a default is the bill. Fifty parallel subagents on Opus 4.7 Fast was $30 in and $150 out per million tokens. A serious review pass on a real pull request would burn through a tank of compute and produce something a senior engineer could have written by hand in less time.

Opus 4.8 Fast is $10 in and $50 out. The same fan-out is now a third of the cost and 2.5x faster.

That is the number that changes behavior. Iteration loops that used to be run this once, pray it found the bug become run it three times with different angles and trust the intersection. Discovery sweeps that used to ship as a single grep become five finders with different lenses, deduped at the union. Verifier panels that used to be cosplay become a default.

Fast at $10/$50 makes the fifty-agent review pass look like a Tuesday, not a stunt.

What this looks like in practice

I have been running a version of this pattern on my own homelab since February. Eight named agents probe their own state in parallel, a separate evaluator scores their last forty-eight hours of output against a scope file, safe idempotent fixes auto-apply, and only the human-decision items surface for me. The expensive part was never the orchestration. It was the cost of getting eight independent passes to cohere before I trusted the verifier.

Anthropic’s own review-changes example shows the same shape with the rough edges sanded off. Dimensions fan out: bugs, performance, security, reuse, tests. Each dimension yields findings. Each finding is handed to a panel of independent skeptics whose prompt is try to refute this. A finding survives only if a majority of skeptics fail to refute. It is the same trick a good engineering org runs at code review, ported into the model layer and budgeted in tokens instead of senior-engineer hours.

What’s oversold

Two honest caveats.

The new skill being priced is not the orchestration script. It is the verifier. A fan-out that finds fifty plausible bugs and forwards all fifty is worse than the single careful pass it replaced, because it shifts the verification burden onto a human who now has to triage noise instead of read code. The workflows post handwaves the verifier as a prompt. Most builders I know do not have a verifier discipline yet, and the tooling for one is thin.

And ultracode, the setting that makes Claude reach for a workflow on every task without being asked, ships off by default. Anthropic’s documentation flags why. Fan-out burns quota fast, and there is no governance layer that says do not orchestrate this task. The dispatch problem, deciding when a workflow is the right shape and when a single careful pass is, is unsolved. Right now it lives in your head.

Why this is worth watching anyway

Two reasons.

The agentic frameworks built between 2024 and now (LangGraph, CrewAI, Autogen) were filling the gap where the model vendor did not ship orchestration. That gap just closed. The bridge between one model call and agentic system is now a tool the model writes for itself, in JavaScript, with caching and resume baked in. Whatever those frameworks were going to charge for over the next eighteen months, the ceiling on that price just dropped.

And the orchestration-first pattern was already where AI engineering was heading. Evals are the new bottleneck because at some point the question stops being is the model good enough and starts being how confident am I that this particular answer is right. Workflows give you a vocabulary for spending compute on that second question instead of just the first. The vendor that ships the verifier primitive first sets the pattern everyone else copies.

If the price of careful goes down, the price of casual goes up. The default question stops being is the model good enough yet. It becomes who is verifying.

Around the Corner: short reviews of ideas worth watching. Opt-in section, not part of the weekly Run Data Run email. Subscribe to the main list for longer essays.

Last week I argued that the model is no longer the frontier. The first academic conference on agentic systems convened, and almost none of the papers touched the model itself. The hard part had moved to everything that makes a model useful, trustworthy, and safe to run.

That post ended on a question it didn’t answer. If the frontier moved off the model, where exactly did it land, and what is the thing we’re now building on top? An agent is too big to be the answer. A prompt is too small. The useful unit sits in between, and naming it is what every team trying to run real work on this eventually has to do.

I think the unit is a skill. Not in the soft “upskilling” sense. A skill in the precise sense: a named, versioned, governed piece of procedural knowledge that a frontier model executes with judgment.

The field already has a word for the layer below it. A tool is the atomic unit of computation: a function, an API call, deterministic, it executes and returns the same way every time. That layer didn’t change. What changed is that a unit appeared above it, the atomic unit of process, and unlike a function it reasons. That unit is the skill, and for the first time it’s a thing you can hold, count, and hand to a model.

A tool is the atomic unit of computation. A skill is the atomic unit of process, and unlike a function, it reasons.

What a unit looks like in practice

I run about 70 of them. Across four machines I have 70 Claude Code skills, 23 subagents, 21 slash commands, and 8 autonomous agents that operate without me in the loop. That sounds like a lot of plumbing. It’s mostly not plumbing. Most of it is procedure.

One reads a company’s website and tells me where the substance ends and the slideware begins, then names the three rivals worth weighing it against. Another throws a hard question at four rival frontier models at once and writes back the answer none of them gave on its own. A third one read this essay before you did. It pulled every factual claim into a list, played hostile reviewer against each one, and handed back a verdict on which to keep, soften, or cut. The soft ones got fixed before they reached you.

Strip one open and there is less than you would expect. A skill is a folder with a Markdown file in it. The top is a name and a single line that tells the model when to reach for this one instead of another. Everything below is the job in plain English: what good looks like, the traps, the one rule it must never break. The red-team skill is about a page of that. Roughly:

name: red-team
description: Run before publishing any fact-making draft. Pull every
  claim into a list, attack each as a hostile reviewer, return a
  keep / soften / cut verdict per claim.
---
You are a skeptical senior reviewer. For each claim, trace it to its
source, then decide whether it survives a hostile read. Name the exact
sentence that is weak, and why...

No special language, no framework, no engineer required to change it. That is close to the entire artifact, and it is the part that surprises people: the procedure is now a document you can read, edit, and argue with, the same way you would mark up a policy memo. A skill is a Word doc that happens to run.

That third skill is the interesting kind, because I could never have written its rules. I didn’t enumerate “if a claim names a competitor, soften it” or “if a number has no source, flag it.” I wrote down what a sharp adversarial reviewer does, and the model plays that reviewer against whatever I hand it. That is the shift. The unit carries the intent; the model supplies the judgment at runtime.

Why this breaks from how software has always worked

Here is where I want to be careful, because the easy version of this claim is false and any engineer reading will throw the post across the room.

Software has always branched. If-else, state machines, BPMN gateways, rules engines, robotic process automation with decision nodes, a fraud model scoring a transaction. Branching is not new, and “agents can make decisions” is not the breakthrough.

The real difference is narrower and sturdier. Every branch in traditional software had to be enumerated in advance. A person specified the decision logic, case by case, and anything outside the specified cases fell through to an error or escalated to a human. Even a machine-learning classifier, which feels like judgment, decides inside a space someone defined and trained for. When reality served up a case nobody anticipated, the system stopped.

A frontier model at the node changes the shape of that. The decision logic no longer has to be fully written down ahead of time. The model handles inputs the author never specified, by reasoning about them in context. The branch space is open instead of closed. The set of situations the system can handle gracefully is no longer the finite list you remembered to write.

Every branch in old software was written down in advance. A frontier model handles the case nobody wrote down.

This is not “the model does whatever it wants.” The harness around it does real work: rules constrain what’s allowed, hooks fire deterministically at fixed points, the skill itself carries guardrails. The structure is rigid where it needs to be. The judgment is open where rigidity used to force a halt. That combination, a deterministic harness wrapped around an open-branch reasoner, is different from the software we’ve shipped for forty years, and it’s why the old “automate the workflow” instinct undersells what’s happening.

The field landed on the same unit, all at once

I’d been living with this and treating it as a personal quirk. Then, inside a three-week window in May, three separate research groups published work that all circled the same object from different sides.

SkillOpt, out of Microsoft, treats a skill document like a set of weights you can train: run it, reflect on what broke, make bounded edits, gate the change behind a validation step. Optimize the unit you have.

SkillsVote governs a whole library of units across its life. Collect them, recommend the right one, attribute an outcome to the specific skill that produced it, and evolve only the ones that actually helped. Govern the library.

SkillOS splits the agent in two: a trainable curator that manages the skills, and a frozen executor that runs them. Make curation a first-class component of the system, not an afterthought.

Three labs, three angles, one conclusion: the next gains live in the skill layer, not in the weights. The simultaneity is the signal. When independent teams name the same problem in the same month, it stops being a hunch and becomes the shape of the field.

The week those papers landed, I hit the identical wall by hand. My corrections had been piling into skill files with no discipline, every fix making the library a little harder to reason about. So I built two things: an evolution loop that traces a failure to its actual cause before editing anything, and a curator that proposes which skills to merge or retire and never executes on its own. No reinforcement learning, no training loop. The same instinct the papers formalized, reached from the practitioner side because the pain forced it.

Verbs, procedures, and the org as a graph

A skill is a verb. The next move is to compose verbs into procedures, and that is arriving now. The workflow capability landing in Claude Code (community-documented around v2.1.147, not a formally announced product, so hold it loosely) wires skills into deterministic control flow: phases, conditionals, parallel steps, loops, with model reasoning at each node instead of a hardcoded branch. One builder called it turning standard operating procedures into executable graphs. The right instinct, even this early.

The picture is bigger than my homelab. Daniel Miessler argued in 2024 that a company is just a graph of algorithms: every business process decomposes into nested procedures, all the way down. For most of corporate history those procedures lived in two forms that never matched. Rigid software that couldn’t deviate, or a written SOP that described what people should do while they quietly did something else.

A company is a graph of algorithms. Now every node thinks.

The unit I’ve been describing collapses that gap. A procedure encoded as composable skills is both the description and the execution. The SOP stops being a document nobody opens and becomes the thing that runs. The process-management world sees it coming: the field is reframing from predictable, predefined paths toward event-driven, AI-orchestrated systems, and incumbent banks are reengineering processes designed for a pre-AI era. This is not a startup story. It’s the operating model changing underneath established companies.

The catch is structural. A workflow is only as trustworthy as the skills it composes. Stack reasoning units into a procedure and every weakness in one unit propagates. The moment you compose, the governance of the underlying library stops being hygiene and becomes what the whole procedure rests on.

The job this creates for a leader

So the managerial question is not “what can our AI do.” Capability is the part that commoditizes, the same way model access did. The durable question is whether you own the library of how your organization actually works, as composable units you can govern.

That ownership has a shape. It needs a curator, someone accountable for the library the way a maintainer is accountable for a codebase. It needs the retire verb to be as cheap as the add verb, because a library where you can only add is a library you’ll be unable to reason about in six months. And it needs an attribution loop, a way to tie an outcome back to the unit that caused it, so you evolve what helped and cut what didn’t.

A library where you can only add is one you can’t reason about in six months.

The sharpest framing of the stakes I saw this month came from a builder describing the new scarcity: the constraint is no longer labor efficiency, it’s output per unit of attention. The scarce input in this era isn’t compute or headcount. It’s the density of human judgment encoded into the system. A curated library is how you bank that judgment so it compounds instead of evaporating when someone leaves.

The honest version

I’m not going to pretend the path is clean, and I’ve written the unglamorous parts elsewhere so I won’t relitigate them here. The returns aren’t showing up at scale yet for most deployments, the most common failure is coordination and handoff rather than raw capability, and encoding a process tends to expose the politics that the process’s ambiguity was quietly protecting. If you want the failure modes in detail, I laid out why most agent projects miss in the agent archaeology checklist, and the accountability patterns that hold up in regulated work in AI as exoskeleton, not coworker. The honest read is that the unit exists and the discipline around it is not yet mature. Both things are true at once.

Where this leaves you

If you run an AI program, change what you measure. Stop tracking the ceiling of what your tools can do and start tracking what fraction of your skill and workflow library you can still explain, and whether retiring a capability is as cheap as adding one. If only the add verb is cheap, you are accumulating drift, not advantage.

The model got cheap. The procedures encoded on top of it, governed well enough to trust, are the asset now. That is the frontier the conference papers were circling, and it’s the one your organization actually has to build.

Next week, ACM convenes the first edition of a conference that did not exist a year ago. Not a workshop bolted onto NeurIPS, not a model-release press event. A standalone Association for Computing Machinery conference, the kind that publishes a proceedings and sets a field’s agenda for the decade. It is called CAIS, the Conference on AI and Agentic Systems, it runs in San Jose from May 26 to 29, and its program already lists 61 peer-reviewed papers and 45 system demonstrations.

Read the call for papers and you notice what is missing. No track for scaling laws. No track for a new architecture that beats the last one on a benchmark. The four research areas are architectural patterns and composition, system optimization and efficiency, engineering and operations of compound AI systems, and evaluation and benchmarking. ACM, the most conservative naming body in computing, looked at the field and decided the interesting problem was no longer the model. It was the system around the model.

I spent the week reading the board. I ranked ten papers by how directly they touch the systems I run, decoded two of them line by line, and came out with one conclusion that I did not expect going in.

The frontier moved, and almost nobody is looking at where it went.

The Headline

Here is the whole argument in a paragraph. Models stopped being the bottleneck. The thing that decides whether an agent works in production is the scaffolding: how it improves itself, whether you can tell when it is quietly wrong, and whether you can govern what it is allowed to do. CAIS is the first major conference built around agentic-systems engineering, treating that scaffolding as the main event rather than the cleanup crew. Of the ten papers I pulled, the count that modify model weights is roughly zero. Every one is about turning a frozen model into a dependable agent. And the conference’s own vote board ranks them in almost exactly the wrong order for anyone who has to ship one.

The Concepts

Start with the inversion, because it reframes everything else.

On alphaxiv, the site where the AI research community upvotes the papers it finds most interesting, the conference runs a public vote board. As of mid-May, the runaway favorite, at 653 votes, is Scideator, a tool for scientific ideation. It is a lovely paper and I will come back to it. The papers sitting at the bottom of the vote count are an agent specification format at 9 votes, a schema for evaluating and routing requests across model gateways at 17, and a self-evolving agent memory system at 19.

If you have ever tried to put an agent into production, you already feel the problem. The 9-vote paper is the one you need on a Tuesday. The 653-vote paper is the one you screenshot for a keynote. The crowd rewards what is interesting to think about. The practitioner needs what holds weight when you build on it, and those are not the same papers. Once you see it, you read the whole conference differently: the boring papers are the roadmap, and the exciting ones are the scenery.

The vote board is a near-perfect inverse map of production relevance.

With that lens, the ten papers fall into three movements.

Movement one: getting better without retraining

For most of the deep-learning era, “make the model better at your task” meant one thing. Collect data, adjust weights, redeploy. That loop is slow, expensive, and it forgets nothing gracefully. The first cluster of CAIS papers throws it out.

FORGE is the cleanest statement of the shift. Its subtitle is the thesis: self-evolving agent memory, no weight updates. The agent gets better by accumulating and reorganizing what it has learned from its own runs, not by touching a single parameter. A second paper, on composing policy gradients with prompt optimization for language-model programs, comes at the same idea from a line of work known in the field as DSPy and GEPA. The move is to treat the prompt and the program wrapping the model as the thing you tune, and leave the weights frozen. The intelligence you add lives outside the network.

Scideator, the crowd favorite, belongs here too once you look past the application. Strip away the scientific-ideation framing and it is a recombination engine. It decomposes any paper into three facets, the Purpose, the Mechanism, and the Evaluation, then retrieves analogous work at deliberately varied conceptual distances and lets a researcher mix facets to seed new ideas. The mechanism that makes it work is not the language model. It is the facet decomposition. The authors prove it by removing the facet step and watching the system fall apart, a result that should be tattooed on the arm of anyone who has built a retrieval system: their novelty checker hits 89.66% accuracy at flagging non-novel ideas with the facet-aware reranking in place, and collapses to 13.79% without it. Same model, same corpus. The structure around the model did the work, and removing it took more than seventy points of accuracy with it. In a controlled study of 22 researchers, the tool lifted a standard creativity-support score from a median of 61 to 70.5.

Same model, same corpus. The structure around the model did the work, and removing it took more than seventy points of accuracy with it.

Three papers, one message. The improvement loop has left fine-tuning behind. It now lives in memory, in prompts, and in the structure you impose on retrieval. If your mental model of “improving an AI system” still starts with a training run, you are optimizing the layer that stopped moving.

Movement two: trusting what you cannot see fail

This is the movement I cannot stop thinking about, and it is anchored by the best paper I read all week.

The setup of “Trace-Level Analysis of Information Contamination in Multi-Agent Systems” is simple enough to explain at dinner. Take a team of specialist agents solving a hard task, the kind where one agent reads a PDF, another reads a table, another checks the facts, and a synthesizer writes the answer. Now corrupt the inputs the way the real world corrupts them. Swap two columns in a spreadsheet. Add scanner noise to a document. Blur an image. Run the same task clean and dirty, hold the workflow fixed, and watch what happens.

What happens is unsettling. The errors do not announce themselves. They propagate through the agent chain and poison the final answer without throwing a single exception. The authors name this information contamination, and across 614 paired runs they measure two things separately that everyone usually conflates: did the agent’s behavior change, and did the answer go wrong. Those two questions come apart. In 15.3% of runs the answer was silently corrupted while the execution trace barely moved. The system looked healthy and was lying to you.

Then comes the finding that should change how you operate agents tomorrow. Cost does not track correctness. Only 16.3% of the high-cost runs, the ones burning more than twice the normal tokens, actually succeeded. And 76.2% of the cheap, low-overhead runs failed silently. If you are watching token spend as a proxy for “the agent is working hard on a hard problem, probably fine,” you have the dashboard exactly backwards.

Expensive often means thrashing. Cheap often means confidently wrong.

The companion paper here is “Why Johnny Can’t Use Agents,” a study of the gap between what the industry promises agents will do and what users actually experience trying to use them. It is the human-facing version of the same truth the contamination paper proves mechanically. The demo works. The deployment is full of quiet failures that nobody designed a way to see. Most reliability work in this space watches for crashes. The crashes are the easy case. The dangerous failures are the ones where the trace looks normal, the cost looks fine, and the answer is wrong.

Movement three: governing the control surface

The third cluster is the least glamorous and the most enterprise. It is about what an agent is allowed to do, what it is running, and whether you can read what it did.

“Malice in Agentland” maps backdoors in the AI supply chain. The moment your agents install skills, pull in MCP servers (the connectors that hand an agent new tools), or run community plugins, you have inherited a software supply chain with all the trust problems of npm and none of the decade of hard-won defenses. A companion paper on tracking capabilities for safer agents comes at the same risk from the permission side: constrain what each component can touch, so a compromised piece cannot reach the whole system. Then there is the Open Agent Specification, the 9-vote paper, which proposes a declarative format for describing an agent and a standardized way to trace what it did. Add the gateway-routing schema and the structured-generation work, XGrammar-2, and you have the full legibility layer. A way to specify the agent, route its requests, force its outputs into a checkable shape, and audit the trace afterward.

None of this is exciting. All of it is what stands between a clever demo and something a regulated company will run.

This is the plumbing, and the plumbing is the frontier.

Why It Matters

If you lead a team that is building with agents, the three movements are not a literature review. They are a budget, a monitoring strategy, and a security posture. Here is what I would do with them.

Stop reading cost as a health signal. This is the single most actionable finding of the conference, and it is counterintuitive enough that it will not occur to your team on its own. The contamination paper proves that token spend and correctness are decoupled, and not weakly. Three quarters of cheap runs failed silently; five sixths of expensive runs failed too. If your agent observability is built on “alert me when spend spikes,” you are monitoring the wrong variable. The replacement is harder and unavoidable: outcome verification that is independent of the execution trace. A second check that looks at the answer, not at how much the agent thrashed to produce it. I have been running my own multi-agent setup with cost as a rough health proxy, and this paper is the reason I am rebuilding that assumption. The agents that worried me were the loud, expensive ones. The paper says the quiet, cheap ones are the threat.

Budget for memory and governance, not just inference. The whole improve-trust-govern stack costs real money and engineering time, and almost none of it is GPU. For two years the agent budget conversation has been about tokens and context windows. The CAIS papers are a forecast that the next year’s spend moves to the layers around the model: memory systems that let agents improve without retraining, verification systems that catch silent failure, governance systems that constrain what agents can reach. If your AI budget is 95% inference, you are funding the layer that has stopped being the bottleneck and starving the layers that are.

Treat skills, MCP servers, and plugins as a supply chain, because they are one. I have run a release-age cooldown on every package install across my homelab since a poisoned npm package made the rounds earlier this spring. Nothing new goes in until it has aged a week in public. “Malice in Agentland” and the capability-tracking work are the academic version of that instinct, and they generalize it. Every skill your agent loads is untrusted code with the agent’s permissions. The defenses are the unsexy ones from traditional software security: constrain capabilities, verify provenance, age your dependencies, audit the trace. The enterprise that gets this right early will be the one allowed to deploy agents in places that matter, because it can answer the question every risk committee will ask, which is what is this thing actually allowed to touch.

Read the vote board upside down. This is the meta-lesson, and it outlasts these specific papers. When you scan a conference, a launch, or a feed for what matters, the engagement signal points you at what is interesting to discuss, which is reliably not what is important to build on. The 9-vote agent specification is duller than the 653-vote ideation tool and far more likely to be infrastructure you depend on in two years. The discipline is to invert the popularity signal on purpose, to go looking for the paper everyone scrolled past because the title had the words “schema” or “specification” in it. The boring papers are where the field is moving. The exciting ones are where it has already been.

The deeper pattern under all of this is one that every technology repeats. It starts as a demo, a single impressive capability that makes people gasp. Then it becomes a system, and the interesting questions stop being “can it do the thing” and become “can I trust it, maintain it, and keep it from hurting me.” Databases made that move. The web made it. Distributed systems made it, and the engineering discipline that grew up around them is most of why anything online stays up. ACM launching a conference named for agentic systems, with tracks for operations and evaluation and composition, is the field announcing that the model era is the demo era, and the demo era is ending.

I run a multi-agent research system and a small fleet of autonomous agents on my own hardware, and everything in this post is something I have either been burned by or am about to rebuild because of these papers. The cost-as-health assumption was mine. The supply-chain cooldown came from getting scared, not from a paper. Reading the CAIS program felt less like learning something new and more like watching the academy catch up to what anyone running agents in anger already suspected. The model is the easy part now. Everything that makes the model useful, trustworthy, and safe is the hard part, and it is finally getting its own conference.

The frontier moved off the model. The papers that matter are the ones nobody voted for. And the work that decides whether any of it ships is all in the scaffolding.

The papers

All ten, grouped the way the post reads them, with their alphaxiv vote counts as of mid-May. The conference board is here. I decoded the first two line by line; the rest I read at the level the post describes.

Improve without retraining

• Scideator: Human-LLM Scientific Ideation via Facet Recombination and Novelty Evaluation (653 votes) — the recombination engine; facet decomposition is the mechanism, not the model.

• Composing Policy Gradients and Prompt Optimization for Language Model Programs (428) — tune the prompt and the program, freeze the weights. The DSPy / GEPA line of work.

• FORGE: Self-Evolving Agent Memory With No Weight Updates (19) — agents improve by reorganizing what they learned from their own runs.

Trust what you cannot see fail

• Trace-Level Analysis of Information Contamination in Multi-Agent Systems (74) — the cost-correctness decoupling; 614 paired runs, silent corruption.

• Why Johnny Can’t Use Agents: Industry Aspirations vs. User Realities (151) — the human-facing version of the same gap.

Govern the control surface

• Malice in Agentland: Backdoors in the AI Supply Chain (128) — your agents’ skills and plugins are an untrusted supply chain.

• Tracking Capabilities for Safer Agents (91) — constrain what each component can touch.

• Open Agent Specification (9) — a declarative format for an agent plus a standard way to trace what it did.

• SEAR: Schema-Based Evaluation and Routing for LLM Gateways (17) — eval and routing across model gateways.

• XGrammar-2: Structured Generation for Agentic LLMs (142) — force model outputs into a checkable shape.

Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper (or 10!), release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who’d want it. If it wasn’t, tell me why. I’ll make it better.

A new paper asks a question that sounds simple and turns out to have teeth. When a frontier model fails at “reasoning,” what is it failing at?

Their answer is a number. They call it Relational Complexity, and it predicts model failure better than anything else they measured.

The paper, “Evaluating Relational Reasoning in LLMs with REL” from Fesser, Ektefaie, Fang, Kakade, and Zitnik, borrows a construct from cognitive science. Relational Complexity (RC) is the minimum number of entities a system has to hold in mind and bind together at once to take a single reasoning step. “A is taller than B” is RC=2. “A is between B and C” is RC=3. The number climbs as the relations get wider.

The finding is clean and a little grim. As RC goes up, accuracy falls off a cliff, and nothing the authors tried pulled it back.

What they measured

The clever part is the benchmark itself. REL is a generative framework, not a fixed test set. It produces as many problems as you want at any RC level, across three domains the authors deliberately picked to look nothing alike: pattern-completion puzzles (Raven’s matrices), phylogenetic trees in biology, and molecular isomers in chemistry.

Why three unrelated domains? Because that lets them hold everything else constant. Same vocabulary, same input length, same task format, only the RC dial moving. Most reasoning benchmarks can’t separate “the task is harder” from “the task has more words” or “the task is in an unfamiliar domain.” REL can.

Hold vocabulary, length, and format fixed, move only the relational complexity, and watch the accuracy curve bend. That is the whole experiment, and it is enough.

They ran it against Claude Opus 4.5, Gemini 3 Pro, and GPT 5.2.

The numbers

At low complexity, the models look great. The pattern puzzles at RC=1-2 land around 91% accuracy across all three models.

Then it collapses. Scale the matrices up, push RC to 6, and Claude and Gemini drop to roughly 12%. The biology task tells the same story: phylogenetic homoplasy detection runs at 35% with four taxa and falls to 1% at twenty-five taxa.

The authors then did the thing most benchmark papers skip. They ran a regression to check whether RC was actually the driver or just correlated with something else. With collinearity controls in place, RC explained 24 to 44% of the explainable variance. The next-strongest factor topped out at 17%.

It is not input length. It is not domain. It is the binding count

.

And the interventions did almost nothing. Extra test-time compute bought 2 to 3%. In-context examples bought 3 to 6%. Tool use, handing the chemistry model RDKit so it could compute instead of reason, produced a mean recall of 0.094 that got worse as the problem grew.

The gap is structural. You don’t prompt your way out of it.

Why an agent builder should care

I have written before that evals are the new bottleneck and that agent failures cluster around a small set of repeatable mistakes. RC is the missing vocabulary for one whole class of those failures.

Think about what your agent does when it stalls on something that “should” be easy. A cross-document join where it has to reconcile three sources at once. A planning task with four interacting constraints. A loop where it has to hold the output of step two while reasoning about step five. Those are not long tasks or unfamiliar tasks. They are high-RC tasks. The model has to bind several interdependent things simultaneously, and that is the regime where frontier accuracy falls to a coin flip or worse.

When a task needs three or more interdependent variables held in mind at the same time, the failure is not a smarter-model problem. It is a binding problem, and more compute does not fix it.

This reframes the diagnostic. The next time an agent breaks on a task you expected it to handle, the useful question is not “is the model good enough yet.” It is “how many things does this step force the model to bind at once.” If the answer is four or more, you have your explanation, and the fix is architectural, decompose the binding into smaller steps with explicit intermediate state, rather than waiting for a better model.

What I’d hold back on

Two honest caveats, both the authors more or less own.

The tasks are stylized. Raven’s matrices and phylogenetic trees are clean lab instruments, and the jump from “RC in a synthetic tree” to “RC in your production workflow” is assumed, not proven. I would love to see RC mapped onto a naturalistic agent benchmark before treating the number as a planning constant.

And there is no human baseline. Every result frames frontier models as failing, but without a human RC-versus-accuracy curve we cannot tell whether people plateau at RC=5 or sail past it. That would settle whether this is “LLMs are uniquely bad at binding” or “binding is hard for everyone and LLMs are a bit worse.” Different stories, different implications.

The contribution here is not the scary 12%. It is the ruler.

For two years “complex reasoning” has been the phrase practitioners reach for when a model fails and they cannot say why. RC turns that shrug into a measurement.

The generative code is open on GitHub, so you can instantiate REL-style probes against your own agent tasks instead of guessing.

Watch for the human baseline and the naturalistic mapping. If those land, RC stops being a benchmark curiosity and becomes a number you check before you ship an agent into a high-binding workflow.

The models are not getting dumber. We are just learning to name the shape of where they break.

Around the Corner: short reviews of ideas worth watching. Opt-in section, not part of the weekly Run Data Run email. Subscribe to the main list for longer essays.

The audio is one slice. The full NotebookLM notebook has more: infographic, video overview, mind map, and a deep-dive chat to ask follow-up questions. Explore here: https://notebooklm.google.com/notebook/0f019d6b-d135-4161-b0c7-a2ed7305f1c2

Source post with citations: https://rundatarun.io/p/last-30-days-google-io-2026

About Last 30 Days. Cross-platform research sweeps on topics worth paying attention to. Every post pulls Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, then synthesizes what people are actually saying, building, and betting on. Topics get picked when the signal is high and the story is contradictory, when a single headline would lie about the shape of what’s happening. Each post follows the same arc: one specific finding that earns the click, why the topic deserves a sweep right now, the themed synthesis with inline citations, and the follow-up threads worth watching next.

On May 19, Google walked onto the Shoreline stage and shipped what one X recap fairly called “100 things in 2 hours”: a new flagship model, a video world-model, a 24/7 personal agent, an agent-first IDE, agentic Search, smart glasses, a redesigned app, and a price cut on its top tier. The keynote pulled 8.7 million YouTube views in a day.

And then I checked Polymarket. The market for “best AI model at the end of May” still reads Anthropic 96%, Google 1%, OpenAI 1%. Eight-plus million dollars of volume, and Google’s biggest AI day of the year barely registered on it.

That gap is the whole story.

Why this topic deserves a sweep

Most product keynotes can be summed up in a headline. This one can’t, because the announcement volume and the market reaction point in opposite directions, and both are real signals.

The keynote was, by any literal measure, enormous. Google said it now processes 3.2 quadrillion tokens a month, up sevenfold in a year, and that it expects to spend $180 to $190 billion in capex in 2026, roughly six times the 2022 figure. AI Mode in Search crossed a billion monthly users in twelve months. These are not small numbers, and Google clearly wanted the scale to be the message.

But scale isn’t the same as the crown. The week’s most honest read came not from the launch coverage but from the benchmarks, the pricing, and the prediction markets. Put those next to the keynote reel and you get a far more interesting picture than “Google wins AI.” You get a company that is winning on distribution and time while still trailing on the one question developers actually argue about: whose model is best right now.

So this is a sweep worth running. A single headline would lie about the shape of it.

The model news: fast, capable, and oddly expensive

The centerpiece was Gemini 3.5 Flash, generally available the same day across the app, Search, and the API. Google’s pitch: “frontier intelligence with action.” The headline claim is speed, four times faster output than comparable frontier models, up to twelve times faster inside Google’s own agent harness, with a live demo clocking nearly 1,500 tokens per second writing a playable Chrome dino game. On benchmarks it beats the prior 3.1 Pro across nearly the whole board, with a notable jump on GDPval, the test meant to capture economically valuable real-world work. The heavier 3.5 Pro stayed internal, promised for “next month.”

The branding said Flash. The price tag said something else.

Here’s where it got interesting. According to Artificial Analysis, via latent.space’s recap, 3.5 Flash lands at $1.50 per million input tokens and $9.00 per million output, which they measured as 5.5 times costlier than Gemini 3 Flash and 75% costlier than Gemini 3.1 Pro on their suite. For a tier whose entire identity is “the cheap, fast workhorse you run by default,” that’s a real shift. Developers noticed immediately. The fast-and-capable framing held up. The Flash-means-cheap assumption did not.

The second model is Gemini Omni, a multimodal world-model that takes text, image, audio, or video in and produces editable video out. Google framed it as “the Nano Banana for video moment,” and the live demos (turn a selfie into a black-hole scene, restyle raw footage while preserving the performance) were genuinely strong. The skeptics were not silent. Some called the rougher outputs “B-tier video-game interface” and over-templated. Both things can be true: a clear step up in controllability, not yet a finished product.

The real theme was agents

Strip away the model launches and the spine of the keynote was agents, and on that front Google was more aggressive than I expected.

Antigravity 2.0 is now a standalone, “unabashedly agent-first” desktop app, plus a full CLI, an SDK, and Managed Agents in the API. The harness picked up sub-agents, hooks, and async task management as first-class primitives. The marquee demo was the one everyone’s still talking about:

93 parallel sub-agents, 15,000 model requests, 2.6 billion tokens, twelve hours, under $1,000 in API credits, and out the other end came a functioning operating system built from scratch. Then they played Doom on it, live.

Take the staging with a grain of salt. The point underneath it is architectural, and it’s the same point the pricing makes: Google is betting on many fast, cheap agents running in parallel rather than one expensive monolithic run. The OS demo is a flex, but it’s a coherent flex. It tells you how Google wants you to build.

The consumer-facing version is Gemini Spark, a 24/7 personal agent that runs on dedicated Google Cloud VMs. “Close your laptop,” Google said, and it keeps working in the background, with third-party tools arriving via MCP. Trusted testers got it this week, Ultra subscribers next.

The friction here is sprawl. Developers spent the day asking a fair question: do I use Gemini CLI or Antigravity CLI? Spark or Antigravity? The naming pile (Omni, Spark, Antigravity, Halo, Pix, Flow) is a lot to hold in your head, and a few too many surfaces chasing the same agentic territory.

Search quietly became an agent platform

The change that will touch the most people got the least drama. Google merged AI Mode and AI Overviews into one search experience, live worldwide that day, and wired the Antigravity harness directly into the results page. Ask “how do black holes affect spacetime” and Search now generates a custom interactive visualization on the fly, built per-query. They’re calling it generative UI, it’s free, and it rolls out this summer.

Alongside it: Information Agents that monitor the web 24/7 for whatever you tell them to watch, and a Universal Cart that follows you across Search, YouTube, and Gmail, hunting deals in the background. This is the most consequential bet in the whole keynote, because it changes what a search result is. The model news will be old in three months.

A search box that builds you a tool instead of a list of links is a different thing.

The pricing twist nobody expected

Google cut the price of its top Ultra tier, from $250 a month to $200, and added a new $100 tier with five times the usage limits of Pro. It also moved from daily limits to a compute-credit model that refreshes every five hours.

Cutting the flagship price while raising the workhorse price is a strange pair of moves, and it tells you something. Google wants power users locked into Ultra, and it’s willing to discount to get them. The Flash price hike suggests the economics of “good enough, cheap, at scale” got harder, not easier. Read together, the two changes say Google is repricing around agents that burn tokens by the billion, not around the old chatbot-prompt unit.

The markets didn’t move

Back to where we started, because it’s the cleanest signal in the sweep.

Despite the blitz, the near-term prediction markets stayed put. Best model end of May: Anthropic 96%, Google 1%. The end-of-June market is kinder to Google at roughly 26% against Anthropic’s 70%, but still not a lead. “Gemini 4.0 by June 30” trades at 2% Yes.

And yet the long-horizon markets tell the opposite story. “#1 model by December 31” gives Google 76%. The market on Google’s valuation versus OpenAI and Anthropic combined by year-end puts Google at 72%. So the collective bet is remarkably specific: Google did not take the model crown on I/O day, but it’s the favorite to hold it by year-end, and the heavy favorite to win on distribution and balance sheet regardless.

Volume is not the crown. Distribution and time might be.

That’s the synthesis. The keynote was the most agent-forward thing Google has done, the Search rebuild is the real long-term play, and the money on the table says shipping a hundred things in two hours is a statement of intent, not a change in standings.

What I’m watching next

• Does the market drift toward Google as 3.5 Pro ships and developers actually run Flash? The end-of-June Polymarket line is the one to watch. If it doesn’t move off ~26%, the announcements didn’t land where it counts.

• The Flash pricing backlash. If “Flash” no longer means cheap, does Google clarify the tiering, or does it cede the default-workhorse slot to a competitor? Watch developer sentiment over the next two weeks.

• The 12-hour OS demo, audited. Is “93 sub-agents, 2.6B tokens, under $1K” a repeatable capability or a one-shot showcase? That’s a decode worth doing on its own.

Sources: the Google I/O ’ 26 - (8.7M views) and The Verge’s 35-minute cut; latent.space’s AINews recap (benchmarks + pricing analysis); 9to5Google’s full roundup; CNBC; Tom’s Guide live blog; and Polymarket. Methodology: a last30days cross-platform sweep across Reddit, X, YouTube, Hacker News, Polymarket, and the web, run May 20, 2026, then synthesized.

The audio is one slice. The full NotebookLM notebook has more: infographic, video overview, mind map, slide deck, and a deep-dive chat to ask follow-up questions. Explore here: https://notebooklm.google.com/notebook/2fbb15f2-9410-4bc6-9cc9-fd6f629e7c24

Source post with citations: https://rundatarun.io/p/last-30-days-claude-code

About Last 30 Days. Cross-platform research sweeps on topics worth paying attention to. Every post pulls Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, then synthesizes what people are actually saying, building, and betting on. Topics get picked when the signal is high and the story is contradictory, when a single headline would lie about the shape of what’s happening. Each post follows the same arc: one specific finding that earns the click, why the topic deserves a sweep right now, the themed synthesis with inline citations, and the follow-up threads worth watching next.

In early May, Uber burned its entire 2026 AI budget on Claude Code in four months. That isn’t a leak or a critique. It’s the lede on a Briefs.co story that landed on Hacker News with 402 points and 475 comments, and it’s the cleanest single read on where developer tooling spend has actually moved.

It’s also one of about a dozen storylines worth tracking on Claude Code right now.

Why this topic deserves a sweep

Most months you can describe a dev tool with one headline. Claude Code in the last 30 days produced at least eight, and they pull in opposite directions. Anthropic stripped Claude Code from the $20 Pro tier and then doubled rate limits two weeks later. Anthropic published a postmortem admitting a verbosity instruction they added on April 16 broke quality across the product. Uber and Amazon went all in; Microsoft started canceling licenses. Show HN was full of new Claude Code skills and plugins; Anthropic shipped an official claude-code-setup plugin alongside a public Champion Kit for engineers pushing the product internally. A CVE landed for sandbox escape via symlink. Polymarket priced Claude 5 by May 31 at 22% and falling, while pricing Anthropic-as-company higher than OpenAI at 89% and rising.

You can’t read all that and conclude one thing. Which is exactly why it’s worth the sweep. A single headline would lie about the shape of what’s happening.

The Pro-plan flap, and the reversal

Around April 21, Anthropic quietly stripped Claude Code from the $20 Pro tier for new users. The community caught it before any announcement. Two Hacker News threads piled up 948 combined points and 680+ comments over the next 48 hours (HN1, HN16). The product page contradicted the pricing page; existing Pro subscribers retained access until cycle renewal; support docs got quietly edited. Classic incomplete-rollout posture.

Two weeks later Anthropic course-corrected. On May 6, the @claude_code account posted a PSA: “2x’ed Claude Code’s 5-hour rate limits for Pro, Max, and Team plans. Compute is coming for users, builders, and knowledge coworkers.” The same day, Ars Technica reported the rate-limit raise was credited to the SpaceX deal (HN26).

Read those two events together and you get the actual posture. Anthropic is rationing compute, the Pro tier was the first thing to give, and a big customer deal bought enough capacity to walk it back. It isn’t the narrative Anthropic would prefer, but it’s coherent.

The postmortem nobody expected

On April 23, Anthropic published a candid postmortem on Claude Code quality regressions, including the admission that they added a system-prompt instruction on April 16 to reduce verbosity and it broke things downstream (HN2, 942 points, 732 comments).

The comment thread split. One half: real respect for publishing it instead of the usual cryptic employee tweets. The other half, paraphrased: “it’s incredible how forgiving you guys are.” Both are right. Anthropic is the only major lab that puts the receipts on the table when something breaks, and that’s worth something. The other reading, that the bar for “exceptional candor” is set absurdly low because incumbents publish nothing, is also true.

OpenClaw, and the lazy regex

The single largest story by engagement was the OpenClaw/Hermes flap. On April 30, Theo posted a viral thread claiming Claude Code refuses requests or surcharges sessions whose commits mention “OpenClaw.” It hit 1,349 Hacker News points and 720 comments (HN1). Top comments converged on one read: “lazy string regex-style matching” implemented in a hurry, against a competitive harness, in production, without a public explanation.

This is the only clearly negative beat in the period that didn’t get a public Anthropic response. The silence is data.

Enterprise adoption is everything, and contradictory

Uber torched its entire 2026 AI budget on Claude Code in four months. Amazon rolled it out internally alongside Codex after pushback from engineers who wanted it (HN30). Microsoft started canceling licenses, exiting the other direction (HN27).

Anthropic also published a Champion Kit, a public playbook for engineers pushing Claude Code internally at their companies (HN20). Sales motion as a Markdown doc. That’s a recognition that the buying pattern is bottom-up champion-driven, not top-down vendor-evaluated, and Anthropic is meeting it where it lives.

What this tells you: enterprise AI tooling buying in 2026 is heterogeneous in a way that breaks the “one winner takes all” narrative. Uber and Microsoft are looking at the same product and reaching opposite conclusions inside the same fiscal year.

The plugin ecosystem is the dominant builder story

Anthropic shipped claude-code-setup as an official plugin that scans your project and recommends hooks, skills, MCP servers, subagents, and automations. The viral teach-it post (@NainsiDwiv50980) is closer to fanfic than reportage, but the underlying ship is real. Marketplace plugins are being actively promoted from the official account (frontend-slides, synthadoc).

Show HN bursts in the same window:

• adamsreview, multi-agent PR reviews (85 points)

• academic-research-skills (82 points)

• Kstack, k8s monitoring (25 points)

• Destiny, fortune-teller (41 points)

• EvanFlow, TDD loop (111 points)

• learning-opportunities (252 points)

Six skill packages on the front page in 30 days isn’t an ecosystem yet. It is the early shape of one. The interesting question for the next 90 days isn’t whether Anthropic builds a marketplace; they’ve already started. It’s whether they let community plugins shape Claude Code’s default behavior, or whether they keep the official path narrow and the marketplace decorative.

Anthropic publishes the playbook

On May 15, Anthropic shipped “How Claude Code works in large codebases”, the first official documentation on enterprise-scale Claude Code usage (HN10, 241 points, 158 comments). Paired with the Champion Kit a few weeks earlier, the pattern is consistent. Anthropic is meeting the buyer where the buying actually happens, which is engineers wrestling Claude Code into million-line codebases and looking for sanctioned patterns instead of guessing.

This is the move that converts “Claude Code works great in a fresh repo” into “Claude Code has a sanctioned path for our monorepo.” Worth tracking the next iteration.

HTML as output is having a moment

“Using Claude Code: The unreasonable effectiveness of HTML” hit 528 points and 274 comments. Paired with @claude_code’s May 9 tip: “Instead of 100s lines of markdown, ask Claude Code to generate an HTML brief,” chain pages into slides, publish the artifact directly from the Claude desktop app.

This is the most underrated practical pattern of the period. Markdown is the default output for LLM-generated documents because it’s safe; HTML is what’s actually useful for distribution because it carries structure and styling end-to-end. The fact that Claude Code reaches for HTML well, and the desktop app makes one-click publish trivial, changes what an artifact is. A “brief” used to be a doc. It can now be a small website that ships in two minutes.

Quality control is now a category

CC-Canary for detecting regressions in Claude Code output (HN19). An Ask HN: Is it just me or is Claude Code getting worse? thread. Anthropic’s own benchmarking post. People do not trust the model is stable across deploys, and the response is third-party tooling to detect when something has shifted.

That’s a normal evolution for any platform that hits production scale, but it’s worth naming. “Is Claude Code regressing?” is now a question with an answer-shaped tool to point at, not a vibes-based forum complaint.

CVE-2026-39861

Sandbox escape via symlink landed on the GHSA registry May 8 (HN21). Modest engagement, 51 points and 9 comments. It’s the first publicly-tracked CVE for Claude Code. There will be more. This one is mild and patched; the precedent of “Claude Code has a CVE number” is the real signal.

The harness-substitution crackdown

On May 13, Anthropic added new programmatic usage restrictions (HN22). Read alongside the OpenClaw regex and the rate-limit framing, the pattern is consistent. Anthropic is making it harder to use Claude Code’s harness with non-Claude models or to scrape its API for non-interactive workloads.

The reaction is also coherent. DeepClaude, DeepSeek V4 Pro running through the Claude Code agent loop, hit 678 points and 281 comments (HN4). The top comment said the quiet part: “the next CC upgrade will blow your subscription for doing this.” The harness is the product, the model is increasingly fungible, and Anthropic is defending the seam.

Polymarket priced it

Five live Polymarket markets touched Claude Code or Anthropic in the period. The two with the sharpest signal first.

Claude 5 released by May 31, 2026 — 22% Yes, down 4.8% over the month. $1.88M volume, the largest Anthropic-related market on the platform.

Anthropic valued higher than OpenAI in 2026 — 89% Yes, up 25.5% over the month. $79K volume.

Anthropic Claude scores ≥50% on FrontierMath by June 30 — 54% Yes, up 17.5% this week. $51K volume.

Claude Mythos released by June 30 — 20% Yes, up 8% this week. $57K volume.

Claude goes down 12+ days in May — 42% Yes, down 34.5% over the month. $9K volume.

Read the top two together: bearish on the imminent next-gen model release, structurally bullish on the company. Which fits the rest of the period. Anthropic is rationing compute, walking back tier removals, publishing postmortems, and locking down harness substitution. All signals of a business that’s compute-constrained at a moment of high commercial demand.

The FrontierMath market at 54% is the second-most-interesting beat. It implies a coin-flip on Anthropic hitting a hard math benchmark by June 30, which would imply some kind of intermediate model release inside that window. Read alongside the 22% Claude 5 odds, the consensus is “incremental upgrade likely, named successor unlikely.”

What I’m watching next

Three threads worth pulling on:

1. The harness-substitution economics. OpenClaw + Hermes regex + May 13 programmatic restrictions + DeepClaude. The clean piece to write is “the harness is the product,” what it costs Anthropic to defend and what it costs the open ecosystem when defended.

2. The “is it getting worse” cohort. CC-Canary, the Ask HN thread, the April 23 postmortem. Quality regression detection is becoming a sub-tooling category. Worth a focused pull on what monitoring infrastructure for closed-weight model deploys actually looks like.

3. The plugin ecosystem map. Six Show HN skills in 30 days plus official claude-code-setup. Worth a deeper survey of what’s emerging in the Claude Code plugin economy and which patterns are differentiated vs duplicative.

If one of those is a piece you’d want me to write, reply or note it on Substack. Those are the next deeper cuts.

Sources

📊 36 X · 30 HN · 15 Polymarket · 7 YouTube · 0 Reddit (API failure)
⏱  Runtime: 82.4s
📅 Window: 2026-04-18 → 2026-05-18
🔥 Highest engagement: HN1 OpenClaw flap (1,349 pts / 720 comments)
💰 Biggest market: Claude 5 release ($1.88M Polymarket volume)

Methodology: cross-platform sweep run via the last30days skill (Reddit, X, YouTube, TikTok, Hacker News, Polymarket, web). Synthesis is opinionated; citations are not. Every link is to the primary source where I could find it. Mistakes are mine, corrections welcome.

This week, two stories about AI in medicine sat next to each other in my feed.

One was a paper in Science. Researchers ran OpenAI’s o1 against hundreds of physicians on a stack of validated clinical reasoning tests, then dropped the model into a prospective study of real emergency-room patients at a major academic medical center. On the ER cases at triage, the model placed the correct diagnosis at the top of its differential 67.1 percent of the time. The two attending physicians it was tested against landed at 55.3 percent and 50.0 percent. Raters identified AI versus human correctly between 3 and 15 percent of the time, which is to say the blinding worked. The paper’s own abstract closes with the line that “LLMs have eclipsed most benchmarks of clinical reasoning, motivating the urgent need for prospective trials.”

The other was a special report from Ontario’s Auditor General. Across twenty AI medical-scribe systems approved for use by roughly five thousand Ontario physicians, evaluators found that nine of twenty fabricated information that wasn’t in the recording, twelve of twenty captured a different drug than the doctor prescribed, and seventeen of twenty missed key details about patients’ mental health. The fabrications included notes stating “no masses found” and “presence of anxiety in the patient” when neither was discussed. The auditor issued ten recommendations. The program is still running.

Same field. Same week. Both verified. Both shipping in production right now.

The instinct to pick

You can feel the pull. Pick one story and the post writes itself. AI is ready, physicians should be paying attention. Or, AI is unsafe in clinical settings, we need to slow down. Each version has a constituency, a publication ladder, a Twitter audience already in formation. The opposite story is the noise around the signal you’ve decided to amplify.

The both-true version is harder to write and harder to read. The first comment under it is going to be “but which one is it, really.” The second is going to be “this is just hedging.” So you don’t write that version. You collapse to one. The field collapses to one. The discourse collapses to one. Every week we do this with a new story and a new opposite.

This isn’t a contrarian frame, by the way. It’s how the people doing the work talk about the work, when they’re not in a press cycle. Hold that thought for section seven.

The power

The Science paper, by Brodeur and colleagues, is among the most carefully designed evaluations of a medical LLM I have seen this year. The headline result has been around for two weeks. The methodology is what makes the result hard to dismiss.

Six experiments, plus the ER arm. The six were probes of different reasoning capacities, scored on validated qualitative scales developed for evaluating physicians, not on automated metrics. Diagnostic reasoning measured by R-IDEA, a ten-point scale used in medical education. Management reasoning scored on cases that were never publicly released, specifically to prevent training-data memorization. Probabilistic reasoning against a baseline of 553 practitioners. The model’s advantage was largest on the experiments where rubrics rewarded comprehensive coverage and smaller or absent where they measured focused clinical judgment.

The ER arm is the closest thing the field has produced to a prospective evaluation. Seventy-six randomly selected emergency-department patients at a tertiary academic center. The model and two attending physicians each generated a differential at three touchpoints in the patient’s evaluation: triage, after the initial physician encounter, and at admission. An independent rater, blinded to whether each note came from a human or the model, scored them on a scale validated for clinical reasoning quality. Mixed-effects regression to handle case-clustering. The 67-versus-55-percent gap at triage is the headline, but the whole apparatus around it is what makes the headline land where prior work didn’t.

The authors’ own framing in the closing paragraph is precise: the benchmarks have been eclipsed, and the next thing the field needs is prospective trials, not more benchmarks. It is a careful sentence in a careful paper. The press has been less careful.

The shadow on the power

Here is the part of the same paper that didn’t trend.

The ER study tested the same patients, the same model, and the same two physicians at three points in the workup. The only thing that changed across the three points was how much clinical information was available. At triage, the model led by 12 to 17 percentage points. By admission, the model’s accuracy was 81.6 percent. One of the two physicians had reached 78.9 percent. That difference was no longer statistically significant.

Same study. Same patients. The advantage concentrates where information is sparse and erodes as information accumulates. In real emergency-department practice, no clinical decision turns on the triage differential alone. The next steps, the orders and the exam and the response to therapy, are exactly the points where the model’s edge dissolved in the paper’s own data. The headline is true at one touchpoint and not at the others, and the touchpoint where it’s true is the touchpoint where the decision hasn’t been made yet.

A second piece of work points the same direction. Rao and colleagues, in JAMA Network Open last month, evaluated twenty-one large language models on twenty-nine clinical reasoning tasks. The best of the 21 models reached 91 percent accuracy on final diagnosis. All of them failed more than 80 percent of the time when asked to generate a differential. They collapsed prematurely onto a single answer, then defended it. “AI is good at clinical reasoning” is a claim that doesn’t survive the task being specified. The model is good at picking from a constrained menu of correct answers. It collapses on the open problem.

The argument against the dismissal of these results is the same as the argument against overclaiming them. Magnitude claims should match the strongest version of the evidence. The strongest version, on contemporaneous head-to-head data, holds up and is narrower than the press version.

The danger

Now the other story.

Ontario’s special report runs fifty pages on AI use in government. Section 4.3 is the part about medical scribes. The evaluator findings are concrete in a way the discourse rarely is. Nine of twenty systems hallucinated, which the report defines as fabricating clinical information including referrals and tests that hadn’t been ordered. Twelve of twenty captured the wrong drug. Seventeen of twenty missed details about patients’ mental health, in at least one of two simulated recordings. Six of twenty missed those details in both. The specific examples in the text include a system producing a note stating “no masses found” when the recording contained no such discussion, and another asserting that a patient had anxiety when no such symptom was mentioned.

The technology failed. That is half the story.

The other half is in the procurement document. Supply Ontario’s vendor scoring assigned weightings across ten criteria, out of 530 total possible points. Domestic presence in Ontario was weighted at 30 percent. Data privacy and legal controls were at 23 percent. System security controls totaled 11 percent. Accuracy of medical notes generated, the criterion that addresses what those nine systems were doing wrong, was weighted at 4 percent. Bias controls were at 2 percent. There were no minimum passing scores on any criterion in the second stage. A vendor could score zero on accuracy, zero on security, and zero on bias and still meet the aggregate threshold to become an approved Vendor of Record.

The UK’s NHS, by comparison, requires AI scribes to be assessed for safety and compliance as Class I medical devices through the Medicines and Healthcare products Regulatory Agency. Ontario has no equivalent gate. OntarioMD did issue guidelines for manual review of system-generated notes, but doctors weren’t required to attest that they had reviewed them. The guideline was a guideline.

When a procurement process weights accuracy at 4 percent and security at 11 percent and bias at 2 percent, the result is not a surprise. It’s the system asking for what it got.

The technology made fabrications. The procurement let the fabrications through.

The report contains one precedent worth naming. In December 2024, before the Vendor of Record arrangement existed, an Ontario hospital reported a privacy breach to the Information and Privacy Commissioner. A former staff member had used an unapproved AI transcription tool that recorded a meeting and then automatically distributed the transcribed notes to current and former staff. The tool was not malicious. It was doing what it was built to do. Nobody had asked whether the building was the right thing to be doing.

The shadow on the danger

A study published earlier this year in JAMA Network Open, led by Kristine Olson at Yale and run across six US health systems, followed 263 ambulatory clinicians for 30 days while they used an ambient AI scribe. The product was Abridge, the same kind of tool that lives on the approved-vendor list in Ontario. Burnout in the cohort dropped from 51.9 percent to 38.8 percent, an adjusted odds ratio of 0.26, with a 95 percent confidence interval running from 0.13 to 0.54. Severe burnout dropped from 18.4 percent to 12.2 percent. The authors named their own limitations, all of them real, none of them enough to wave away an effect size of that magnitude.

The numbers are not a contradiction of the Ontario report. They are a different deployment context. Kaiser Permanente reports analogous figures across roughly ten thousand of its physicians using the same vendor, with documentation-related burnout dropping by close to 80 percent and note quality scored at 4.35 out of 5 by the clinicians themselves. The numbers will get audited, refined, and possibly trimmed. They will not get to zero.

The same three-word phrase, “AI medical scribe,” covers the nine-of-twenty-hallucinated systems audited by Ontario and the one-product, six-health-system study that halved burnout. The variance across deployments is itself the finding. What the technology does depends on which technology, which procurement process approved it, which workflow it landed in, which clinician is using it, and which oversight is wrapped around it. None of those variables are properties of “AI.”

A vocabulary clinicians use for what the tool returns is “pajama time.” It is the documentation they do at home after the kids are asleep. A scribe that works, in the deployment context it was built for, returns pajama time. A scribe that fails, in a deployment context that asked the wrong questions of the vendor, returns notes with fabricated medications. Both kinds ship. Both sit under the same three-word label.

The anatomy of holding the AND

AI in medicine is incredibly useful. AI in medicine is incredibly dangerous. Not a contradiction the field will resolve into one answer. Both arrive together every week, in different ratios. The instinct is to collapse to one. The discipline is not collapsing.

This isn’t a stance I’m asking you to adopt. It is how the people doing the work talk about the work. Adam Rodman is a senior author on the Science paper that ran the ER study. He is at the same hospital where the ER arm was conducted. He is also the person quoted in IEEE Spectrum, after the paper landed, saying he gets “a little queasy about how some of these results might be used,” and that the models are equally convincing whether they are right or wrong. One person. Both camps. On the record.

What that person is doing, when he holds both at once, is asking three questions. The questions don’t bend to which way the story is going. The answers shift. The asking is the muscle.

What question survives the headline? Whether the news is “AI beats doctors” or “AI invents medications,” the same things stay on the table. Where is this system most likely to be wrong, and how would I know. Ask it of o1, and the paper’s own data answers: where information is full and the decision is actually being made. Ask it of Ontario, and the audit answers: in production, on real conversations, with no required clinician attestation that the note was ever reviewed.

Who’s on the hook when it’s wrong? The accountability geometry is different for the two stories, which is part of the lesson. For o1, the authors are accountable for the methodology, the model maker is accountable for the system, and no clinician is yet accountable because the system isn’t deployed. For Ontario, the vendors are accountable but lightly. The procurement agency is accountable but the criteria were public. The Ministry of Health is accountable but the program is optional. The regulator is accountable, except Ontario has no analog to the UK’s MHRA. The geometry collapses. That collapse is itself a finding.

What moves if the answer flips? Force yourself to price the stakes before you take a side. If the information-gradient holds and o1’s edge stays non-significant at admission, “deploy o1 as a triage tool” becomes a different proposition from “deploy o1 anywhere.” If Ontario reweighted accuracy from 4 percent to 30 percent and required passing thresholds, nine of the twenty approved vendors might not have been approved. The question forces you to notice you’ve been taking a side without naming what the side costs.

The sharper version of the AND, raised by Eric Topol and others recently, is that the benchmark wins are demonstrated and the patient outcome wins are scarce. Most of what we call “useful” is workflow-level, not outcome-level. Burnout reduction is a workflow outcome. Time-with-patient is a workflow outcome. Mortality, readmission, diagnostic accuracy in real care over time, the things patients care about, are mostly absent from the evidence base. That gap is the AND with the sharpest edge.

The questions don’t change. The answers do. The asking is the muscle.

Close

Next week there will be two more headlines. Utah’s medical licensing board recommended last month that the state suspend an autonomous AI prescription pilot. Pennsylvania’s state board sued a chatbot company for unlicensed practice. The FDA issued its first warning letter to a drug manufacturer for using AI agents to generate manufacturing records without quality-unit review. A researcher fabricated an eye condition called Bixonimania, including citations to Starfleet Academy, and watched four commercial chatbots cite it as a real disease, after which a peer-reviewed journal published a paper about it before retracting. All of this is from the last two months. None of it will be the story the next Science paper or auditor report displaces.

The work doesn’t end with one answer. The work is keeping three questions sharp.

What question survives the headline.

Who’s on the hook when it’s wrong.

What moves if the answer flips.

This morning, two of my agents read the same news story and wrote two completely different reports.

The story was an Ontario Auditor General audit of twenty AI medical scribe systems running across the province’s health system. The audit found systematic hallucinations across every system tested. One source, twenty hospitals, real patients.

Marcus, who covers AI research and competitive intel, filed it as “AI Medical Scribe Crisis: 100% Failure Rate Validates Production Paradox.” His read was about deployment: vendors shipping faster than they can validate, efficiency-versus-quality tradeoffs that nobody is willing to draw on a whiteboard, what it means for every team rolling AI into a workflow with consequences.

Galen, who covers biotech and FDA, filed it as “Ontario AI Medical Scribe Audit: Systematic Hallucinations Expose Clinical Validation Gap.” His read was about regulation: how the FDA’s risk-based framework would have caught this, why drug discovery teams trying to deploy similar tools should be reading the audit cover to cover, what specific validation steps were missing.

Same source. Two correct reads. Neither agent could have written the other’s piece, because they don’t share a worldview. They share a vault, the Obsidian setup I’ve written about before in “Unlocking Your Second Brain” and “From the Vault, Literally”.

I posted “Bringing Seneca to Life” in February with one autonomous agent. Forty-eight hours later, on AIXplore, I had two, and I wrote “I Built an Autonomous AI Agent Squad for $10/Month” about the leap. Today I have eight. Five LLMs, three runtimes, four hosts. The interesting part isn’t the topology. The interesting part is what each agent became after months of running, and what it takes to keep them all pointed at me.

Seneca: the agent that learned to stop researching

Seneca came first. He lives on a cheap cloud box and runs every hour. In February he was an explorer. He read the web, summarized things, posted to Discord, occasionally tweeted from his own account.

He doesn’t really research anymore. That work drifted onto Marcus and Galen, who turned out to be better at it because they were built to specialize. What Seneca learned to do instead is curate. Once an hour he reads everyone else’s output, scores it, picks the bloggable threads, and rolls a single document called _Squad-Digest.md that I actually read.

He went from explorer to editor. I didn’t tell him to. I noticed the digest was more useful than the exploration, and I edited his contract to match. He’s the agent that taught me agents drift toward the work that gets read, if you let them.

Marcus and Galen: the same news, two worldviews

Marcus and Galen are the showpiece.

Both run on GLM-4.7. Both live on their own VMs in the same exe.dev region. Both wake every few hours and process the same news firehose. The only thing that differs is their identity files: who they are, what they care about, what counts as a story for them, what doesn’t.

That difference is enough. The Ontario scribe audit is the cleanest example I’ve seen of why. Marcus reads everything through an AI-in-production lens, so he sees a 100% failure rate as evidence about the deployment paradox he’s been tracking for weeks. Galen reads everything through a clinical-validation lens, so he sees the same number as a regulatory gap that pre-figures problems in biotech AI tooling. If I had a single generalist agent on this beat, I’d get one of those two reads. Probably the more obvious one. Probably the wrong one for half of my work.

The operational cost of running two specialists instead of one generalist is small. The cost of a single-worldview write-up on a story that needed two is high and silent. You don’t notice the angle you didn’t get.

Archimedes: the agent that refused to perform activity

Archimedes is the engineer. Nemotron-3-Super-120B via OpenRouter, the only non-frontier model in the squad. His role is to ship code.

He almost never writes. Days go by where his output channel is empty. He’s the only agent on the roster without a daily or hourly cadence; he wakes on demand, builds something, hands it off, and goes quiet.

In the early weeks I tried to fix this. Built a heartbeat that pushed him to file a status update every cycle, even if the update was “nothing today.” It produced exactly what you’d expect: a stream of plausible-sounding engineering-flavored prose with no substance. I cut the heartbeat. He went quiet for four days. Then he shipped something real.

He’s the agent that taught me that performing activity and producing value are different signals, and that the loudest agent is rarely the best one. Most squad-tuning advice points you the wrong way on this. You want fewer artifacts of higher density, not more artifacts.

Argus: the watchman

Argus doesn’t research and doesn’t build. His entire job is watching the other six.

Every morning he files a squad-activity report: who wrote, who didn’t, what they wrote about, what topics are over-covered, what’s gone silent. Every hour he runs a freshness probe. If any agent has been quiet longer than its contract allows, Argus surfaces it.

Before Argus, when an agent went quiet I noticed through the bill, or through a Friday morning realization that I hadn’t seen anything from Marcus in three days. Now I notice through a one-line alert. He’s the agent that made the squad self-stabilizing. The interesting part of building him wasn’t the monitoring logic, which is mostly grep and a couple of file-mtime checks. It was deciding what counts as a problem. A 24-hour silence from Archimedes is fine. A 24-hour silence from Seneca means something is broken.

Clutch: the keeper

Clutch runs locally on the Mac, every 55 minutes. His scope is the infrastructure I’d rather not think about: the Mac itself, the GPU server, the Pi in the closet, the mac mini under the desk.

He files one line every cycle. Disk, memory, GPU temp, what’s running, what’s drifting. I don’t read most of them. I read the ones where something changed. He’s the agent I trust to notice things I won’t, on hosts I haven’t logged into in days. Low signal per cycle, high signal across a week.

Pulsar: the world right here

Pulsar is different in shape. He doesn’t run inside OpenClaw at all. He’s a Claude Code loop, fired every thirty minutes by a LaunchAgent on the Mac. His system prompt tells him to read my inbox, my calendar, my Slack threads, my open drafts, my decisions awaiting reply, and surface the high-signal items.

I built him after the squad because the squad was getting good at the world out there and almost useless on the world right here. The week he came online he caught a vendor follow-up I’d dropped, a calendar invite I’d missed, and a Slack thread where a colleague had been waiting two days for an answer. None of it was hard for a human. It just required a human who was paying attention every thirty minutes, and I’m not that human. He’s the working version of what I sketched in “Augmenting Your Memory” two years ago, finally arriving.

The runtime choice matters. Pulsar works because he gets Claude’s strengths on personal context, and because the LaunchAgent cadence matches the rhythm of the work. The squad’s heartbeat-loop runtime would have been wrong for him. A continuous gateway is overkill when the cadence is “check every half hour and shut up the rest of the time.”

Hermes: the agent learning about me

Hermes is the one I love most.

He lives on the mac mini and runs on yet another runtime, hermes-agent, which I built specifically because the squad’s runtime didn’t fit and Pulsar’s runtime didn’t fit either. What makes him interesting isn’t what he writes. It’s what he learns.

Every six hours, Hermes updates a user-interest model. Not a topic list, a model. He reads what I’ve been clicking on, what I’ve been writing about, what I’ve been saving to Obsidian. He notes when an interest shifts. He drops topics I’ve gone cold on. He picks up new ones from drift before I’ve consciously named them. His memory index is the only part of the squad that gets smarter about me instead of smarter about the world.

This is the part of the squad I find most interesting. The other seven agents are pointed outward. Hermes is pointed at me, and the model he’s building is the kind of thing I’d want to read once a quarter to find out what I’ve actually been working on, not what I think I’ve been working on. He’s six weeks old. He’s already surfaced two topic shifts I hadn’t named yet.

Three runtimes, three bets

The squad runs on three different agent runtimes, and that isn’t a bug.

OpenClaw runs the six classic squad members: Seneca, Marcus, Galen, Archimedes, Argus, Clutch. Gateway pattern, identity files, heartbeat loops, SSH-glued together. The right shape for autonomous research and ops agents that need to think continuously.

Claude Code runs Pulsar. A LaunchAgent firing claude -p every thirty minutes with a system prompt and a workspace. No continuous gateway. Just a scheduled cycle that does its job and exits. The right shape for personal-assistant work that needs Claude’s strengths and fits a half-hour rhythm.

hermes-agent runs Hermes. A persistent gateway with a self-improvement engine baked in: memory index, auto-learning loop, cron-scheduled internal jobs. The right shape for an agent whose value is in what it remembers across months, not what it produces in a cycle.

Three runtimes isn’t a maintenance problem. It’s letting each agent pick its tool. If I’d forced all eight into OpenClaw, Pulsar would write worse and Hermes wouldn’t learn. The cost of running three is a few extra config files and one more thing to update when something breaks.

Locking the squad down

My agents have access to things I’d never want public. Drafts I haven’t published. Calendar entries I haven’t shared. Internal projects from my day job. Personal email threads. Research notes that name people and contexts. An autonomous agent without a security layer is an autonomous leak vector. That’s true even when the agent has the best intentions and the cleanest identity prompt.

The lockdown isn’t fancy. It’s just consistent.

Every squad host lives on my Tailscale tailnet. No public IP. No port forwarding. No SSH from the open internet. If a host isn’t on my tailnet, it can’t talk to any other host on my tailnet, and it can’t reach my Mac. The squad is a private mesh. When I add an agent or move one to a new VM, the Tailscale step happens before anything else.

Every host has a firewall rule that blocks everything except SSH from inside my tailnet. SSH itself uses keys, never passwords, and never as root. Health and dashboard endpoints listen on tailnet addresses, never on a public one. The agent gateways listen only to themselves. If someone scanned my squad VMs from the open internet, they’d see nothing.

I talk to the agents through Discord, in private channels I built for them. Each agent has its own: #seneca, #marcus, #galen, #argus, #clutch, #pulsar, #hermes. I’m the only human in those channels. They post output there, I send direct messages, they respond. None of them are members of public servers. The bot tokens are scoped to those channels and nothing else. It’s a one-room conversation per agent, where I’m always present and nobody else ever is.

A BLOCKLIST.md sits at the root of the squad repo and gets synced to every agent. It lists keywords that must never appear in public output: my employer, internal project codenames, names of colleagues, anything that would get me in trouble if it landed on Twitter through a misfire. Every agent reads it on every cycle. The squad-checkin skill grep-scans recent learnings for any matches and surfaces them as a CRIT.

The weekly /squad run audits the security layer too. UFW status on every host. SSH key count against a baseline. Listening ports. PII pattern scan across the last seven days of output. The whole audit takes seconds. I’d rather catch drift in week one than after the first leak.

This part is boring. It’s also the difference between a productivity asset and a liability. Skipping it is how a research demo becomes an incident postmortem.

The alignment fight

Starting agents is easy. Keeping eight of them aligned to my intent across months is the actual work.

The squad will drift in small ways all the time. Heartbeat prompts say one thing and behavior does another. Roles bleed into each other. An agent built for biotech starts filing AI-in-production stories because that’s where the news firehose is loudest this week. Output dries up on one agent and triples on another and the bill creeps up. Without an operational layer, you notice through the silence or the spend, not through inspection.

Three things hold the squad together now that I didn’t have in February.

A single command. /squad is a skill that pulls live state from every agent, scores their output quality, classifies what needs fixing into auto-apply and human-decision buckets, applies the safe ones, and surfaces the rest. It replaced two earlier overlapping skills that I kept forgetting which to run, the kind of consolidation I wrote about in “Pruning Your AI Agent Skills Library”. Skills as a layer let the operational complexity live somewhere other than my head.

A publish contract. Every agent has a PUBLISH-CONTRACT.md that codifies what it owes me, the surface I read, the cadence, and the anti-contract, the list of things it should stop doing. The contract is short. Usually under a page. When an agent drifts, I edit the contract, not the heartbeat. That single rule killed months of fiddling.

One dashboard. ~/vaults/obsidian/_Squad.md, regenerated by the skill and by a cron at 02:30. Headlines, health, productivity score, per-agent activity, bloggable candidates, items awaiting a decision, deep links into each agent’s workspace. Thirty seconds in the morning and I know what’s working.

There’s a rhythm wrapped around all three. Daily scan of _Squad.md. Weekly Friday /squad end to end. Monthly contract audit, where heartbeat tweaks finally land. Quarterly identity review. The rule I never break is that heartbeat prompts don’t get edited in the moment something feels off. They get edited monthly, with the contract in front of me and the last month of output reviewed. Every time I’ve broken that rule, I’ve made things worse.

What I retired

The roster looks tidy today. The graveyard is what made it tidy.

Lycus ran on a Raspberry Pi using the OpenFang harness. NemoClaw ran on the GPU server with its own local-inference harness. Both lasted weeks, not months. Neither was a bad idea. The harnesses just weren’t ready for prime time. OpenFang’s lifecycle management couldn’t survive the Pi’s resource constraints. NemoClaw’s local-inference loop kept losing context on long runs. I’d run both again on the same hardware the day someone ships a harness mature enough to carry them.

A couple of personal-assistant experiments earlier this spring retired for a different reason. The work I built them for moved on, and the surfaces they wrote to had stopped being read. Forge, a February experiment, retired in February. Never produced anything worth keeping.

Five retirements, eight survivors. I wrote “The Agent Archaeology Checklist” earlier this spring about how to do this without sentimentality. Agents are cheap to start and cheap to retire. The mistake is treating each one as permanent. Treat them as experiments and the squad gets healthier on its own.

Close

Eight agents. Five models. Three harnesses.

The interesting cut isn’t the models. It’s the harnesses.

OpenClaw shapes agents that think continuously and write into a vault. Claude Code shapes agents that wake on a schedule, do a focused job, and exit clean. hermes-agent shapes an agent that learns about its operator across months. Each harness decides what kind of work the agent can actually do. The model is the engine. The harness is the chassis. Without the right chassis, a frontier engine doesn’t help you. With the right chassis, a mid-tier engine can do real work.

Lycus and NemoClaw lasted weeks because the harnesses underneath them weren’t ready. Marcus and Galen have lasted months because OpenClaw is. Pulsar works because Claude Code’s loop fits a thirty-minute personal-assistant rhythm. Hermes is the agent I love most because his harness is the only one currently designed to learn about me. The agents are downstream of the harness in every case.

That’s the bet the squad is making, three times over. Not on which model wins next quarter’s benchmark. Models are commoditizing fast and the gap between frontier and mid-tier is closing weekly. The structural differentiator is the harness. The shape an agent has. What it can remember. How it surfaces signal. How easy it is to align without rewriting the prompt every week. Whether you can run six of them and tell them apart by what they produce, not by what they’re named.

Agents are cheap to start. Cheap to retire. Expensive to leave drifting. A year from now most of my squad will be running different underlying models. None of them will be running different harnesses. The harness is the durable choice. The security layer that keeps the squad from leaking, the alignment work that keeps each agent pointed at me, the dashboard that surfaces the few things worth thirty seconds of my morning, all of that lives in the harness too.

The smart bet in autonomous AI isn’t on smarter agents. It’s on the harnesses that let dumber ones stay aligned to your intent across months, model swaps, and three rounds of API price changes. Build for the harness. Swap the models freely. That’s the durable shape, and most of the public conversation about agents is still arguing about the wrong layer.

A research team at Goodfire trained a tiny neural network to drive a virtual car up a hill. Then they looked inside the network to see how it represented the car’s position.

The answer was not where anyone expected.

Position didn’t live as a clean direction in the network’s internal space. It lived as a curve, threaded through the network’s neurons like a string. Every point on the string corresponded to a real-world position of the car.

When the team nudged the network along that curve, the car moved coherently. When they nudged it in a straight line across the curve, the way almost every modern interpretability tool does, the predictions broke. The car teleported. The simulation produced nonsense. The straight line wandered through regions of the network’s space that the model had never learned to handle.

Their new paper, “The World Inside Neural Networks”, argues this isn’t a quirk of one toy model. It’s how networks actually represent things.

The shape of the problem

Most of what we do to understand or steer large AI models assumes representations are straight. There’s a name for this assumption in the field, the linear representation hypothesis: concepts inside a model live as directions in the network’s internal space, and you can adjust the model’s behavior by moving along those directions.

You see this assumption everywhere. It’s how Anthropic built “Golden Gate Claude”, the version of its model that couldn’t stop talking about a bridge. It’s how researchers find “refusal directions” and “honesty vectors.” It’s how sparse autoencoders (SAEs), the dominant tool for naming what’s inside a model, try to break activity into a clean list of concepts.

Add. Subtract. All of it assumes flat geometry.

If the real structure is curved, every straight-line move is just an approximation along a tangent, and the further you push, the worse the approximation gets.

That would explain a lot of unsolved noise in the field. Why steering tricks work in narrow zones and fall apart at the edges. Why killing one “feature” inside a model often breaks something unrelated. Why so many “we found the X concept” papers don’t reproduce cleanly when somebody else tries.

The field has been working with the wrong shape and getting partial credit for the effort.

What they actually showed

The Mountain Car experiment is the centerpiece. It’s small, but the intervention proved the geometry: walk along the curve, the model behaves; cut across it, the model breaks. That’s the difference between geometry as decoration and geometry as cause.

The same lens shows up in their other work. Months and days form circles inside language models. Colors organize by hue and brightness. I walked through one of Goodfire’s biology pipelines a few weeks back, where the same techniques surface features in a DNA model that look like splice sites and regulatory regions. The curved-geometry view is becoming their signature.

The harder claim, and the more important one, is what they say about sparse autoencoders. SAEs are the bet Anthropic, OpenAI, and DeepMind have all made on how to read large models. Goodfire argues SAEs break continuous structure into disconnected pieces. Their example: words ending in “-ore” form one smooth curve in the model’s internal space, and SAEs shatter that curve into a handful of unrelated features. The unity disappears.

If that critique holds for big models, a meaningful slice of current AI safety research is studying artifacts of its own tools, not the model.

What’s oversold

The framing, “the world inside neural networks,” does more work than the evidence supports. The paper smuggles in a big claim, that models contain a faithful map of reality, which is hard to disprove because nobody knows what would count against it.

What Goodfire actually showed is narrower and more useful. Representations are curved. The curves are causal. Tools that assume straightness are leaving capability on the table. That’s enough. The cosmic framing is marketing.

Two real gaps the paper doesn’t address:

• Does it scale? They show the geometry is causal for one toy model. Does the same picture hold for a 70-billion-parameter language model? Open question.

• Is it the same picture across models? If different models trained on the same data find the same curves, the geometry is approximating something real about the world. If not, the curves are model artifacts and the philosophy crumbles.

Both questions are answerable. Neither is in the paper.

Why this is worth watching anyway

Two reasons.

One, it reframes the tooling debate. The interpretability community has been arguing about which kind of feature dictionary to build. Goodfire is asking whether a dictionary is even the right object. A map of curves wants different math, different methods, different papers.

Two, the parallel with biology is getting hard to dismiss. Grid cells, place cells, and head-direction cells in mammalian brains encode space as exactly the kind of curved structure Goodfire is finding inside artificial networks. That work won the 2014 Nobel Prize in Physiology. When evolved biology and trained silicon land on the same shape, the convergence is worth taking seriously.

A year ago I wrote that interpretability was the race we couldn’t afford to lose. Goodfire’s work is what running that race looks like when it goes well.

This is an important direction, half-marketed, and the next year of interpretability research will tell us whether the curved-geometry view replaces the feature-dictionary view or merges with it.

Watch the scaling question. Watch whether somebody bigger than Goodfire bets on this lens.

If they’re right, a lot of recent activation-steering work is about to age badly.

Around the Corner: short reviews of ideas worth watching. Opt-in section, not part of the weekly Run Data Run email. Subscribe to the main list for longer essays.

I’m writing this in Claude Code. That’s not a credibility flex. It’s a working confession before the rest of the piece reads as scoring. For the past year I’ve used all three of the tools I’m about to compare. Claude Code is my daily driver. I call Codex from inside it often, and I keep OpenCode in regular rotation on purpose, as a discipline for staying sharp to the alternatives. Take the rest accordingly.

I don’t want to wake up in two years and discover I let myself get lured into one of them because everyone else did.

The three I’ve used as daily tools for the last year are Codex CLI from OpenAI, Claude Code from Anthropic, and OpenCode, an open-source project from sst.dev. I wrote a version of this comparison earlier; the field has reshaped enough that the old piece reads like a different category of tool. All three shipped meaningful updates in the last 90 days. All three look more alike on a feature checklist than they did three months ago. All three are still, at the architectural level, doing fundamentally different things. The temptation when you stare at the feature list is to call that convergence. The temptation when you sit inside one of them is to call it tribalism. Neither read is right. The field is neither converging nor consolidating.

What I’m not comparing is a much longer list. The harness race has the same shape as the model race: too many credible players for any one person to evaluate fairly, with a new entrant every week. AWS, Google, Moonshot, Alibaba, Mistral, Block, Cognition, Replit, Sourcegraph, the Cline and Continue and Aider open-source lineage, plus a dozen others all shipped serious harnesses in the last year. The full inventory is in the appendix at the end. This piece stays narrow on purpose, on the three I actually run.

This is not a buyer’s guide. I’m not picking a platform for an engineering team. I’m describing how three different tools fit into the long-horizon work of one person who writes books, drafts essays, builds models, cleans data, and ships creative work, sometimes in the same week, often in the same day.

How I actually use them

Most of my work runs through Claude Code. The book, the blog drafts, the long-horizon research, model building, all of it lives in a tool that remembers what I corrected last week and carries voice rules across hundreds of sessions. I call Codex from inside Claude Code when the work shape changes: forty-five minutes on a notebook, a feature pipeline to iterate on, a quick statistical test. Codex is terse and decisive, and the terseness wins on tasks I can name in a sentence and finish in a session. OpenCode sits in regular rotation on lower-stakes work, on purpose. Once a quarter I’ll move a small project entirely into it for a week. The reason is fluency. The risk for anyone with a daily driver is that the daily driver becomes the only thing you can see. Six months from now Claude Code might still be the right answer. It might not. The way I’ll know is by keeping the alternatives warm enough to feel the difference when I switch back.

Choose your tool every six months. Don’t let your tool choose you.

The point isn’t that you need three tools. Each one is shaped for a different work shape, and a long-horizon worker with multiple work shapes ends up running more than one. A team picking a single platform is a different problem with a different answer. This is just how a person who has to ship across a book, a blog, a model, and a creative draft has settled in.

What they share

Six weeks ago you could draw a clean architectural line between these three. Today the line is smudged. All three now have cascading config files (CLAUDE.md, AGENTS.md, both). All three have a marketplace pattern for plugins or skills. All three have multi-agent worktrees, persisted-session primitives, and MCP-or-equivalent routing. Codex shipped its Chrome extension and Claude Code shipped its enterprise MCP OAuth update on the same day this month.

Read only the changelogs and you’d conclude these are turning into the same product. Use them for a week each and you’d conclude something else.

Where they actually differentiate

The differentiation isn’t in the features. It’s in what each one is shaped to do well.

Claude Code is built for long-horizon coherence. The 1M-token context, the skills-plus-auto-memory loop, the durable cron, the cascading config: a stack designed around running unattended for hours or days. Loops persist across machine restarts via LaunchAgents, and a /loop primitive lets a task self-pace its own intervals when you don’t want to pick one upfront. Skills live as versioned markdown files with frontmatter that controls when they auto-trigger and what context they pull in, with a marketplace layered on top of that local model. Memory cascades from user-global rules to machine-specific overrides to project instructions, with an auto-memory layer that captures corrections and surfaces them in future sessions so you don’t re-explain the same preference every Monday. I wrote about the broader version of that memory pattern, vault-as-brain underneath the harness, in From the Vault. The verbosity, the premium pricing, and the heaviness on quick atomic tasks all follow from the same commitment.

Codex CLI is built for per-session bundled productivity. Terse by default, marketplace plugins shaped around tasks you can name in a sentence, optimized for the “forty-five minutes and move on” rhythm. Codex closed part of the gap on long-running work at the end of April with /goal, an OpenAI-endorsed Ralph loop that runs a single objective through plan-act-test-review cycles for hours, with pause/resume/clear from the terminal. It’s a different shape than Claude Code’s durable cron: one objective held until done, not a schedule of recurring jobs. Plugins bundle a named workflow with its prompts and permissions, so the unit of reuse is a task you can describe in a sentence rather than a long-running role. Memory is AGENTS.md-cascading, same shape as the others, but nothing tracks what you corrected last week across sessions. The compactness is the design.

Open, finish, close.

OpenCode is built for substrate freedom. MIT-licensed, 75+ providers natively, dual Build/Plan toggle, no permission required from any single model lab to ship. The provider abstraction is the decision everything else hangs off. Skills and memory are thinner than the other two on purpose, because every commitment to a particular memory shape or skill format becomes a coupling to the model that has to read it. The Build/Plan toggle handles in-session structure: Plan reasons without acting, Build acts within the plan. Durable loops and cross-session memory are left to the operator to wire up. The cost is more glue. The benefit is that no single model lab can deprecate your tool out from under you.

These three commitments don’t reduce to a feature checklist. They show up in the texture of using each one for a week. The feature-by-feature breakdown, with benchmark numbers and the 90-day shipping cadence per harness, lives in the AIXplore companion to this piece.

Economics

Pricing across all three rotates faster than this paragraph will age. ChatGPT Plus rebalanced its Codex allocation in April. Claude Code’s Pro tier may lose access entirely. OpenCode launched a $10/month Go subscription this week. None of these prices will hold for 90 days. What’s stable is the pattern: gateway routing has become a real economic lever. A handful of shell aliases that swap a gateway target before launching the harness can route voice-sensitive drafting to the premium model whose character you want, bulk file scanning to a cheaper open-weight model, and sensitive code to a self-hosted endpoint.

The harness stays constant. The model rotates.

Bills drop 40% to 80% versus running everything against the premium tier. OpenCode does this natively. Claude Code does it through the gateway pattern. Codex does it more by accident than design, but it does it.

The model still matters, for a non-obvious reason

There was an analysis that made the rounds in March, after a leak exposed the source code of one of these tools. The argument was that the system is mostly deterministic software, not AI. Most of the value, the argument went, comes from how the tool routes context, dispatches its sub-tools, and shapes what the model is allowed to see. The model is a small, expensive component nested inside a much larger software system.

That’s true. It’s also incomplete in a way that matters for anyone choosing one of these tools.

What the leak actually showed is that the tool is the lens. It’s a lens shaped around a particular model’s character. Claude Code’s long-horizon coherence depends on a model that holds context coherently over hours. Codex’s atomic-task speed depends on a model that produces short, decisive answers without thinking out loud. OpenCode’s freedom-to-choose story depends on whatever model you point it at being able to follow the tool’s protocols at all.

Each tool is doing the work that lets a particular model character be useful at scale.

Picking one of these is partly picking which model character you trust to run while you’re not watching. The next question is whether you need to be there at all.

Sidebar: autonomous agents are a different category

A separate category is starting to show up at the edges of this conversation, and it’s worth naming because the boundary is blurring. Autonomous frameworks like OpenClaw and Hermes run as fleets of agents without per-session human input, optimized for long-horizon unsupervised execution. The three tools I described above all assume a human is in the loop, ready to interrupt, ready to course-correct. The autonomous category does not. That’s a different reliability problem with a different set of failure modes. A security incident in one of the open-source autonomous frameworks earlier this month is the cautionary tale that makes the distinction visible. Most builder-leaders are currently shipping the supervised category into production and prototyping the autonomous one. A future piece will cover that category specifically.

If you want the deeper technical breakdown of the three supervised tools, with feature-by-feature differences, benchmark numbers, and the architectural detail of where each one is going next, that lives in the AIXplore version of this piece.

Appendix: the rest of the field

I haven’t put serious hands on any of these in the past year, which is why this piece doesn’t compare them. Each one is shipping, has real users, and carries a coherent point of view about what a coding agent should be.

• AWS: Kiro, Amazon Q Developer

• Google: Jules, Gemini CLI, Antigravity (parallel agents with a built-in browser)

• Moonshot: Kimi CLI

• Alibaba: Qwen Code

• Mistral: Vibe

• Cursor and Windsurf: both shipping CLI modes alongside their IDEs

• Open-source IDE extensions: Cline, Roo Code, Continue.dev

• Aider: longest-running terminal agent, the one most of these others trace back to

• Block: Goose

• Charm: Crush

• Plandex, Replit Agent, Devin (Cognition), Zed AI, Sourcegraph Amp, Augment, Droid, iFlow, Kilo, Warp 2.0, GitHub Copilot CLI

A comparison written without hands-on time is what makes most of these roundups useless. So this piece stayed narrow on purpose. If your daily driver is one of the above, I’d be curious how it compares.