Somebody Is Finally Checking
A contest opened eleven days ago has produced more reproduction attempts than the field's own dedicated effort has managed in any year. Whether that becomes a check on the literature turns on a detail sitting in the scoring rules.
Every Sunday I pick one paper or release that's worth your time, break it apart, and tell you why it matters. No hype. No summaries of summaries. Just the idea, explained.
A published result is a promise. Run this, and you will see what we saw. For most of the history of computational science nobody collected on that promise, because collecting cost about as much as the original work and earned a fraction of the credit.
On July 15, Hugging Face and alphaXiv opened a contest to collect. It closes Sunday August 2. Roughly 6,800 papers from ICML 2026 are in scope, anyone can attempt a reproduction, and every attempt gets published as a public logbook that an automated judge then scores. The prize pool is $4,500, paid in GPU credits rather than cash, which tells you who the organizers think is entering.
As of this morning, eleven days in: 4,147 judged logbooks from 312 people, carrying 20,594 separate verdicts on individual claims. About 1,600 papers have been attempted at least once, roughly a quarter of the conference.
To see why that is a strange number, you need the thing it replaces. Joelle Pineau started the ML Reproducibility Challenge at ICLR 2018 as "a small community experiment." Eight years on it is the field's serious effort, mostly run through graduate courses, and its NeurIPS 2019 edition drew 173 papers claimed, itself a 92% jump on the year before. Claimed, not finished. Published reports have run in the tens. ReScience C, which demands a fresh independent implementation rather than a re-run of the authors' code, manages about a dozen a year.
173 papers claimed in a year, against 4,147 judged logbooks in eleven days. That is not an improvement on the existing effort, it is a different quantity.
What is actually being scored
One design decision separates this from a leaderboard, and it is easy to skim past. The unit of judgment is not the paper. It is the sentence.
Every paper has been broken into its individual claims, and each claim gets its own verdict: verified, falsified, toy (a scaled-down version that says so), or inconclusive. Nobody asks whether a paper "reproduced," which is a question with no answer. A paper can come back partly verified, partly toy and partly inconclusive at once, and most of them do.
Then the scoring, from the contest's own FAQ: "2 points for a full reproduction or full falsification, 1 point for a toy-scale reproduction, 0 otherwise."
Falsifying a claim pays what confirming one pays. Against a literature where negative results are half as publishable as positive ones, that is a deliberate inversion.
When Nature surveyed 1,576 researchers in 2016, 24% had published a successful replication and 13% a failed one. The same survey found more than 70% had tried and failed to reproduce someone else's experiment and 52% agreed the field had a reproducibility crisis, and then 73% said they still trusted at least half the papers in their own field.
The failures are also the deliverable. A logbook, in the organizers' description, holds "the experiments, simplifications, failures, and results it found." Prior efforts published conclusions. This one publishes the attempt.
What it does not do is test the hard version. Machines have been tried on this before and were not good at it: every benchmark that measured reproduction topped out between 16% and 27% on rebuilding a paper from scratch, with top ML PhDs ahead of the models, and only reached about 60% when handed the original code and data. This contest tells participants to start from the authors' code. Re-running someone's code, rebuilding from the paper alone, and getting the same finding on fresh data are three different tests, and headline percentages get quoted across that boundary constantly.
What the inside looks like
For a large class of papers the easy version is now free, which is the whole reason this works. We downloaded a full-marks logbook from the contest leader and read it. The entire reproduction ran on a laptop processor in three seconds. No GPU, no cluster, no cloud bill. The real ceiling is not compute but publishing: Hugging Face caps accounts at 20 new pages a day, and when a participant asked for relief the organizer's answer was flat, "we cannot lift the 20 spaces/day limit so I'll close this issue."
I entered, which is the only reason I have anything to say about the texture of it. We are 23 judged logbooks and 174 points in, against a leader at 1,862. Not a contender, and far enough inside to see how the work divides.
It divides into two lanes, and we ran them differently on purpose.
The volume lane is an autonomous agent called Vulcan, and it never publishes anything. It runs on a box in my house on free local compute, and its job is narrow: take a paper whose claims are all theory and check every formula numerically, against a brief that fixes the method rather than the answer.
A check that passes on every input you can construct certifies nothing.
Its audit of one paper came back as 721 lines of code that matched the paper's algebra to the last digit a computer can represent, plus a deliberate sanity test that failed when it was supposed to. Careful work, at no cost, with a hard limit written into the contract: it stages, a human publishes. No autonomous process of mine puts a verdict about somebody else's paper into the world.
The other lane is me and Claude Code running on Opus, for the parts the first one cannot do. Two things came out of it.
The first is that picking the right paper beats checking it well. Our most careful audit matched its paper exactly and scored 2 points out of a possible 8, because half that paper's claims were hardware benchmarks that sit at inconclusive until somebody spends days of compute on them. The judge said so directly: the logbook left "half the paper's claims without any experimental evidence." A paper whose claims are all theory earns 12 points in an afternoon on a laptop. Selection is the skill, not rigor.
The second is what the process catches. One paper states a theorem using a fixed setting, and then, four thousand lines later, its own proof assumes that setting changes with the length of the run. Not a typo, and not a disagreement between the paper and the world: the paper disagrees with itself, in print, peer-reviewed and accepted. Nobody had noticed because nobody had reason to run the theorem as written and watch it fail.
Beyond those two, the work was unglamorous. It caught our own staging script dropping most of a conclusion, so a 60-line write-up shipped as three sentences. Vulcan itself lost twenty hours to an outage at its model provider, with no backup configured and nothing raising an alarm.
None of those failures announce themselves. Every one produced output that looked finished.
The three percent
Of 20,594 claim verdicts, 615 are falsified. That is 3.0%. The rest split 43.5% verified, 28.9% inconclusive, 24.6% toy. The distribution has barely moved for days while the corpus grew by hundreds of logbooks: the shape is stable, the counts are not.
Three percent is low, and it has two readings. Either the literature is in better condition than the surveys suggest, or falsification is the hardest verdict to reach and the easiest to get wrong.
In one day, four of our audit agents returned a falsification. Three of them were wrong. All four came out of the automated lane. All three errors were caught by the other one.
One agent read a formula out of a PDF that had lost a character in scanning, then correctly proved the mangled version false. Another was checking a sentence the authors never published, because the claim it was handed came from a draft that is not the public paper. The third found a difference too small to distinguish from rounding error and called it a contradiction.
Every one produced a confident, well-formatted, coherent falsification. I wrote about that pattern in The Failure That Leaves No Corpse, and this is the cleanest instance of it I have run into.
The rule we ended up with fits on one line: does the paper disagree with itself, or does the claim disagree with your copy of the paper? Only the first is a falsification. The second is a bug in your pipeline wearing a falsification's clothes.
Applying it cost us the verdict. Across the 130 claim verdicts we have published, the number labelled falsified is zero. The one candidate that survives is the self-contradicting theorem above.
The default failure of an automated checker is not a missing answer. It is a wrong one, delivered with the same confidence as a right one.
This is not only our problem. The contest's own claim lists are extracted by a language model, and a participant found a claim in one paper's list that belonged to a different paper. The organizer confirmed it in the open, "That claim belongs to a different Safety-area paper and leaked in during auto-extraction," removed it, rescored, then swept all 6,768 papers for the same contamination.
And the bias is measured, not hypothetical. A June 2026 study of reproduction agents found that showing the agent the original paper made it more likely to confirm claims that could not actually be reproduced, and that small changes in wording pushed it further the same way. In this contest the agent always has the paper and the claim in front of it.
The judge is a language model
The thing scoring 20,594 claim verdicts is itself a language model.
There is published criticism aimed squarely at that. A 2026 position paper, "Stop Automating Peer Review Without Rigorous Evaluation", compared human against AI reviews of ICLR 2026 submissions and found two things. The AI reviewers showed "a hivemind effect of excessive agreement within and across papers that reduces perspective diversity." And their scores turned out to be "trivially gameable through paper laundering": rewriting a paper with a language model raised the scores it got, on style rather than science.
None of that is a reason to dismiss the contest, and I want to be careful here, because I am competing in it and have every incentive to be generous.
The defence is structural rather than technical. The artifacts are public. Every logbook is a page anyone can open, the verdicts are a public dataset downloaded 38,438 times last month, and the errors get argued in threads with the organizers answering. The contamination case is the proof, and the fix landed inside a day. A closed judge making the same error produces a number nobody can audit.
So the position I hold is that this design pairs a known-weak instrument with an unusually strong correction loop. Whether the loop outruns the instrument is an empirical question, and a week from now there will be enough public data to start answering it.
What it would take to trust this
I am not going to predict how it turns out. The contest closes August 2 and every distribution above is provisional.
The concrete claim is narrower and I think it holds. For a meaningful slice of published science, checking a claim now costs less than making it. Three seconds of laptop processor against however many months produced the original theorem. That ratio is why verification stayed volunteer work for a decade, and it has flipped.
What it does not change is the direction of the error. A checker that costs nothing will be run constantly, and cheap checking produces far more false accusations than misses. Confidently accusing a sound paper is worse than missing a bad one.
The contest's design anticipates this better than most things I have seen. Paying the same for a falsification as a verification removes the incentive to go looking only for confirmations. The logbook has to carry the failures, so there is no tidy way to bury the attempts that went nowhere. And because every artifact is public, a wrong answer stays inspectable instead of disappearing into an aggregate.
Nobody has to trust the checkers. You can read what they did. That is a weaker guarantee than peer review claims to offer, and a stronger one than peer review delivers.
Sources
Reproducing ICML 2026, Open Reproductions: the contest, its FAQ, rules, and public discussion threads. Verdicts live in a public dataset.
MLRC 2026: Reproducibility as an Official Track at NeurIPS: the ML Reproducibility Challenge's history and Pineau's founding description of it.
Improving Reproducibility in Machine Learning Research, Pineau et al., JMLR 22(164): the 173-papers-claimed figure.
ReScience C: the journal that requires a new independent implementation.
1,500 scientists lift the lid on reproducibility, Monya Baker, Nature 533: the 2016 survey of 1,576 researchers.
PaperBench and CORE-Bench: the agent-reproduction benchmarks this contest inherits from.
Reproducibility agents and confirmation bias: the June 2026 benchmark showing that giving an agent the paper biases it toward confirming.
Stop Automating Peer Review Without Rigorous Evaluation: Baumann, Pei, Koyejo and Hovy, 2026: the hivemind effect and paper laundering.
The Failure That Leaves No Corpse: the earlier piece on failures that produce a confident answer instead of an error.
Sunday Deep Dive is a weekly series on Run Data Run. Every Sunday I pick one paper, release, or technique worth understanding, break it apart, and tell you what it means for your work. Free every Sunday, no paywall. If it was useful, the easiest way to support it is to subscribe and forward it to one person on your team who'd want it. If it wasn't, tell me why. I'll make it better.