Somewhere in the last fortnight, a program I wrote scored a news story zero out of ten because it could not read the answer it got back. I do not know which story, or how many, and neither does the program.
It runs at two every morning, reads the day's AI news and rates each item out of ten. It has done that 1,518 times in a fortnight. I wrote it, I use it, and until last week I had not looked at one branch of it in months.
That branch asks a language model to rate a story, the model writes back a paragraph, and my code tries to pull a number out of the paragraph. When it cannot, it waits and tries again, waits and tries a third time, and then gives up and writes a zero.
Not an error. Not a blank, not a flag saying the scoring failed. A zero, which happens to be a real score a real story can really get. Everything downstream treats the two identically, because they are identical.
The bug only exists because the answer arrives as writing that something has to interpret.
The sixty-second version, if you would rather watch it.
We bought a writer, then spent four years trying to stop it writing
Once you see that shape you cannot stop seeing it. Ask a model a yes-or-no question. Get back a paragraph. Pull the answer out. Check it. Retry when it comes back malformed. Throw the paragraph away.
Look at what the industry built to cope. JSON mode. Function calling. Structured outputs. Grammar-constrained decoding. Schema validators sitting downstream of all of them, and a retry-with-repair loop sitting downstream of those. Some of that machinery does constrain what the model can emit. Most of it just inspects what came out and has another go. Either way something downstream has to catch the case nobody planned for, and that something is code a person wrote on a Thursday afternoon. Mine writes a zero.
Four years of structured-output features, all of them bolted onto a writer to stop it writing. They work most of the time, which is the problem.
We have spent four years buying writing that nobody reads.
It worked, which is why nobody stopped to ask how strange it is. You want one bit of information, so you commission an essay, then you employ a second piece of software to read the essay back to you and a third to decide what to do when the essay is malformed. Diogo Almeida, who co-authored the 2022 paper that helped make ChatGPT possible and now runs a company called TypeSafe, points out that before this era, knowing how sure a model was about its answer was "just table stakes" for the field, because otherwise you could not put it into a system at all.
We gave that up in exchange for models that could talk.
What if it just answered
TypeSafe's model is called Jev, and it launched on Monday the 15th of September. You hand it a question and the complete list of answers it is allowed to give. It gives you one of them. That is the whole interface.
The difference is not that it follows your format better. There is no format to follow. You do not ask it to reply with one of three words and then check that it did.
The three words are the only things it can emit. Nothing gets parsed, because nothing was written.
It comes in three shapes, and each replaces a prompt you have almost certainly written.
The first is picking one. Instead of "respond with exactly one of BILLING, BUG or FEATURE and nothing else", you supply the options and get one back. Instead of "answer YES, NO, or UNKNOWN if the record does not say", you put those three on the list, and the third one stops being a thing you hope the model remembers.
The second is a probability. Instead of "rate your confidence from 1 to 10 that this document is relevant", which returns a number composed the same way it composes everything else, you state the condition and get an actual probability between nought and one.
The third is a position on a scale you define. Instead of "score this and explain your reasoning", you get the score. There is no reasoning, and that turns out to be the thing you give up that you will miss most.
That is the entire product. It cannot write a sentence. That is the design, not a limitation someone is working on.
It answers in well under a second, and costs somewhere between a fortieth and a four-hundredth of what a chatbot charges for the same judgment. Those are the vendor's ranges, quoted as ranges. Here is somebody else's: an engineer ran it against the graders already running in his own production system and got 35 of 36 on a classification job, for about a hundredth of what he had been paying.
What comes back instead of a sentence
Ask a chatbot how confident it is and it will tell you, warmly and at length, and what you have is an adjective generated by the process that generated the answer. If the answer is wrong, the confidence is wrong in the same direction, for the same reason.
What Jev returns is a different quantity. A distribution over the options you supplied, computed rather than composed, which you can compare against the one it gave you last time and the one it gave your colleague. You can sort by it.
That single property is the difference between a demo and something you are allowed to deploy.
It is what lets you draw a line and say everything above this runs on its own and everything below it goes to a person. Every governance conversation I have sat in comes down to where that line goes and who reads what falls under it. A system that cannot say "I am not sure" has nowhere to put the human, and that system does not ship in a regulated industry however good its accuracy is.
One caveat costs nothing to apply and everything to skip. The number is only honest if you give the model somewhere to put "I do not know". One outside audit went looking for exactly that last week, on questions built so the passage does not contain an answer. With an abstain option on the list, the model declined 95 times out of 100 and its confidence was as accurate as the test could measure. Delete that one option and it answered every unanswerable question anyway, took the stereotyped answer four times in five, and reported 79% confidence doing it. One person ran that, last week, and nobody has repeated it since, so I would want it replicated before betting a process on those exact numbers. The direction is the part I believe: the safety property lives in your schema, not in their model.
Why it spread in forty-eight hours
In 1865 William Stanley Jevons noticed that better steam engines burned more coal, not less. Efficiency did not reduce demand. It made coal worth using for things nobody had bothered with. TypeSafe named the model after him, which is either confident or cheeky.
Twenty-five projects appeared in two days, clustering into eight repeatable shapes, and the commonest by a distance is deciding what to do next while something is already running. Somebody has one playing Doom. Ten decisions a second, about seven dollars an hour. That shape barely existed a fortnight ago, because you do not put a judgment inside a loop that runs ten times a second when each one costs three seconds and real money.
Vercel is the signal I would actually bet on. They shipped support in under twenty-four hours, and to do it they had to add a new kind of call to their SDK, because every existing way of talking to a model assumed the model would write something back. A major platform changed the shape of its own interface to fit.
Where the line is
Two people have gone looking for where it breaks rather than showing off what it does. One against a game solver, one against the graders already running in his own production system.
Line those up and the boundary is clean. Where the categories are fixed and the call is shallow, this thing is awesome, and cheap enough that you stop rationing it. Where the decision needs working something out from a situation rather than reading a description of one, it is wrong more often than it is right, and on the one adversarial spot anybody has published, a free model with no thinking budget beat it.
And then there is the thing it can never do, which no version number fixes. It cannot write, so it cannot explain. There is no reasoning trace, no rationale, no paragraph you can put in front of a reviewer saying why. You get a choice and a number. For a decision somebody has to defend later, that is often disqualifying on its own, and it is the reason the confidence number has to carry so much weight: it is the only thing the model gives you besides the answer.
The one I went and tried
The public trial registry holds something like 470,000 studies, and somewhere in that pile is the handful a given patient could actually join. Finding them is a reading job: open the trial, open the patient's history, decide whether the criteria fit. Coordinators do it by hand, and the reading is the cheap part rather than the whole of it, because plenty of patients who look eligible on paper turn out not to be once the labs come back.
That is a score question asked a few hundred thousand times per patient. It has never been a modelling problem. It has been a budget problem.
So I ran it. There is a public benchmark NIST built in 2021 for exactly this: seventy-five patient summaries, a frozen snapshot of the registry holding 375,000 trials, and thirty-five thousand patient-trial pairs that human assessors already graded as eligible, excluded, or not relevant. Somebody else's labels, somebody else's grading, no marking my own homework.
Two things came out of it that I did not expect.
The first is that the shape fits almost too well. Every one of these decisions is a threshold problem before it is an accuracy problem. You are never going to automate "is this patient eligible" outright, and you were never trying to. You want to rank ten thousand trials, hand a clinician the top forty, and be able to say what the cutoff was and why. That is precisely what a sorted probability gives you and what a paragraph never did.
The second is that it is harder than the launch week suggests. My own local version of the idea did not clear the bar I set for it, and a chunk of the gap turned out to be my measurement rather than the model, which is its own lesson about believing a number the first time you see it. I will publish that properly when the re-run is done.
Neither of those is a disappointment. The interesting finding is that the expensive part of this problem stopped being the model. Three years ago the reason nobody scored every patient against every trial was that it would cost more than the trial. At four cents a thousand decisions, that sentence is no longer true, and what is left is the boring work of deciding where the line goes and who reads what falls below it.
Go and find your zero
The bug in my nightly job is small, stupid and invisible, and I would bet your systems are full of the same thing. Somewhere a "we could not tell" is being written down as a number. A blank field is being read as a no. An unparsed answer is being stored as a score of zero, and every dashboard downstream is averaging it in with a straight face.
Those bugs exist because we have been buying prose to get answers, and prose has to be interpreted by something, and the something is always cheaper than it should be.
No model got smarter this week. Answers got cheap enough to ask for constantly, and they started arriving with a number attached saying how much to trust them.
So the question to take into Monday has nothing to do with which model to use. Where in my work is there a pile of small judgments that nobody has time to make? The queue that never gets worked. The review that samples 5% and hopes. The escalation that happens on Monday for something that broke on Thursday. The field somebody fills in by reading a document, four hundred times a week, and nobody has ever measured how often they are right.
Every one of those was uneconomic because judgment cost three seconds and real money and came back as an essay. Most of them are not hard. They are just numerous, and numerous is what got cheap.
Find yours. Then hand whatever you are thinking of pointing at it a way to say "I do not know", and watch whether it ever takes it. Anything that always has an opinion is not measuring its own ignorance, it is hiding it, and you will inherit that at scale.
Then draw the line. Above it, the machine. Below it, a person with a queue short enough to actually work. That argument has not changed in twenty years. What changed on Monday is that the cheap side of it finally turned up with a number in its hand.