The Internet has been abuzz about Jev, an AI model from TypeSafe AI that returns typed values with probability estimates. The company claims it is fast and cheap. The question is whether anyone has stopped to ask whether it actually works.
Why Doors Don’t Work
The writer opens with a complaint about doors. In one story, a door fails because a body is in the way. In another, roughly a billion dollars worth of gold blocks the path. The writer’s model of doors is simple: doors should not “suck” inexplicably. When the character mutters “stupid thing sucks,” the writer finds it hilarious, though they admit their brain might just be the problem.
Jev is faster and cheaper than other approaches. But the writer asks a practical question: what does it mean to know Jev is working?
The Hard Part Nobody Checks
The writer compares using Jev to getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. You still have to do the hard part. To know if Jev is working, you have to build tests and a truth-check pipeline. If you have those, you are most of the way to building your own solution.
Nobody buying Jev is running those tests. They hand opaque questions to the model and get opaque answers. They can check the “AI-powered” box and ship before Friday. When this breaks downstream logic, they shrug and say “well, AI makes mistakes.”
Confidence Scores Are Worse Than Nothing
Jev gives confidence scores. The writer asks what anyone is supposed to do with them. You need to know how well those scores are calibrated and what the costs of uncertainty look like. Jev’s own ads focus on benchmark scores, not calibration. The documentation suggests a 0.5 threshold for “do nothing” and a 0.9 threshold for “do high-risk actions,” with an aside noting thresholds depend on the domain and the model’s performance.
The writer describes this as cargo-cult behavior at best and an excuse for failure at worst. The model was only 73% confident! That means the error budget is 27%.
Who Owns the Failure
When a button breaks on a website, the writer expects someone whose job is to understand why the endpoint is failing. The ownership is well-defined, if opaque. With Jev, the experience is “stupid thing sucks.” Software already feels capricious; more failures just change the rate of frustration.
The writer fears that “sometimes it just sucks” will become the accepted endpoint of investigations. That is sad because LLM-accelerated development could help solve some of these issues. Automated testing workflows exist but are not written because of lack of engineering time.
The tragedy of software engineering today is that we are actively building systems where neither the user nor the builder checks whether there is a body behind the door. We just shrug and conclude: stupid thing sucks.
| Stage | Activity |
|---|---|
| Setup | Get FTP account, mount with curlftpfs |
| Old workflow | Use SVN or CVS on the mounted filesystem |
| New workflow | Hand opaque questions to Jev, get opaque answers |
| Testing | Build evals and a ground-truth pipeline |
The writer’s final worry is that the industry is normalizing inexplicable failures. Jev is fast and cheap, but nobody is checking whether it works. The confidence scores are not a fix. The endpoint of every investigation is not a root cause; it is a shrug.
Source material: “The Normalization of Inexplicable Failures,” ihatethefuture.com.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.