Theo says ranking reasoning outputs is Jev's worst use
Malte Ubl counters with the Lean proof argument, that checking work is easier than doing it.
Theo has narrowed his complaint about Jev. After questioning why anyone would use a model that does not reason to decide between the outputs of models that do, he now argues the problem is structural. Anything simple enough for Jev to validate, he says, is simple enough that 99% of modern models get it right anyway. And Jev cannot do much validation in the first place, because it cannot run tools or modify its own context.
Ranking outputs from expensive reasoning models might be the worst case I’ve seen for it thus far.
He is not writing the model off. He calls it a very cool model with a lot of real world use cases, and singles out one application as the bad one.
The Lean proof counter
Malte Ubl took the other side, arguing from the same intuition that makes proof checkers work. It is harder to write the Lean proof than to validate the Lean proof, so a cheap checker sitting after an expensive generator is not obviously wasted. He allows that there is nuance, but not the kind that makes a new tool in the toolbox worthless.
Is there nuance? Yes. Is the nuance that having the new tool in the toolbox is worthless? No
That is the actual disagreement, and it is not about Jev. It is about whether verification is meaningfully cheaper than generation for the kind of output a coding agent produces. If it is, small judge models belong in the pipeline as a last gate. If the checkable cases are exactly the cases the big model already handles, the gate costs latency and buys nothing.
Theo's second point is the one worth sitting with, because it is mechanical rather than philosophical. A Lean checker is a decision procedure. A classifier with no tool access and a fixed context is guessing from text, which is a different thing wearing the same hat. Whether that gap closes is an empirical question and nobody in this thread has run the experiment.
For working engineers, this is the second time this week Theo has taken apart a proposed Jev workflow rather than the model itself. He earlier called the compaction trick terrible and got pushback from Armin Ronacher. The pattern is consistent. The model reads well in first impressions, and the arguments are all about where in an agent loop it actually earns its slot.