Theo's Jev router benchmark shows no win on cost or speed
A $1,000 overnight run on DeepSWE put OpenRouter's new router level with GPT-6 Astra on low, slightly pricier and almost 5x slower.
Theo has numbers now. He says he stayed up til 2am and spent $1,000 benchmarking OpenRouter's typesafe/jev-router, and on DeepSWE it performed roughly the same as GPT-6 Astra on low, cost slightly more, and took almost 5x longer to run. That fills in the earlier version of his objection, which was structural rather than measured.
I'm just tired of these dumb ideas.
OpenRouter announced the router as a cache-aware model router powered by Jev and typesafeai, picking the best model and reasoning effort for each request while balancing quality, speed and cost. Theo's argument against it is that Jev is a system one model, meaning it categorizes rather than reasons, and the thing being categorized is a prompt with no surrounding context.
In order to know how much thinking a task requires, you have to think. Jev does not "think
Why a prompt is not enough
His example is the one everybody has typed. Ask an agent to port this to Rust and the router has no way of knowing whether the target is a 100-line TypeScript file or a million-line complex application. Same for bug fixes, where a request can mean a two-line change or a tour through the messiest part of the system. As he puts it, to know how much thinking a task requires, you have to think, and Jev does not think.
The second half of the argument is about money, which is the part the routing pitch rests on. Theo says routing to a dumber model halfway through a task is not going to save much in most cases, because cache writes are already such a large share of cost. If that holds, the savings from a mid task downgrade are smaller than the sticker prices of the two models suggest.
He also reported an earlier impression from partway through the benchmark, that the router was behaving basically like a DeepSeek 4.1 flash router.
None of this is a verdict on Jev itself. Theo says he loves Jev, uses it for a ton of things, and calls it one of his favorite releases of the year. The complaint is narrower, that a fast classifier is being pointed at a decision it cannot see well enough to make.
For working engineers, the useful takeaway is the cost shape rather than the ranking. If cache writes dominate your agent bill, per request model selection is optimizing the smaller number, and one engineer's DeepSWE run is one result on one benchmark, not a settled case.
Have the good opinion, not the loud one
One email a day. Under three minutes. What working engineers found out about coding with AI.