Theo's tool eval cuts T3 Code context 79%

Theo says agents now use the tools in T3 Code 4x better while using 79% less context, the result of a week spent building an eval out of his own day to day tool usage and his failure cases. He posted it as day 23 of his running build log, and put the token cost of the exercise at $1,500.

Theo - t3.gg

@theo

X

Amazed at how many problems you can kinda just turn into evals and then do shit like this.

The problem started with growth. T3 Code began as a wrapper so thin that the agent did not know what it was, and then Orchestrator V2 arrived, which lets harnesses call each other. Add a built-in browser, a device manager for mobile, subagents and delegates, send to other thread, and inline HTML visualizations, and the tool schema got, in his word, bloated. The worst single offender was listing every model available for subagents. Harmless for most people, bad for OpenCode and Pi users who have hundreds of models in the list. That one is fixed.

The eval was the actual work

Rather than trimming descriptions by feel, Theo turned the ways his agents used the tools, plus the failures and rage moments, into a generic eval he could re-run against any change to a tool or its description. He expected the job to be mostly shortening text. Instead the eval surfaced error cases he did not know he had. It ended up with over 80 cases, and Opus failed more than half of them.

Then the hill climb, which follows the same shape as his earlier runs. Bench Sonnet 5.5, adjust tool descriptions until it scores perfectly, add Sol and repeat, add Opus and repeat, merge because error rates are now better, then open a fresh thread and tell Opus to trim until context use drops 50% with no regressions on the eval. Ship.

For working engineers

The transferable part is not the 4x or the 79%. It is that tool descriptions are usually tuned by vibes, and the failures they cause show up as agent misbehavior that nobody traces back to a schema. Theo's move was to write down the moments the agent annoyed him, make them a fixed scoreboard, and let a model optimize against it. One engineer, one codebase, and the numbers are his own eval scoring itself, so treat them as a direction rather than a benchmark. The method is the part you can copy this week.