Ball says watching an agent use GitLab beats a unit test
The verification argument moved from code review to tests, with Thariq arguing smarter models will not need to check their own work at all.
Thorsten Ball asked an agent to set up a GitLab instance and then build an integration against it. The agent opened merge requests, filed issues, pushed commits and recorded videos of itself doing it. Ball then went and looked at the commits it pushed, and asked whether that is better proof the integration works than a unit test.
That is one engineer's run on one integration, not a study. But it is a concrete artifact, which is more than most of the verification argument has produced this week.
Thariq pushes it further
Thariq read it as the bitter lesson showing up in day to day work, arguing the smartest models will need fewer and fewer tokens and will not need to verify their work or check it for bugs because they will just be right. Ball agreed in six words.
the smartest models will need fewer and fewer tokens, they won't need to verify their work or check it for bugs, they will just be right
The bitter lesson, for anyone who has not run into the phrase, is Rich Sutton's observation that general methods riding scale tend to beat hand built structure. Applied here, the hand built structure is the scaffolding we wrap around agents, the test suites, the review passes, the second model checking the first one's work.
It is a prediction about future models, not a measurement of current ones. Nobody in this exchange ran a comparison between an agent that verified and one that did not.
The same fight, one layer down
This is the second day of the same argument. poteto called adversarial reviews mostly a waste of tokens and said she would rather spend those tokens on empirical combined with formal verification, having found agents tend to get lost in nitpicks and YAGNIs. Ball and Thariq take the swipe further, from review to tests to self checking in general.
Note that poteto and Ball are not actually making the same argument. poteto wants to move tokens from review into verification that actually checks something. Ball is asking whether a working integration you can inspect by hand is itself the verification. Thariq is saying the whole category eventually goes away.
For working engineers, this is the old integration versus unit test fight with the cost structure flipped. The thing you are economizing on is no longer your afternoon, it is tokens spent proving the work, and the tradeoff curve moves every time the models get better. Ball's GitLab run is evidence that end to end artifacts can be convincing. It is not evidence that your test suite is now optional.
Why have training wheels if you never fall over?
Have the good opinion, not the loud one
One email a day. Under three minutes. What working engineers found out about coding with AI.