Ronacher says agents can write their own test suite

Armin Ronacher said he is convinced an Opus 5.5 agent could re-implement all of CPython in about two weeks with no human input, working against just the tests, and probably land full ABI compatibility, no GIL and a JIT while beating it on all measurable metrics. His one piece of advice to anyone who wants to try it is to have the agent look at the JVM rather than CPython for design decisions.

The replies went where you would expect, that the real asset is the test suite, not the implementation. Ronacher pushed back on that in a follow-up. He said you do not actually need a test suite to rebuild something, and that you are probably better off letting the agent build its own suite specifically for the port, in part because a lot of pre-existing tests are bad for hill climbing (iteratively tweaking until a score goes up). He is not arguing the existing suite is worthless, only that it is not necessary. His evidence is one run, an agent that re-implemented Excel's formula engine a few weeks ago and wrote its own tests from scratch.

Armin Ronacher ⇌

@mitsuhiko

X

you don't actually need a test suite to rebuild something

Zechner says that is exactly the problem

Mario Zechner replied with two sentences and no hedging. He strongly disagrees, because his problem with agents is trust, and preexisting test and conformance suites increase that trust.

That is the whole split, and it is not about whether the model can do the work. Ronacher is talking about what makes an agent converge on a working port. Zechner is talking about what makes him believe the result. A suite the agent authored during the port is a grader written by the thing being graded, which is fine for hill climbing and close to useless as independent evidence that the port matches the original. CPython's own suite was written by people who did not know what the new implementation would get wrong.

For working engineers, the practical read is that these are two different jobs wearing the same name. Generated tests are a search signal. Conformance suites are an oracle. If you are porting something that already has a decades-old suite, the cheap move is to use both, let the agent write whatever scaffolding helps it make progress and keep the original suite as the thing that decides whether you ship. Nobody in this exchange ran the CPython experiment, so the two-week claim is a prediction, not a result.

Mario Zechner

@badlogicgames

X

my problem with agents is trust. preexisting test and conformance suits increase my trust.