AI Coding Weekly

Astra 6.2 beats Sol 6 by 20 points on security, Ubl says

Vercel's own held-out cyber benchmark, posted by the people who run it, is the only evidence so far.

Malte Ubl says OpenAI's Astra 6.2 scores 20 points better than the previous best model, Sol 6, on cybersecurity tasks, and that it is "entirely in its own class". His practical conclusion is that teams should run security review again with Astra 6.2 as the model, because it will now find vulnerabilities the previous AI security review pass would have missed.

Malte Ubl
@cramforce
X
Astra 6.2 is scoring 20 points better than the previous best (Sol 6). It's entirely in its own class.
Nov 17, 2026 · View on X

Guillermo Rauch, posting alongside him, adds the one detail that makes the number interesting. Vercel's DeepsecBench is a stealth cyber benchmark, and no lab has had access to the codebases in it to train on. That is the usual defence against the obvious objection, that a model scoring well on a public security benchmark may simply have seen the vulnerable code during training.

Guillermo Rauch
@rauchg
X
DeepsecBench is a stealth cyber benchmark. No lab has had access to the codebases to train on.
Nov 17, 2026 · View on X

What the number does and does not say

This is a single benchmark, run by the vendor that built it, reported by that vendor's own people. Ubl opens with "other feedback aside", which is a reasonable signal that he is making a narrow claim about one capability and not a general verdict on the model. A 20 point gap on held-out security tasks is not evidence that Astra 6.2 is better at refactoring, instruction following, or any of the other things people argue about week to week.

It also matters that the advice is to rerun, not to switch. Security review is one of the few agent workloads where rerunning the same codebase with a stronger model is cheap and the downside of a false positive is a wasted triage, not a broken build. That is a different economics from swapping your daily driver coding agent.

Vercel has been running security review at scale for a while. Its agent found 57,897 vulnerabilities and patched 14,665 in a week by the company's own count, so the comparison baseline here is a system already in production rather than a lab demo.

For working engineers, the useful takeaway is the method rather than the score. A benchmark nobody could train on is the only kind whose numbers survive contact with a new model release, and almost nobody publishes one.

Have the good opinion, not the loud one

One email a day. Under three minutes. What working engineers found out about coding with AI.

Get the next one by email

Coding with AI, read daily so you do not have to. The experiments, the receipts and the arguments from engineers shipping real software. Not a changelog.