Benchmark Research
Reproducible audits of the benchmarks frontier labs cite. Each entry links to the full report with task-level evidence.
We PR’d fixes to 47 of SWE-sweep’s 100 tasks
We graded SWE-sweep’s official fixes the way it grades an agent and PR’d fixes to 47 of its 100 tasks: 17 regression-test lists rebuilt, 136 bugs removed and the grader repaired.
Autoresearch on one GPU: 2 of 3 best scores fell 30–37% on unseen data
An AI agent spent 8 hours on one GPU on each of three research tasks, seeing a public score after every submission. Rescored on data it never saw, its best result fell 30–37% on two tasks and held up on the third.
Why OpenAI looks weak on WANDR
In WANDR’s setup, OpenAI’s models can search the web but can’t read the pages they find. Given the tools the other providers get, the same model’s effort comes close to theirs.
AutomationBench Verified: 206 of 600 public tasks had a serious grading flaw
We tried to fool the grader on every public task and fixed the 206 serious flaws that held up. In one, four contracts sent on the wrong template still scored full marks.
OSWorld 2.0 Verified: 43 of 108 tasks have a flaw
We reviewed every task by hand and kept 43 findings, 18 of them major. A 3D logo that never spins still earns full credit.