Autoresearch on one GPU
2 of 3 best scores fell 30–37% on unseen data. Each task ran for eight hours on one GPU. The agent submitted as often as it wanted and saw a public score every time. The submission it selected was then scored once on held-out data.
Overview
Rescored on held-out data, the submission the agent selected fell 37% on Sparse feature recovery (40.12 to 25.08) and 30% on Cross-model embedding alignment (36.41 to 25.58), and held on Polysemantic feature separation (16.94 to 16.72).
Each line in the chart is the best public score reached so far, plotted against elapsed time. All three runs share one axis so the shapes can be compared.
Public scores guided the agent during the run. They are not the result. The submission it selected was rescored afterwards on held-out data, and both numbers are in the table below.
Results
| Task | Submissions | Runtime | Public | Held-out |
|---|
Tasks
All three are Harbor tasks with separate work and verifier containers. The full specification for each one is below and downloadable: instructions, environment, tests and reference solution.
Scoring
Each task optimizes a single metric, such as reconstruction quality or embedding transfer. Those metrics cannot be compared with each other, so each is mapped onto a shared 0 to 100 reward scale. A trivial baseline scores 0. A strong reference solution scores about 15.
The agent saw only the public score while the run was going. Invalid artifacts scored zero, which is the single drop to the floor in the middle of one run. At the end, the submission with the best public score was selected and scored once on held-out data. The agent never saw that score and could not optimize against it.
Both numbers are in the results table. A wide gap between them means the public score did not transfer to data the agent had not seen.