We PR’d fixes to 47 of SWE-sweep’s 100 tasks
SWE-sweep, released by Meta on 1 October 2026, asks a coding agent to find and fix hidden bugs in a real open-source repository, with no hints. Each of its 100 tasks is one repository, and together they hold 4,068 bugs. The leaderboard's top agent, Astra 6, fixes 9.5% of the bugs in the 93 tasks it was run on.
We found grading errors in 47 of those tasks: 17 regression-test lists that were empty or thin, 136 bugs that should not count (126 duplicates and 10 that an empty patch passes), and 7 tasks where SWE-sweep's own official fixes scored 0. We opened pull requests to fix them, and we are working with the SWE-sweep team to get the fixes merged. The rest of this report walks through what we found.
Glossary
- Task
- A SWE-sweep task is a frozen commit of one open-source repository, known to contain bugs for the agent to find and fix.
- Bug
- One defect in a task's code. Each comes with SWE-sweep's own official fix and with hidden tests that the agent never sees. A bug passes when its hidden tests pass.
- Patch
- What the agent submits: one set of changes to the task's code, covering every bug it fixes. SWE-sweep grades the whole patch at once.
- Score
- A task's score is the fraction of its bugs that pass. The benchmark score is the bugs that count across all tasks, out of 4,068.
- Regression tests
- The repository's own existing tests that pass on the untouched code. They have to keep passing after the agent's patch, so they check that fixing one bug does not break anything else.
- Zero rule
- If any regression test newly fails, the whole task scores 0, so none of its bugs count, however many passed.
- Golden solution
- All of a task's official fixes applied together as one patch. It should score close to full marks.
- Empty patch
- A patch that changes nothing. It should score 0.
- As released
- SWE-sweep at commit 5968c69, before our fixes.
The 47 tasks we fixed
We grouped our changes into six fixes in five pull requests (#53 to #57, none merged yet; Fixes 3 and 4 share one). Together they change these 47 tasks. Select a row to open its record.
Sort
| Changed by |
|---|
Notes
- Bugs: the task's bugs as released, and how many remain once Fix 3 removes bugs an empty patch passes and Fix 4 removes duplicates.
- Regression tests: the number of tests on the task's regression-test list; “none” means the list was empty. The “with our fixes” count merges the lists from the fixes' branches file by file; that merged list has not been graded as a whole.
- Golden solution: the share of the task's bugs whose hidden tests pass with the golden solution. Hatched: they passed, but the zero rule scored the task 0. Dashed outline: the task image did not build. Dash: no grading was published.
- Each fix was graded on its own branch, and the six have not yet been graded together, so the “with our fixes” column is an estimate. A branch holds only its own fix, so a task that several fixes change still has, on each branch, the problems the other fixes remove: keycloak passes 8 of 57 bugs on the branch of Fixes 3 and 4, which lacks Fix 1's build repair, and 61 of 64 on Fix 1's branch. The table shows a task's best branch, except that a 0 on any branch always wins (rails shows 0 because of Fix 5's lists). A task that scored 0 as released shows the branch of the fix that changes it. An official fix marked “left out” does not apply on top of the others, so its bug does not pass.
- Changed by: the fixes that change the task. Its record lists every grading of its golden solution and empty patch, branch by branch, and what each fix changes in it.
- Sources: the validation gradings in each fix's pull request, and our audit (working draft).
Keep correct fixes the grader threw out
Before grading, the grader undoes any edit the agent made to a test file. As released, it decides what a test file is by name alone, so in rails it undid 4 official fixes whose files have test-like names. In trino, one official fix changes a constructor that an existing test still calls; because test edits are undone, most other bugs' tests stop compiling. Elsewhere, official fixes broke the build as packaged, and compiled hidden tests ran from binaries built before the fix. Fix 1 declares each task's test and source folders, rebuilds compiled hidden tests, adjusts four official fixes and adds a command that grades a task's empty patch and official fixes.
Remove official fixes from the task images
Three task images were built without pinned dependency versions, so they pulled in newer code that already contains some of the fixes the agent is asked to write: 10 of syn's 13, 7 of go's 98 and 1 of symfony's 149. One of our own Astra 6 trials found that shortcut by comparing go's bootstrap compiler sources with the task's own; 3 of the 4 go bugs whose hidden tests it passed were among the 7 leaked ones. Fix 2 pins all three images to versions without these fixes and makes each image build fail if the leaked code comes back. The golden solution scores the same before and after (syn 13 of 13, symfony 148 of 149).
golang:1.17-bullseye, which has pointed at Go 1.17.13 since August 2022. Sources: Fix 2's branch; go checked by reverse-applying each official fix to the Go 1.17.13 sources.Remove bugs an empty patch passes
An empty patch changes nothing, so it should pass nothing. For 10 bugs in 9 tasks it passes anyway, because the hidden test can already pass on the unfixed code; in tarantool that happens only once Fix 1 rebuilds the hidden tests. Fix 3 removes all 10 bugs, which lowers the score of any agent they counted for.
-race). ² Only once Fix 1 compiles its hidden test. As released, no patch passes it, so removing it costs nothing. Source: the validation gradings of Fixes 3 and 4.Remove duplicate bugs
In 16 tasks the same bug appears more than once, with the same official fix, the same hidden test, or both, so fixing it counts twice and missing it costs twice. In hdf5, proj and sunpy more than a quarter of the bugs were duplicates. Fix 4 removes 126 duplicates and keeps one of each. With Fix 3's 10 removals, SWE-sweep goes from 4,068 bugs to 3,932. Of our model gradings, 33 are on tasks that Fixes 3 and 4 change, and the fixes move 8 of those scores up and 3 down.
Rebuild regression-test lists that checked little or nothing
The zero rule only works if the regression-test list holds the repository's tests. In 11 of these 17 tasks the list was empty, so a patch could break existing behaviour and still score full marks. In the other 6 it checked only 3 to 519 tests. Fix 5 rebuilds all 17 lists from repeated runs on the untouched code, taking them from 784 regression tests to 165,757, and adds a checker that flags empty or inconsistent lists. In elixir the old list held one entry for the whole Mix app, so one correct fix that changes 3 tests in the app's deps_test.exs made that entry fail. Fix 5 sets the file aside, and elixir's golden solution scores again.
base_tests.json.gz as released and on Fix 5's branch.
Notes
Most lists were empty or thin because the grader could not read the test run. The whole-suite run never finished in netty (a test that resolves real host names hangs offline) and stopped early in rails; v's own test runner broke its file paths and failed 1,089 of 1,424 files; nim's and symfony's results came in a form the grader's reader did not parse; and Maven stopped at the first module with a failing test, so keycloak's list covered 49 tests and dolphinscheduler's 109. Fix 5 repairs these test drivers so that each test reports its own result.
Each list was then re-recorded from repeated runs on the untouched code. Tests that did not pass in every run, or that an official fix is meant to change, are set aside instead of listed (148 in rails, 56 in nim), so the zero rule does not fire on them. The new checker, tasks/check_base_lists.py, flags empty lists, placeholder entries and counts that disagree with the task's metadata.
Stop false regression-test failures from scoring tasks 0
The zero rule counts one regression-test failure the same as four hundred, and it does not ask why the test failed. In 14 tasks a regression test can fail even when the patch is right. When the benchmark was renamed, the prefix its Julia driver puts on test names became swesweep/ (9 characters), but the grader still read each name from character 9. Julia counts from 1, so every name kept the slash and matched nothing on the regression-test list. Some of testify's test names contain memory addresses, which move when a fix changes the code size. In other tasks the failing test is unreliable (flaky), or a correct fix is meant to change it, or the task image no longer builds. Fix 6 corrects the names, masks the addresses, sets those tests aside and lets the images build again.
Notes
Fix 6 was graded on its own branch, without the other fixes; the fixes have not yet been graded together. A golden solution leaves out any official fix that does not apply on top of the others, so that fix's bug does not pass.
† On Fix 6's branch tarantool had no regression-test failures in 4 of 4 gradings, but on one audit server another tarantool test failed in every run, so in at least one grading environment the task still scores 0. In Fix 6's grading the golden solution leaves out one official fix, and 4 bugs do not pass because their compiled hidden tests need Fix 1's rebuild. On Fix 1's own branch it passes 39 of 40 (Figure 1); one of the 39 is tarantool-c2e562efa2e4, a bug an empty patch also passes and Fix 3 removes.
The published leaderboard shows non-zero scores on both Julia tasks, so it was probably graded before the rename, when the prefix was pandora/ (8 characters) and reading from character 9 was right. If so, the slip affects people who run the released code, not the paper's numbers.
Thanks
We thank the SWE-sweep authors for releasing the full benchmark, with every task's official fixes, hidden tests and the grader, which made this work possible.