Benchmark Research SWE-sweep

Parsewave / Benchmark Research / SWE-sweep / Method

Method and sources

How we graded

The report's glossary defines the terms used here. SWE-sweep's README states its scoring, including the zero rule, like this (it calls bugs subtasks):

if any_new_failures(base_commit_results):  task_score = 0else:  task_score = sum(passed_subtasks) / len(subtasks)

Figure 1 traces one example through the grader's four steps: matplotlib's golden solution passes the hidden tests of 76 of its 78 bugs, but one regression test fails, so the task scores 0 and none of the 76 counts.

    Figure 1. SWE-sweep's grading steps, traced with matplotlib's official fixes applied together, and the fixes that act at each step. Sources: README; Fix 6's validation.

    We graded the golden solution and the empty patch for every task with SWE-sweep's own grader on Harbor, the harness it ships with; neither involves a model. Both ran as released and on each fix's branch, and results that varied between runs were graded again. Before writing the fixes, three of us audited 92 of the 100 tasks as released (17 to 19 October 2026).

    The six fixes

    Table 1. What each fix changes and what its validation gradings show; values left of an arrow are without the fix.
    FixWhat it changesTasksResult
    Fix 1
    #54    		 Stops the grader from undoing correct fixes as test edits and from running stale compiled tests. Repairs official fixes that broke the build, and adds sweep validate, a command that grades a task's empty patch and official fixes. 16 tasks The golden solution passes more bugs, and scores, in 5 tasks (trino 5 → 41 of 43).
    Fix 2
    #57    		 Pins three task images to dependency versions that do not contain the official fixes. 3 tasks The golden solution's score is unchanged (syn 13 of 13, symfony 148 of 149).
    Fix 3
    #56    		 Removes the 10 bugs an empty patch passes. 9 tasks In the 4 tasks graded with these bugs present, the empty patch passes at least one of them in each. With Fix 3 it passes no bug in any of the 9 tasks.
    Fix 4
    #56    		 Removes 126 duplicate bugs. With Fix 3, SWE-sweep goes from 4,068 bugs to 3,932. 16 tasks Checked together with Fix 3, on one branch.
    Fix 5
    #55    		 Rebuilds 17 empty or thin regression-test lists from repeated runs on the untouched code (784 → 165,757 tests) and adds a list checker. 17 tasks The empty patch passes every new list. The golden solution scores again in elixir, and now scores 0 in nim and rails.
    Fix 6
    #53    		 Reads Julia test names correctly, masks memory addresses in test names, sets aside unreliable tests and tests a correct fix is meant to change, and repairs two task images. 14 tasks The golden solution scores again in 7 tasks. The empty patch's regression-test failures in 3 tasks are gone, and babel builds.

    For Fix 1, the values left of the arrow come from the released grader run on images built from Fix 1's branch. Each task's gradings are in its record in the task table in the report.

    What the fixes mean for agents' scores

    We re-scored 120 of our own model gradings with the regression-test lists as Fix 6 edits them and without the bugs that Fixes 3 and 4 remove. No run gains or loses more than 3 counted bugs, so the effect on these scores is modest. The re-scoring does not re-run tests, so it leaves out the fixes' other changes.

    Limits

    • Each fix was graded on its own branch, except Fixes 3 and 4, which were graded together on one branch. The six have not yet been graded together (Fixes 1 and 6 edit the same grader file).
    • Besides the 8 tasks where the fixes make the golden solution score again, it still scores 0 in at least 18 tasks, for causes outside these fixes. For most of them, that 0 comes only from our audit, which is a working draft.

    Sources

    • SWE-sweep as released. facebookresearch/swe-sweep at commit 5968c69, released 1 October 2026.
    • The paper. SWE-sweep: Can Agents Autonomously Find and Fix Bugs?, Kilian Lieret and co-authors, 1 October 2026.
    • The leaderboard. swesweep.com, updated 8 October 2026 and unchanged on 3 November. Astra 6's 9.5% is 363 of 3,828 bugs on 93 tasks.
    • Our fixes. Pull requests #53 to #57 on SWE-sweep's tracker, opened 6 November 2026, each with its Harbor validation gradings in the description.
    • Our gradings. 120 model gradings, and our audit of 92 tasks (17 to 19 October 2026, working draft).

    The script that builds these pages' data checks their numbers against the source data (103 checks) and stops if one fails.

    Parsewave Benchmark Research · 3 November 2026

    ← Back to the report