Both these submissions score a full 1.0. We consider this a major defect because an agent could plausibly forget to create the rotation effect altogether and still receive full marks. In this case the task does not test what it is supposed to.

Another major issues appears in task 61 where the agent is supposed to apply a color grade to a snowy image. The verifier is very forgiving however, and other transforms which generate very incorrect images, such as a per-channel affine color match followed by a blur also score 1.0.

An example of a minor issue can be found in task 50 where an agent can receive full credit for an edited podcast which accidentally mutes one of all of the tracks. This is not ideal, but since agents are highly unlikely to do this in practice we don't think this will seriously effect the task's usefulness.

Our hope is that researchers using this benchmark can use our findings to get even more out of the excellent tool that is OSWorld 2.0.

NOTE: we are auditing release v2026.08.08 which is the most recent release at time of writing (10/09/2026), though we believe the OSWorld authors are working on a new release.

007, 017, 020, 037, 043, 045, 046, 051, 056, 061, 087, 088, 089, 091, 092, 093, 095, 102

001, 002, 005, 008, 018, 023, 026, 038, 042, 044, 047, 049, 050, 060, 062, 063, 070, 071, 084, 085, 096, 098, 106, 107, 108

001, 002, 005, 007, 008, 017, 018, 020, 023, 026, 037, 038, 042, 043, 044, 045, 046, 047, 049, 050, 051, 056, 060, 061, 062, 063, 070, 071, 084, 085, 087, 088, 089, 091, 092, 093, 095, 096, 098, 102, 106, 107, 108