Method and limits
Model: GPT-6.1-Sol at high reasoning effort in all of our runs. The paper used GPT-5.5, so our numbers sit next to the paper's for reference, not as a like-for-like comparison.
Grading: every number from our runs uses the same page loader, the software that reopens each cited page (ours, with Perplexity's fetching service as a fallback), the same judge (GPT-6 Luna, WANDR's current judge) and the same grader. That grader is WANDR's with seven fixes for problems we documented. Under WANDR's unmodified grader the original setup scores 0.246 F1 instead of 0.242. One exception: the page-copy test under "Other issues" ran earlier, with stand-in judges (GPT-5.5 and GPT-5.6 Terra). The judge model doesn't matter for that finding: the judge is shown the text the submission supplies, and a run with no model at all shows the same effect.
Model access: both OpenAI setups called GPT-6.1-Sol through OpenRouter, which passed every request to OpenAI; OpenRouter's records confirm that OpenAI served the model. For the original setup this required four transport changes:
- requests sent to OpenRouter's address instead of OpenAI's;
- no stored or background responses;
- a request timeout raised to the 16-hour task limit;
- the answer returned in the final message instead of a sandbox file.
The prompt, tools and limits are otherwise WANDR's.
Fixed setup: the evidence rules and quote checker are general and contain nothing task-specific. A gate inside the task container refuses any request for a model other than GPT-6.1-Sol, including from helper agents; none was refused. An earlier run of the same setup through a different proxy scored lower (0.28 F1). That route also gave the agent a different search tool (ChatGPT's own web.run rather than the API's web_search), and we could not confirm which model served its requests, so we don't use it.
Tasks: 25 of WANDR's 500, or 5%, chosen at random across WANDR's domains and depth levels. We reviewed every task we ran, and the problems under "Other potential issues" came from that review. With 25 tasks we don't estimate a benchmark-wide effect.
Perplexity runs: an answer counts only if Perplexity's response completed normally. Responses that failed on Perplexity's side, such as overloads and internal errors, were retried from scratch and not counted against WANDR's 2 restarts.
Runs: one run per task per setup. The judge is noisy. Grading the same answers twice flipped 15% of verdicts in our tests, and WANDR grades once.
Cost: the paper reports average solver cost. For our OpenAI runs we estimate it from each run's token counts at GPT-6.1-Sol's list prices ($2 per million input tokens, $0.10 per million cached input tokens, $10 per million output tokens), averaged over the 25 tasks; this gives $8.15 for the fixed setup, with a median of $4.27, since one very large task pulls the average up. It leaves out OpenAI's higher rate for prompts above 272,000 tokens, so the true cost may be somewhat higher. Perplexity's figure is the cost it reported for completed responses; failed responses were billed as well.
Tokens: input tokens include cached and repeated context, so they measure the work done per task, not the amount of unique text read.
We make no claim about intent. The paper discloses its setups and raises the shared-fetcher question itself; we measured how much the results depend on them.
Figures: drawn from the numbers above by scripts/wandr-figures/make_figures.py in this site's repository, which lists the source of each value. Evidence: all runs, traces, grades and analysis code are available on request.