Why OpenAI looks weak on WANDR
WANDR is a benchmark released by Perplexity for studying how well AI systems perform web research. The WANDR paper authors make a convincing case that web research is far from solved and their work features on the recent Claude Opus 5.5 launch page. However, we noticed that the openai trials were given a legacy configuration which may have been unfairly limiting.
OpenAI's effort values on WANDR are outliers
Along with accuracy, WANDR reports what each provider costs to run.
| Provider | Soft F1 | Cost per task | Median minutes | Input tokens (M) |
|---|---|---|---|---|
| Perplexity | 0.363 | $5.20 | 14.9 | 3.79 |
| Anthropic | 0.249 | $46.43 | 73.7 | 51.75 |
| OpenAI | 0.121 | $0.50 | 8.6 | 0.13 |
| Exa | 0.070 | $0.50 | 5.3 | – |
| Parallel | 0.069 | $1.19 | 30.0 | – |
| Gemini | 0.055 | $15.24 | 22.3 | 2.96 |
OpenAI stands out among the leading providers. It ranks 3/6 on accuracy, yet it ties for the lowest cost and input tokens. At an average of 0.13 million input tokens per task this is 23 times fewer than Gemini's 2.96 million, 29 times fewer than Perplexity's 3.79 million and 398 times fewer than Anthropic's 51.75 million. Such low input token counts seem significant on a web research benchmark, where reading web pages is key.
OpenAI models can't read search results
When we inspect the released code carefully we find that under the WANDR configuration all providers are given the ability to use search tools and read search results except Openai. WANDR gives OpenAI's models two tools. One is web_search_preview, a legacy search tool which only provides snippets of search results, not the whole document. OpenAI's documentation recommends the newer web_search and had been doing so for 9 months prior to the publication of WANDR. The other tool is a code interpreter inside a sandbox. Under its default settings, OpenAI's API lets this sandbox reach a list of allowed sites, and its newer hosted environments can have full internet access. However, the WANDR authors chose to prevent all internet access to their version of Openai's sandbox. As a consequence, OpenAI models can search via web_search_preview but cannot actually ever read search results; the code tool has no internet access, and web_search_preview only returns result metadata and snippets.
This seems problematic a priori because every WANDR task asks for verbatim quotes from each cited page. One task, for example, asks for "official-document provenance for 175+ airworthiness directive rulemaking documents" from the Federal Register, with quotes showing each required fact. That literally cannot be done without access to full search results.
OpenAI's limitations are significant
To test the impact of this configuration, we first ran this setup exactly as shipped, on 25 (5%) of WANDR's 500 tasks selected at random. We ran every OpenAI experiment with GPT-6.1-Sol, one of OpenAI's newest models at the time of writing. The original setup behaved much as it does in the paper, at 0.16 million input tokens and 9.4 minutes per task. Across all 25 tasks the model made 380 searches and opened 0 pages, relying on the web_search_preview snippets alone.
We then ran the official WANDR Perplexity setup (except using gpt-6.1-sol as the base rather than 5.5) on the same 25 tasks. It used 9.1 million input tokens and took 22 minutes per task. These higher token counts are expected since the WANDR configuration runs Perplexity's full agent product which can fetch full result pages. This indicates that the unique tool configuration chosen for Openai models by the WANDR authors may be having a significant impact on model behavior.
OpenAI models perform better with Codex
Next we ran GPT-6.1-Sol in OpenAI's Codex agent on the same 25 tasks. We equipped it with the current web search tool (web_search) and a shell that can reach the internet. Under these conditions we achieve effort values closer to the other 3 leading providers, with 25 million input tokens on average and a median of 36 minutes per task. In addition, with codex the F1 score increases to 0.63, nearly performing as well as Perplexity
This indicates that the outlier behavior of OpenAI on WANDR was heavily influenced by the limited harness it was given rather than by the Openai models themselves, and that the choice of harness has a very large impact on both effort and performance.
Methodological Notes
All our scores are higher than those reported in the WANDR paper. This could be due to our use of GPT-6.1-Sol rather than GPT-5.5, or drift in the LLM judge models between our experiments and the original publication and the fact we chose only 5% of the total task set. Due to these limitations it may not be valid to compare our results with the original WANDR results (though we will note that they remain in the same ballpark), but this should not impact the validity of comparisons between our own experiments.
It is also important to note that giving OpenAI its usual, full agent does not confer an unfair advantage. WANDR already runs Anthropic models using a managed agent with unrestricted internet access and Gemini's via Google's Deep Research agent with a tool for reading web pages while Perplexity, Exa and Parallel run their own research products. OpenAI's was the only entry relegated to a limited harness.
Other potential issues
Along the way we found a few other potential issues which we will briefly enumerate here:
- A submission can include its own copy of each cited page, and the grader shows that copy to the judge in place of the real page. In a brief experiment we performed, fifty invented rows all passed, while ten matching rows without the copy all failed.
- One task,
container-shipping-providers, asks for 80 answers in total across four shipping stages, and its grader's own notes say the same, but the grader counts 80 in each stage. Fixing that count increased the score for a generated Openai submission from 0.40 to about 0.98 F1 with the same judge verdicts. - Three Forbes tasks give the judge a hidden list of "known errors" that overrides what the cited page says, and some entries on it are wrong. For instance one says that Mark Bradford did not grow up in a boarding house, even though which he described growing up in a boarding house during an interview.
We note these as possible areas of improvement in the benchmark, an in-depth discussion would exceed the scope of this article.
Closing remarks
We believe we have made a convincing case that WANDR's configuration for OpenAI was partly the cause of its outlier performance. We are also very grateful that the WANDR authors chose to expose their full benchmark to public scrutiny in the first place. Lately this level of commitment to transparency has become rare among benchmark authors, most of whom fail to release any tasks to the public, which makes it difficult to determine if many private benchmark leaderboards are subject to issues like those discussed here.
Contact via @lewingtonpitsos.