We evaluate paper retrieval on 23 complex queries submitted to Undermind, specified as natural language research goals. We sample randomly from non-sensitive anonymized queries, stratifying for diverse research fields. We filter out overly broad review queries.

For each goal, we construct a gold standard by scoring every paper any retrieval system retrieved, using GPT-5.6 Sol (xhigh) as a judge. The judge assesses relevance from each paper's abstract. Thus, no retrieval system is disadvantaged: every retrieved paper gets a chance to enter the gold standard. A paper enters the gold standard when its score reaches the rubric's threshold for precise relevance. Additionally, to ensure a high degree of completeness of the gold standard, we run several of the retrieval systems multiple times with high budgets to aggregate a comprehensive set of candidates.

We compare Undermind against frontier models running as search agents: GPT-5.6 in the Codex CLI harness and Claude Opus 5 in Claude Code. Each agent has web search and shell access, so it can reach public web APIs. Undermind runs the production deep search with an extended budget to trace the full ten-minute curve. A production search stops on its own after 2.9 minutes on average. Resubmitting a similar search continues from existing results, so users can reach the rest of the curve via multiple search submissions.

We ask each retrieval system to score its findings with the same relevance rubric the gold standard judge uses. Those scores order each retrieval system's returned list, and precision is measured over that order. Search time is wall-clock time from the start of each search. Retrieval systems report their findings in batches at periodic intervals, and for the web agents the timing of those reports depends on the prompt, so we interpolate recall and precision between reports.