Given Only a Paper's Pre-Publication Bibliography, Frontier Models Recover Its Original Idea 3–15% of the Time

A new benchmark, Reconstruction (arXiv:2608.16645), hands a model only the literature that existed before a paper was published — with author names and metadata stripped — and asks what that paper proposed. Across six scientific domains and 643 papers, seven frontier models reach Match rates of roughly 3–15%; a reference-only multi-agent pipeline with no web search pushes that to about 42%. The authors note that many "AI scientist" evaluations let models see full paper text, author information or post-publication signals, so high scores under those conditions cannot be read as genuine inferential capacity.

The Task Design Matters More Than the Score

The question is plain: put the literature that existed before a paper in front of you, and work out what that paper went on to propose. For each seed paper the authors build a blind reference context containing only work published strictly before the seed's publication date. During generation the model never sees the seed's title or abstract; afterwards an LLM judge compares the model's hypotheses against the withheld ground-truth idea, and the primary metric is Match rate. Anti-leakage runs on three layers: a temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies. The anonymous IDs specifically close one shortcut — recognizing a research group's trajectory and extrapolating the next step. The result across six domains and 643 papers: seven frontier models land at roughly 3–15%. The authors then test a reference-only multi-agent pipeline with no external web search, combining cross-model review with a Swiss tournament over aligned hypothesis slots, which raises Match rates substantially over the best single model. Press coverage puts that figure near 42%, about a 2.4× lift; the paper itself frames it as an observed association still pending candidate-matched and equal-compute controls.

What It Actually Challenges Is the Evaluation Setup

The weight of this result is methodological. Plenty of products sell large models as hypothesis-generation engines on evidence drawn from evaluations that let the model see full paper text, author information or post-publication signals. Reconstruction excludes all of it, so scores obtained under looser conditions cannot be assumed to reflect reasoning. The same model, under a different information constraint, can differ by an order of magnitude — and that fact is more worth remembering than whether the number is 3% or 15%.

Limits

The authors list their own: coverage spans only six domains (machine learning plus five Nature-family areas), and this draft is positioned as an arXiv timestamp of the protocol, the anti-leakage design and current results. Reconstruction is also an intermediate step — the stated goal is a Generation mode where models propose forward-looking, feasible directions, at which point there is no answer key to compare against. One line is directly usable when evaluating tools: if a research-oriented AI product shows you benchmarks where the model had access to full paper text or author information, that score is measuring retrieval and restatement, not whether it can find your next research question.

via: arXiv:2608.16645, full text, Tech Times