OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery
Original reporting by arXiv (cs.AI)

OpenFrontiers is a novel, cross-disciplinary resource designed to rigorously evaluate large language models' (LLMs) capacity for research-level problem solving, focusing on distinguishing between true discovery and information retrieval. While LLMs demonstrate increasing prowess in complex tasks, a comprehensive method to assess their ability to independently *solve* novel research problems – rather than merely recall existing information – has been lacking. OpenFrontiers fills this critical gap by combining a structured corpus of over 12,000 open scientific problems with a specialized benchmark of 200 recently resolved mathematical problems. This expansive resource provides a robust framework to measure an LLM's capacity for 'rediscovery' – arriving at a solution independently – versus 'retrieval' – efficiently finding existing answers, a key distinction for advancing AI.
Measuring True Discovery
For evaluation, a meticulously curated subset of 50 high-quality mathematical problems (HQ-50) was established, with each problem formally stated in Lean 4 and verified by independent reviewers. Seven LLM agents underwent stringent testing across various conditions: 'closed-book' (without external knowledge), 'frozen-retrieval' (limited to a pre-defined knowledge base), and 'live-web' access. The protocol further included two independent checkers validating formal proofs in an isolated sandbox, ensuring unbiased assessment. The compelling results revealed that genuine closed-book rediscovery of solutions proved remarkably rare, accounting for at most 7 of 40 resolved problems. In stark contrast, success in live-web conditions predominantly reflected the models’ ability to retrieve published solutions. OpenFrontiers thus provides critical, unprecedented insight into the current state of AI's research capabilities, emphasizing the fundamental difference between accessing and generating knowledge and laying a foundation for future progress in scientific problem-solving.
OpenFrontiers represents a pivotal stride in establishing a robust and standardized framework for evaluating the advanced reasoning capabilities of large language models. By meticulously curating a diverse corpus of open research problems and a formally validated benchmark of resolved mathematical challenges, it provides an unprecedented lens through which to gauge genuine scientific problem-solving. The initial evaluations, revealing that LLMs predominantly succeed by retrieving existing solutions rather than demonstrating true closed-book rediscovery, offer a critical insight. This finding starkly highlights the current boundary between sophisticated information synthesis and the independent generation of novel knowledge, a crucial distinction for the field.
Charting new frontiers
The implications of OpenFrontiers stretch beyond current performance assessments, promising to fundamentally reorient the trajectory of AI research. This resource establishes a concrete, measurable goal for developing models capable of original thought and independent scientific discovery. By offering a granular understanding of where LLMs fall short in complex problem-solving, it provides invaluable guidance for training future systems to transcend mere information retrieval. OpenFrontiers is poised to become an indispensable tool for accelerating progress towards AI that can actively participate in scientific breakthroughs, offering a quantifiable path toward machines that can not only understand but also expand the boundaries of human knowledge across myriad disciplines. Its success will be measured by its ability to catalyze the development of AIs that genuinely contribute to solving humanity's most challenging open problems.
Frequently asked questions
- What is OpenFrontiers and how does it help evaluate AI on complex research challenges?
- OpenFrontiers is a cross-disciplinary resource designed to evaluate large language models (LLMs) on research-level problem-solving. It combines a corpus of over 12,000 open research problems with a benchmark of 200 recently resolved and still-open mathematics problems. This platform provides a standardized way to measure LLM capabilities in rediscovering solutions and tackling novel scientific challenges, addressing the current lack of comprehensive problem collections and robust evaluation protocols for AI in research.
- How does OpenFrontiers assess large language models' ability to solve research-level mathematical problems?
- OpenFrontiers employs a rigorous evaluation protocol to assess LLMs. Formal proofs generated by LLM agents are graded by two independent checkers within an isolated sandbox environment. Tests are conducted under three conditions: closed-book (no external information), frozen-retrieval (access to a fixed dataset), and live-web (internet access). This setup helps distinguish genuine problem-solving rediscovery from mere information retrieval, providing insights into AI's deeper reasoning capabilities for complex mathematical challenges.
- What were the main findings when large language models were evaluated using OpenFrontiers' benchmark?
- Evaluations using OpenFrontiers revealed that large language models rarely achieve true rediscovery of solutions under closed-book conditions, succeeding in only a small fraction of resolved problems. Their success in live-web scenarios primarily stemmed from retrieving already published solutions rather than independently generating novel proofs. This indicates that while LLMs can effectively access and utilize existing knowledge, their current capacity for independent, original research-level problem-solving, particularly in mathematics, is limited.