The Disconnect Between AI Benchmarks and Math Research

Current AI systems boast impressive scores on mathematical benchmarks. Yet when confronted with the questions mathematicians actually ask in their daily research, these same systems often struggle, and don't even realize they are struggling.

As explored in "The Cultural Divide between Mathematics and AI", there's a significant gap between the AI and Math worlds. Thus I have been trying to build tools to help with research, especially around paper search and AI question-answering.

Special thanks to everyone who submitted questions and provided feedback on answers! With well over a thousand questions asked in just the past week, we are seeing some interesting patterns and I wanted to present some preliminary results.

For those new here, please sign up, join our chat, or reach out with any feedback.

The Unique Challenge of Mathematical Research Questions

Mathematical research presents some fundamental challenges making it a particularly stringent test of genuine intelligence rather than pattern recognition.

1. The Abstraction Gap: In mathematics, the tools you need to solve a problem often look entirely different from the original problem, so semantic search is insufficient.
   1. A request to "build an optimal order router" requires understanding Multi-Armed Bandit algorithms.
   2. The breakthrough in "ranking web pages" came through eigenvalue analysis of adjacency graphs.
   3. Solving "high-dimensional sphere packing" requires mastery of Leech Lattices and Modular Functions.
   4. Finding when to "stop evaluating options" leads to the Secretary Problem.
2. The Deep Historical Context: Mathematics is cumulative knowledge spanning centuries:
   1. Sometimes you have to track a thread back a hundred years (and different languages) before finding the right reference.
   2. Notation evolves dramatically between eras and subfields, requiring translation between mathematical "dialects".
   3. The cost of missing a key historical reference isn't merely inefficiency—it can represent years of redundant work or false directions.
3. The Critical Nuance Factor: Mathematics is exquisitely sensitive to small variations. Consider these seemingly similar equations, each requiring entirely different solution approaches:
   1. x²-y²=n (straightforward factorization)
   2. x²+y²=n (requires Gaussian integer theory and Cornacchia's Algorithm)
   3. x²-2y²=n (requires Pell's equation techniques and continued fractions)
   4. x²-4y²=n (returns to straightforward factorization)
   5. x²-y³=n (involves Mordell curves and Catalan's conjecture)
4. The Benchmark Reality Gap: Current mathematical benchmarks fundamentally misrepresent research mathematics:
   1. Benchmarks typically test algorithmic solving of known problem types with established solution methods.
   2. Actual research questions demand identifying which area of mathematics is relevant in the first place.
   3. Research advances often come from seeing unexpected connections across disparate mathematical domains.
   4. The "known-unknown" nature of benchmark problems versus the "unknown-unknown" nature of research creates a fundamental evaluation mismatch.

Real-World Examples: Where AI Systems Struggle

Let's examine a few examples that illustrate the challenges AI systems face.

Question: Does there exists a divergent series which converges on every subset of N with arithmetic density 0?

This question came up in a Reddit thread, and a user quickly explained that there isn't, and gave an algorithm for iteratively constructing such a sequence.

Sky-T1 and O1 correctly said there isn't, while all other models claimed there is and gave a faulty construction.

This demonstrates how even relatively straightforward questions can reveal gaps in AI systems' reasoning.

Question: Can nowhere vanishing entire functions have a non-trivial linear dependency?

This was resolved in the negative by Borel in 1897 and there's a great explanation in Lang's Introduction to Complex Hyperbolic Space and in some books on Nevanlinna Theory such as by Rubel and by Kodaira.

Interestingly, none of the AI systems (including Sugaku) provided the fully correct answer, although Sky-T1 and GPT-4.5 mentioned relevant concepts like Wronskians and growth rates that appear in the proof. This highlights how even well-established mathematical results can be challenging for AI systems when they involve specialized areas of mathematics.

Question: What are the latest results on the Selberg Class Degree Conjecture? or Question: What is the Selberg Class Degree Conjecture and what is currently known about it?

This was part of my research area, and is the conjecture that a certain quantity (the degree) is always an integer.

Most AI systems provided partially correct but flawed answers, with common problems including:

• Incorrectly focusing only on classifying elements of natural degree while missing the critical question of whether non-integer degrees (such as 1<d<2) can exist
• Making false claims about the state of research (such as claiming degree 2 has been fully classified)
• Inventing non-existent papers or results
• Fabricating computational approaches or evidence
• Creating fictional connections to other mathematical problems

Sugaku's answer correctly summarized the known results, including the significant 2011 proof by Kaczorowski and Perelli that the degree cannot be strictly between 1 and 2, demonstrating a more accurate grasp of the research landscape.

Sugaku's Approach: Building for Real Mathematical Understanding

The above challenges are especially poignant in mathematical research questions, but also show up in many other intellectual endeavors. I'm building Sugaku to directly address these challenges:

1. Reference-aware architecture: Designed to efficiently locate and integrate relevant mathematical literature spanning centuries
2. Concept mapping across notation systems: Recognizes equivalent concepts expressed in different notational conventions
3. Fine-tuned on actual research questions: Trained on the types of questions mathematicians genuinely ask, not artificial benchmarks
4. Expert-guided development: Incorporating direct feedback from research mathematicians across specialties
5. Mathematical consistency verification: Built-in mechanisms to check logical consistency and proof validity, and acknowledge uncertainty.

You can go to sugaku.net to test this out yourself with your own questions, browse and investigate papers, you can sign up to get a custom newsfeed of papers, track your projects and collaborators, and you can generate new ideas.

Whether or not you currently use AI tools to help with your research I would love your feedback. You can join our chat, or reach out with any feedback.