Epoch AI Launches FrontierMath AI Benchmark to Take a look at Capabilities of AI Fashions

Epoch AI, a California-based analysis institute launched a brand new synthetic intelligence (AI) benchmark final week. Dubbed FrontierMath, the brand new AI benchmark assessments massive language fashions (LLMs) on their functionality of reseasoning and mathematical problem-solving. The AI agency claims that current math benchmarks aren’t very helpful attributable to elements like information contamination and AI fashions scoring very excessive scores on them. Epoch AI claims that even the main LLMs have scored lower than two % on the brand new benchmark.

Epoch AI Launches FrontierMath Benchmark

In a publish on X (previously often known as Twitter), the AI agency defined that it collaborated with greater than 60 mathematicians to create a whole bunch of origins and unpublished math issues. Epoch AI claims that these questions would take even mathematicians hours to unravel. The explanation behind creating the brand new benchmark was cited as the restrictions with current benchmarks equivalent to GSM8K and MATH, the place AI fashions usually rating a excessive level.

The corporate claimed that the excessive scores achieved by LLMs are largely attributable to information contamination. This implies the questions in some way had been already fed into the AI fashions, leading to them simply fixing the questions.

FrontierMath solves the issue by together with new issues which can be distinctive and haven’t been revealed anyplace, mitigating the dangers related to information contamination. Additional, the benchmark contains a variety of questions together with computationally intensive issues in quantity concept, actual evaluation, and algebraic geometry, in addition to subjects equivalent to Zermelo–Fraenkel set concept. The AI agency says all of the questions are “guess proof”, that means they can’t be solved by chance with out sturdy reasoning.

Epoch AI highlighted that to measure AI’s aptitude, benchmarks ought to be created on artistic problem-solving the place the AI has to take care of reasoning over a number of steps. Notably, many trade veterans imagine that the prevailing benchmarks aren’t enough to accurately measure how superior an AI mannequin is.

Responding to the brand new benchmark in a publish, Noam Brown, an OpenAI researcher who was behind the corporate’s o1 mannequin welcomed the brand new benchmark and stated, “I really like seeing a brand new eval with such low cross charges for frontier fashions.”