MathArena:
Evaluating LLMs on Uncontaminated Math Competitions

MathArena is a platform for evaluation of LLMs on the latest math competitions and olympiads. Our mission is rigorous assessment of the reasoning and generalization capabilities of LLMs on new math problems which the models have not seen during training. To ensure a fair and uncontaminated evaluation, we exclusively test models on competitions that took place after their release, avoiding retroactive assessments on potentially leaked or pre-trained material. By performing standardized evaluation we ensure model scores are actually comparable and are not dependent on the specific evaluation setup of the model provider.

To show the model performance, we publish a leaderboard for each competition showing the scores of different models individual problems. Additionally, we will include a main table that includes model performance on all competitions. To evaluate performance, we run each model 4 times on each problem, computing the average score and the cost of the model (in USD) across all runs.

We open sourced our evaluation code at: https://github.com/eth-sri/matharena.