AI
Many AI benchmarks overweight some difficulty levels and miss others
Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang · Submitted 21 Oct 2026 · 30 upvotes yesterday
Preprint · not peer reviewedTrending
The authors treat a benchmark as a spread of questions across difficulty levels. They define two problems: the spread does not match what the benchmark is meant to measure, or some difficulty ranges have no questions at all. They study 25 popular benchmarks across 7 domains.
Key finding: Many benchmarks lean too heavily on certain difficulty ranges, and others leave large gaps. A rewriting method called BenchFill reduces those gaps.
Why it matters: Leaderboard rankings may say less than they seem; this gives a way to check how much a benchmark can really tell you.