RightSignal
1 of 6

Your RightSignal

AI

Higher-Scoring Coding Models Did Not Get More Work Done

Yining Hua · Submitted 17 Nov 2026
Preprint · not peer reviewed

When a US export directive cut off access to Claude Fable 5 for 19 days, most users switched to a lower-scoring model. The author tracked GitHub pull requests of 297 such users against 8,908 developers who never switched, and repeated the check across five model upgrades.

Key finding: Access to the higher-scoring model raised the 7-day pull request merge rate by only 1.4 percentage points, which was not statistically significant. The five upgrades showed no significant gains either.

Why it matters: Benchmark scores are how coding models are sold. This natural experiment suggests they may not track how much real work gets done.