Higher-Scoring Coding Models Did Not Get More Work Done
When a US export directive cut off access to Claude Fable 5 for 19 days, most users switched to a lower-scoring model. The author tracked GitHub pull requests of 297 such users against 8,908 developers who never switched, and repeated the check across five model upgrades.
Why it matters: Benchmark scores are how coding models are sold. This natural experiment suggests they may not track how much real work gets done.