The Gaming Techniques
The author dug deep into the benchmark's questions and scoring mechanism, and the problems listed are tragicomic: the format of some test cases' answers can be guessed from context; training-data contamination means the model may have memorized the original questions; the scoring script only verifies the final output, so a messy intermediate process still passes; and the tasks' success criteria are loosely defined, so borderline answers can score. In the face of these loopholes, the relationship between a high score on the leaderboard and real work ability is far weaker than vendors advertise.
The Benchmark Trust Crisis
This isn't the scandal of one benchmark, but a structural disease of the whole evaluation system: once a benchmark becomes famous, it becomes every vendor's optimization target, Goodhart's law kicks in, and the decoupling of score from ability is only a matter of time. Academia's response is to keep pushing new benchmarks, private test sets, and dynamic question banks, but as long as launch events still tell stories with benchmarks, the arms race can't stop. For ordinary users, the actionable takeaway is the same one: treat public benchmarks as clues rather than conclusions, and if you really need to select, build your own private eval set of twenty or thirty questions and run it—more reliable than looking at the leaderboard.
via: Hacker News