Why it is contested rather than merely open
Some hold that this is unsolvable in principle — that Goodhart’s law applies to any published measure, and that the correct response is a rotating private evaluation regime rather than a better metric. Others hold that a measure defined over task-acquisition cost rather than task performance would be robust, because optimising against it is indistinguishable from genuinely improving.
The archive files this as contested because both positions have serious proponents and the disagreement is substantive, not terminological.
What would count as progress
- A measure with a stated invariance property, plus evidence it survives direct optimisation.
- A formal argument that no such measure exists, which would settle it the other way.