AI Benchmarks Face a Crisis: Experts Push for Practical, Physics-Based Testing
Moritz Hardt argues that traditional AI benchmarks are fundamentally broken because models can 'game' them, leading to a crisis where rankings no longer reflect true capability. In response, researchers are developing new evaluation methods like 'Artifact Arena,' which tests AI by having them design robots to fight in virtual physics simulations, and 'The Agent Company,' which assigns AI virtual jobs in simulated firms. These shifts aim to measure practical task completion rather than fixed quiz scores, addressing concerns that current metrics are easily gamed and do not reflect real-world performance.