Its getting hard to benchmark frontier agent performance on longer tasks. Repeated measurement is very expensive and there are differences between using models in harnesses versus via APIs.

I suspect benchmarks understate progress， they are built for models， not harnessed agents