This is illuminating, but it also raises a deeper question.
The old scaling paradigm: More compute during pretraining -> smarter model
The current scaling paradigm: More compute during inference -> better answer
The next scaling paradigm: ? -> ?
If small models + test-time scaling + tools are beginning to hit a ceiling, and the future shifts back toward much larger models that can solve more problems in a single forward pass, what exactly is the next scaling axis?
When language models first started using tools well, I was sympathetic to the narrative that instead of scaling up language models, all we needed was a strong e...