Fresh data is finite.
The next scaling law finds what truly limits learning.
Paper:
Bridging Compute- and Data-Optimal Pretraining
(1/N) 🧵 Chinchilla assumes you'll never run out of fresh data. That era is ending! Compute keeps growing exponentially, but high-quality tokens don't. So what'...