Rohan Paul@rohanpaul_ai · X陶哲轩指出，大型语言模型（LLMs）的训练和运行主要基于线性代数、矩阵乘法和微积分等简单数学，本科生即可掌握。然而，核心谜团在于LLMs为何在某些任务上表现卓越，却在其他任务上失败，且无法提前预测。这源于现实世界数据的性质：它介于完全噪声和完全结构化之间，而数学对此中间状态的理解薄弱，类似于物理学在原子和连续介质之间的介观尺度困境。因此，尽管我们能描述LLMs的机制，却无法解释其能力跳跃或提供可靠的任务级预测。简单机制与难以预测行为之间的不匹配，构成了当前研究的核心难题。
Terence Tao says the math behind today's LLMs is actually simple. Training and running them mostly uses linear algebra， matrix multiplication， and a bit of calculus， material an undergraduate can handle. We understand how to build and operate these models.
The real mystery is why they work so well on some tasks and fail on others， and why we cannot predict that in advance. We lack good rules for forecasting performance across tasks， so progress is largely empirical.
A key reason is the nature of real-world data. Pure noise is well understood， perfectly structured data is well understood， but natural text sits in between， partly structured and partly random. Mathematics for that middle regime is thin， similar to how physics struggles at meso-scales between atoms and continua.