返回
原文
Chubby♨️@kimmonismus
41AI 编辑部评分，满分 100
2026-08-02 16:17· 24分钟前
跳到正文
AI 摘要

Karpathy 提出用《指环王》首段测试 LLM 长程生成能力：给 Opus 5 100 万 token 预算（约 $10），它耗时约 2 小时写出 5500 行 Three.js 代码，程序化渲染整个故事。该测试考验模型跨数千行代码的连贯性与自我审查能力，但暴露了 LLM 无法高效感知视频或游戏内反馈的短板。

As models improve， the benchmarks we use to evaluate them have to evolve too.

"Create an SVG of a pelican riding a bicycle" was once surprisingly helpful. Today， frontier models routinely produce convincing results， so the test tells us increasingly little about where the actual frontier is.

Karpathys experiment is a fascinating attempt to push the benchmark forward： give Opus 5 the opening of The Lord of the Rings， a massive token budget， and two hours to turn it into an interactive Three.js world. And voila！

This tests far more than one-shot generation： The model has to maintain coherence over thousands of lines of code， translate prose into a spatial system， coordinate objects and animations， and continuously inspect its own work.

Really love where the testing / benchmarks are moving to！

Love to see GPT-5.6 and the new DeepSeek flash on this one.

Andrej KarpathyWe're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested wh...
Anthropic多模态评测/基准
在 X 查看原推导出 Markdown
返回精选列表
41导出 Markdown
Chubby♨️ · @kimmonismus · X·2026-08-02 16:17·24分钟前
在 X 看原推· x.com
AI 摘要

Karpathy 提出用《指环王》首段测试 LLM 长程生成能力：给 Opus 5 100 万 token 预算（约 $10），它耗时约 2 小时写出 5500 行 Three.js 代码，程序化渲染整个故事。该测试考验模型跨数千行代码的连贯性与自我审查能力，但暴露了 LLM 无法高效感知视频或游戏内反馈的短板。

As models improve， the benchmarks we use to evaluate them have to evolve too.

"Create an SVG of a pelican riding a bicycle" was once surprisingly helpful. Today， frontier models routinely produce convincing results， so the test tells us increasingly little about where the actual frontier is.

Karpathys experiment is a fascinating attempt to push the benchmark forward： give Opus 5 the opening of The Lord of the Rings， a massive token budget， and two hours to turn it into an interactive Three.js world. And voila！

This tests far more than one-shot generation： The model has to maintain coherence over thousands of lines of code， translate prose into a spatial system， coordinate objects and animations， and continuously inspect its own work.

Really love where the testing / benchmarks are moving to！

Love to see GPT-5.6 and the new DeepSeek flash on this one.

Andrej KarpathyWe're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested wh...
Anthropic多模态评测/基准
在 X 查看原推x.com