Some architecture choices that make transformers cheaper or more stable also make them harder to extend to long context.

A model can look completely healthy during short-context pretraining and still be a bad candidate for long context later.

This paper tests 26 comparable 7B models while keeping the data, tokenizer, and context-extension recipe fixed.

The only differences are 4 architecture choices already used across Llama, Qwen, and Olmo: QK normalization, grouped-query attention, sliding-window attention, and shorter pretraining context.

Individually, most of them hurt only a little. Stack several together and the damage compounds.

On HELMET at 32K, scores ranged from 29.9 to 56.4, even though standard short-context evaluations barely separated the models.

One example is especially clear: adding sliding-window attention cost just 1.1 points without grouped-query attention, but about 9 points on average when grouped-query attention was already present.

More long-context training did not erase the gap either: the worst architecture after 50B extension tokens still trailed the Llama-like architecture after only 1B.

So long-context capability is partly decided before long-context training even begins. Architecture needs an early long-context stress test, not just good short-context loss.

– arxiv. org/abs/2608.10296

Title: "Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension"