New Google Deployment Paper shows instead of forcing an LLM to reread its entire context for every token, let the model choose what to read

cutting attention work by 52.0%

Today’s models often reread the entire stored context for every new token, even when only a small section matters.

Declarative Attention changes that: the model itself says which part of the context it needs, and the inference engine skips the rest.

No separate scorer is needed to search the whole context first.

Across 15 long-context tasks, attention work fell by 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B, while accuracy dropped by 1.27 and 2.75 percentage points.

Larger models handled this trade-off better, suggesting the approach could improve further with training.