New Harvard+Chicago study.

A year of production data shows that efficient LLM serving depends less on a single clever scheduler than on understanding how traffic changes and repeats.

The researchers studied 6.12B requests across 9,174 models for 1 year and found repeatable patterns that serving systems can use.

Users often come back to the same model with the same growing context. 99% of the reuse they measured came from requests returning within 15 minutes.

So the GPU may already have much of the previous prompt computed and can reuse it instead of doing that work again.

A load balancer that only spreads traffic evenly can weaken this advantage by sending the next turn to another GPU.

The traffic itself also changes over months: popular models turn over, outputs get shorter, and newer users tend to send larger inputs.

That makes short or synthetic workload benchmarks a poor picture of long-running production.

Overall, routing, caching, and capacity planning should use the history and structure of real traffic, rather than treating every request as an isolated job.

– arxiv. org/abs/2608.13573

Title: "A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing"