New Alibaba ByteDance paper shows that AI agents can no longer be served like ordinary LLM requests.

Because most of the performance problem now sits across tools, memory, environments, and the model together.

Bottleneck can also move from the LLM to embeddings, databases, sandboxes, or network transfer as the request and deployment change.

This paper builds AgentSysBench around 10 agentic applications and finds that model inference is often no longer the main bottleneck.

They find task-aware serving cuts latency by 29–40%, communication-aware placement delivers up to a 4.5× speedup, state offloading cuts memory by 4.6×, and caching removes 35.2% of redundant search calls.

So optimizing tokens per second is no longer enough. Agent infrastructure has to schedule models, tools, memory, and communication as one workload.

– arxiv. org/abs/2608.15127

Title: "From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems"