Great paper if you are building production-grade agents.

It's a good way to understand what is actually incurring costs in your agents, which involve more complex components and decisions.

This works finds that across ten instrumented agentic applications, non-LLM components dominate latency in five of them.

How so?

Sandbox working sets peak at 28 GB per session. Task latencies inside one application diverge by up to 32x across GPU-bound inference, memory-bound retrieval, and CPU-bound sandboxes. Production sessions hold state idle for minutes to hours between active steps.

Auxiliary LLM calls and tool schema overhead form a control-plane tax that crowds out productive compute.

Task-aware serving cuts latency 29 to 40%, state offloading cuts memory 4.6x, and tool-result caching removes 35.2% of redundant search calls.

Paper: https://arxiv.org/abs/2608.15127

Track more trending AI papers in our academy: https://academy.dair.ai/