返回
原文
elvis@omarsar0
62AI 编辑部评分，满分 100
2026-08-04 07:00· 17分钟前
跳到正文
AI 摘要

TokTier 提出有状态 tokenization 服务，解决智能体场景下 token 边界漂移导致的缓存失效问题。在 153,951 次真实调用中，tokenization 耗时占首 token 延迟的 64%；该方案通过增量修复，将延迟降低 16-34%，速度最高提升 437 倍。

TokTier makes tokenization stateful for agentic serving.

Across 153，951 real agent calls with a 94.1% prompt-cache hit rate， tokenization eats up to 64% of time to first token. Coding agents resubmit a long transcript after every tool result， and even a short append can shift token boundaries at the tail of the previous sequence， so nothing gets reused.

TokTier is a stateful tokenization service. For a session continuation it re-tokenizes a small window around the append， runs a stable-boundary check， and splices only when that check passes. When there is no reusable prefix it runs exact pre-tokenization and BPE on a GPU.

Emitted token IDs always match full reference tokenization. Differential campaigns across 17 tokenizer families covered 1.5e10 split checks with zero divergence.

Incremental repair takes 0.5 to 1.1 ms from 100K to 3M characters， up to 437x faster than HuggingFace. Under vLLM， median time to first token drops 16 to 34%. Four repair cores plus one GPU sustain 1，821 requests per second where a 16-core stateless front end saturates at 40.

Paper： https://arxiv.org/abs/2607.29678

Track more trending AI papers in our academy： https://academy.dair.ai/

产品更新推理部署/工程
在 X 查看原推导出 Markdown
返回
62导出 Markdown
elvis · @omarsar0 · X·2026-08-04 07:00·17分钟前
在 X 看原推· x.com
AI 摘要

TokTier 提出有状态 tokenization 服务，解决智能体场景下 token 边界漂移导致的缓存失效问题。在 153,951 次真实调用中，tokenization 耗时占首 token 延迟的 64%；该方案通过增量修复，将延迟降低 16-34%，速度最高提升 437 倍。

TokTier makes tokenization stateful for agentic serving.

Across 153，951 real agent calls with a 94.1% prompt-cache hit rate， tokenization eats up to 64% of time to first token. Coding agents resubmit a long transcript after every tool result， and even a short append can shift token boundaries at the tail of the previous sequence， so nothing gets reused.

TokTier is a stateful tokenization service. For a session continuation it re-tokenizes a small window around the append， runs a stable-boundary check， and splices only when that check passes. When there is no reusable prefix it runs exact pre-tokenization and BPE on a GPU.

Emitted token IDs always match full reference tokenization. Differential campaigns across 17 tokenizer families covered 1.5e10 split checks with zero divergence.

Incremental repair takes 0.5 to 1.1 ms from 100K to 3M characters， up to 437x faster than HuggingFace. Under vLLM， median time to first token drops 16 to 34%. Four repair cores plus one GPU sustain 1，821 requests per second where a 16-core stateless front end saturates at 40.

Paper： https://arxiv.org/abs/2607.29678

Track more trending AI papers in our academy： https://academy.dair.ai/

产品更新推理部署/工程
在 X 查看原推x.com