TileMix:面向 LLM 推理加速的以 Tile 为中心的混合精度注意力机制

HuggingFace Daily Papers(社区热门论文)·2026-08-18 08:00·8天前
AI 导读

TileMix 提出一种以 tile 为中心的精度路由内核,将注意力矩阵划分为硬件对齐的 score tile,通过紧凑位掩码将每个 tile 组分配给 FP16 或 INT8 计算,同时共享在线 softmax 状态。

HuggingFace Daily Papers(社区热门论文)
53AI 编辑部评分,满分 100

TileMix:面向 LLM 推理加速的以 Tile 为中心的混合精度注意力机制

2026-08-18 08:00· 8天前
AI 导读

TileMix 提出一种以 tile 为中心的精度路由内核,将注意力矩阵划分为硬件对齐的 score tile,通过紧凑位掩码将每个 tile 组分配给 FP16 或 INT8 计算,同时共享在线 softmax 状态。

Long-context prefill in large language models (LLMs) incurs substantial computation and memory traffic because dense self-attention computes quadratic query-key scores. Existing methods either use a uniform low-precision path or select token interactions, leaving spatial precision routing over hardware-aligned score tiles outside fused dense attention. We introduce TileMix, a tile-centric precision-routing kernel that makes numerical precision an executable spatial decision over score-tile groups within fused dense attention. TileMix partitions the attention matrix into hardware-aligned score tiles, packs routing decisions into compact bitmasks, and dispatches each tile group through FP16 or INT8 score computation while both paths update a shared online-softmax state. Scalable precision grouping lets each routing bit govern multiple adjacent key tiles, preserving hardware-aligned compute tiles and compact metadata at long contexts. By routing all legal tile groups, TileMix preserves dense token connectivity, requires no training, and supports grouped-query attention, variable-length batches, and INT8 key/value caches. Across LongEval, LV-Eval, and A100 prefill benchmarks on LLaMA, Qwen, and Vicuna, TileMix recovers long-context quality lost under uniform INT8 and improves prefill throughput over FP16, yielding a controllable accuracy-efficiency frontier across model families. The implementation is available at https://github.com/HanzhiZhang-Ulrica/TileMix.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org