Sparse attention reduces computation by selecting relevant tokens, but finding those tokens becomes extremely expensive at ultra-long con...
摘要
论文提出 Latent Sparse Attention（LSA），将相似 token 聚类并组织成可搜索的层级结构，使查询无需扫描整个上下文即可定位相关信息。在 10M token 上下文下，LSA 单层注意力的 prefill 比 GLM-5.3-Flash 原稀疏注意力快 22.7 倍、解码快 4.6 倍，且准确率相近。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
Sparse attention reduces computation by selecting relevant tokens, but finding those tokens becomes extremely expensive at ultra-long contexts, sometimes accounting for 98% of attention's prefill time.
This paper Latent Sparse Attention (LSA) groups similar tokens into clusters and organizes them into a searchable hierarchy, letting queries to find relevant information without scanning the entire context.
At 10M tokens, LSA makes one attention layer 22.7x faster during prefill and 4.6x faster during decoding than GLM-5.3-Flash's original sparse attention, while maintaining similar accuracy.
来源：alphaXiv · x.com
引用原文
askalphaxiv. Sparse attention reduces computation by selecting relevant tokens, but finding those tokens becomes extremely expensive at ultra-long con...[EB/OL]. (2026-11-07)[访问日期]. https://x.com/askalphaxiv/status/2118940099362557982.