Zyphra 提出 ZHA 分层注意力压缩 KV-cache
摘要
Zyphra 新论文提出 Zyphra Hierarchical Attention（ZHA），将 KV-cache 组织成压缩摘要树，查询自上而下搜索以缩小需关注的 token 范围。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
Zyphra's new paper proposes Zyphra Hierarchical Attention (ZHA), which organizes the KV-cache into a tree of compressed summaries. Queries search the tree from top to bottom, narrowing down which tokens deserve attention.
With just 1B tokens of additional distillation training, ZHA preserves most of the performance of full attention on models up to 35B parameters. On RULER at 256K context, it achieves 93.3% vs 96% accuracy, while attending to just 1-6% of the KV-cache.
来源：alphaXiv · x.com
引用原文
askalphaxiv. Zyphra's new paper proposes Zyphra Hierarchical Attention (ZHA), which organizes the KV-cache into a tree of compressed summaries. Querie...[EB/OL]. (2026-11-07)[访问日期]. https://x.com/askalphaxiv/status/2119015504363106653.