"Depth Memory"
摘要
Linear Depth Memory（LDM）用递归最小二乘法在Transformer各层间维护固定大小的记忆，替代随深度增长的注意力残差存储。在400M和1.5B模型上，LDM的困惑度低于标准残差、mHC和AttnRes，零样本准确率具竞争力，配合优化内核吞吐相当。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
"Depth Memory"
Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth.
Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream.
On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels.
来源：alphaXiv · x.com
引用原文
askalphaxiv. "Depth Memory"[EB/OL]. (2026-10-20)[访问日期]. https://x.com/askalphaxiv/status/2112455949881573494.