Ai2 测试架构与预训练对 LLM 失败的影响
To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs.
摘要
为测试失败是源于架构还是预训练，我们直接在 StateBench 上训练 LLM，然后用更长的输入对它们进行评估。 混合模型 + Gated DeltaNets 在前 3 个级别上近乎完美；transformer 随长度增加而退化，且无法解决一般有限问题。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs.
Hybrids + Gated DeltaNets were near-perfect on the first 3 levels; transformers degraded with length & couldn’t solve general finite.
来源：Ai2 · x.com
引用原文
allen_ai. To test whether failures came from architecture or pretraining, we trained LLMs directly on StateBench, then evaled them on longer inputs.[EB/OL]. (2026-11-04)[访问日期]. https://x.com/allen_ai/status/2117670299298599001.