循环Transformer：仅循环中间层效果更优

"Towards Looped Models Done Right"

Not all ways of looping Transformers work equally well. This paper finds that looping only the middle layers performs better than repeatedly looping the entire Transformer.

The researchers compare different loop architectures and find that the Huginn-style design, which places recurrent layers between separate input and output layers, generally performs better, especially on reasoning tasks.

At 8B parameters, Huginn MoE outperforms Ouro on 8 out of 10 benchmarks. It also uses 75% fewer stored parameters than a compute-matched 32B feedforward MoE, while achieving better GSM8K accuracy (83.6% vs. 80.8%)

https://www.alphaxiv.org/abs/2610.23036