"Decoupling Exploration from Optimization in RLVR"
摘要
论文提出 Exploration-Distillation（ExpDis），把 RLVR 训练拆成两个模型：探索者因找到新颖解法获得奖励，学生模型只从探索者已验证正确的解法中学习，再继续标准 RL。在三个模型上，相同算力预算下 ExpDis 在数学推理上优于标准 DAPO，准确率与解法多样性均提升，甚至超过训练步数多 4 倍的 DAPO。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
"Decoupling Exploration from Optimization in RLVR"
What if LLMs could explore completely new reasoning strategies during RL without risking their existing capabilities?
This paper Exploration-Distillation (ExpDis) separates training into two models where one explorer is rewarded for finding novel solutions, and another student that learns only from the explorer's verified correct solutions before continuing standard RL.
Across three models, ExpDis outperforms standard DAPO on math reasoning at the same compute budget, improving both accuracy and solution diversity, and even outperforming DAPO trained for 4x as many steps.
来源：alphaXiv · x.com
引用原文
askalphaxiv. "Decoupling Exploration from Optimization in RLVR"[EB/OL]. (2026-10-30)[访问日期]. https://x.com/askalphaxiv/status/2116078149612052645.