论文提出 softmax 与交叉熵数值修复方案

摘要

一篇论文指出 softmax 和交叉熵存在数值缺陷，会使极小梯度错误地舍入为零或本应消失却保持非零，进而引发损失突增并干扰知识蒸馏。作者提出简单修复，在小规模 grokking 基准上消除了 Slingshot 损失尖峰，某设置下测试准确率从 94.3% 提升至 100%，且无需更高数值精度即可防止蒸馏中学生模型偏离教师模型。

摘要由模型根据原文撰写，以原文为准。

"A Fix for Softmax and Cross-Entropy"

Turns out two of the most fundamental functions in deep learning have numerical bugs that can actually destabilize training.

This paper proposes simple fixes to softmax and cross-entropy that prevent tiny gradients from incorrectly rounding to zero, or remaining nonzero when they should vanish, as these errors can cause sudden loss spikes and interfere with knowledge distillation.

On small-scale grokking benchmarks, the fixes eliminate Slingshot loss spikes and improve test accuracy from 94.3% to 100% in one setting. They also prevent student models from drifting away from their teachers during distillation, without requiring higher numerical precision.

https://www.alphaxiv.org/abs/2610.31642

论文提出 softmax 与交叉熵数值修复方案

摘要

正文 · 原文

引用原文