ICML 2026 随机实验与调查：LLM 在同行评审中的使用与影响

DOI: XXXXXXX.XXXXXXXConference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYISBN: 978-1-4503-XXXX-X/2018/06

Sunnie S. Y. Kim Note: Both authors contributed equally to this research.
Corresponding authors: {sunniesykim, wesleydeng, mdudik}@microsoft.com Affiliation: Microsoft Research, New York, USA Wesley Hanwen Deng Affiliation: Microsoft Research, New York, USA , Jennifer Wortman Vaughan Affiliation: Microsoft Research, New York, USA , Buxin Su Affiliation: University of Pennsylvania, Philadelphia, USA , Weijie Su Affiliation: University of Pennsylvania, Philadelphia, USA , Alekh Agarwal Affiliation: Google Research, Kirkland, USA , Sharon Li Affiliation: University of Wisconsin-Madison, Madison, USA , Martin Jaggi Affiliation: EPFL, Lausanne, Switzerland , Daniel G. Goldstein Affiliation: Microsoft Research, New York, USA , Nihar B. Shah Affiliation: Carnegie Mellon University, Pittsburgh, USA and Miroslav Dudík Affiliation: Microsoft Research, New York, USA

2018

Abstract.

LLMs are rapidly reshaping peer review, making it important to understand how reviewers use them in practice and how different LLM-use policies affect review outcomes. We investigate these questions through a randomized experiment and an anonymous post-survey at ICML 2026, a major machine learning conference involving over 24,000 papers and 17,000 reviewers. Reviewers were assigned to either a conservative policy prohibiting all LLM use or a permissive policy allowing limited assistance, with randomization among a subset of main-track papers and reviewers. Policy assignment had near-zero effects on final paper decisions, paper scores, and reviewer confidence, although reviews under the permissive policy were 5.5-7% longer. Post-survey responses (N=1,486) revealed diverse attitudes toward LLMs and substantial noncompliance: 22.5% of conservative-policy reviewers reported using an LLM despite the prohibition, and 36.5% of permissive-policy reviewers reported at least one explicitly disallowed use. We discuss implications for future peer-review policy and tool design.

1. Introduction

Rapidly growing submission volumes are increasing the burden on peer-review systems and amplifying longstanding concerns about “reviewer fatigue” (Publons, 2018). At CHI and ICML, two leading conferences in human-computer interaction (HCI) and machine learning (ML), the increase in submissions between 2023 and 2026 was 3,182 to 6,730 (+112%) and 6,538 to 24,661 (+277%), respectively. In response to these pressures and the growing capabilities of large language models (LLMs), it has become common for researchers to incorporate the use of LLMs into their peer review practices (Liang et al., 2024; Liao et al., 2024; La Rosa and Nasser, 2026; Lemberger et al., 2026). Computer Science (CS) conferences, in turn, have begun to publish explicit peer-review LLM policies and explore the use of LLM assistance in the review workflow (Thakkar et al., 2026; International Conference on Machine Learning, 2025; International Conference on Artificial Intelligence and Statistics, 2025; IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026; European Conference on Computer Vision, 2026; IEEE/CVF International Conference on Computer Vision, 2025; International Conference on Artificial Intelligence and Statistics, 2026; ACM CHI Conference on Human Factors in Computing Systems, 2026; Nature Portfolio, 2026; ACL Rolling Review, 2026; Beygelzimer et al., 2023).¹¹ 1 In Computer Science, conferences are a terminal publication venue: many conferences including ICML review full papers rather than just abstracts, and are often considered comparable to or more prestigious than journals. (See Appendix A for an overview of these policies.)

Research to date has concentrated on what LLMs can and cannot do for peer review: exploring whether they can evaluate if the claims made in a paper are supported with evidence (Liu and Shah, 2023; Xi et al., 2025), benchmarking the quality of model-generated reviews against human ones (Thakkar et al., 2025; Baumann et al., 2026; Chen et al., 2026), probing models’ failure modes such as sycophancy and hallucination (Sharma et al., 2026; Liang et al., 2024), and, more recently, designing and building LLM-powered tools to assist reviewers (Chen et al., 2025; Zhu et al., 2026).

However, we currently lack an empirical understanding of how peer-review LLM policies shape review outcomes—a causal question best addressed through randomized controlled trials (RCTs) in live review settings rather than retrospective observational analyses (c.f., (Jecmen et al., 2020; Saveski et al., 2023)). In addition, we know surprisingly little about how reviewers actually use LLMs in practice—at which stage of their reviewing workflow, for which tasks, with how much verification, and where use shades into misuse—and whether these uses comply with conferences’ LLM policies. Understanding these questions is important for designing future policies and tools that are grounded in the realities of peer-review practice.

In this work, we investigate these questions through a large-scale RCT and a post-survey conducted in collaboration with the organizers of the International Conference on Machine Learning (ICML) 2026, which involved 76,159 authors, 17,886 reviewers, and 24,661 submitted papers. The RCT randomized a subset of submitted papers and reviewers across two policies for LLM use (see Section 3.2 for the full policy descriptions):

• •

  Conservative Policy: No LLM use was permitted during the review process.

• •

  Permissive Policy: LLM use was permitted for understanding submitted papers and related work and for polishing reviewer-written text, but not for evaluating submitted papers or drafting reviews.

The study examined how policy assignment affects paper scores, final paper decisions, reviewer confidence, review length, and review quality ratings from area chairs (ACs) overseeing the review process. Across the two randomizations, we found that assignments of the reviewers to the conservative versus the permissive policies had near-zero effects on final paper decisions, paper scores, and reviewer confidence. However, we found small effect on the review length and perceived review quality, with reviews produced under the permissive policy slightly longer (5.5%–7.7%) and of slightly higher perceived quality than those produced under the conservative policy.

After the review process concluded, we conducted an anonymous post-survey (N=1,486) of ICML 2026 reviewers, ACs, and senior ACs (SACs) to understand how reviewers actually used LLMs under their assigned policy, how LLM use shaped reviewers’ experience, and respondents’ perspectives on future LLM policies. We found substantial variation in experiences and attitudes toward LLMs: reviewers and ACs who used LLMs generally reported benefits such as improved review quality and reduced time spent, while many non-users reported little need for LLM assistance or raised principled objections to its use. We also observed substantial policy noncompliance: 22.5% of reviewers under the conservative policy reported using an LLM despite the prohibition, and 36.5% of reviewers under the permissive policy reported at least one explicitly disallowed use. Finally, respondents broadly supported LLM use for assistive tasks but were more cautious about tasks requiring substantial reviewer judgment, while many questioned whether such boundaries could be effectively enforced. Key findings are summarized in Figure 1.

Overall, this work provides a rare empirical look at LLM use in peer-review. In particular, it provides:

• •

  Evidence from a large-scale RCT on how policies for LLM use in peer review affect review outcomes (Section 4);

• •

  Insights from a large-scale post-survey on reviewers’ and ACs’ LLM use, experiences with LLMs, or reasons for non-use, as well as community perspectives on future LLM-use policies (Section 5);

• •

  Implications for the design of peer-review policies and tools, informed by perspectives from reviewers, ACs, and SACs, many of whom were also authors of submissions at the same conference (Section 6.1);

• •

  Broader implications for how LLMs can support meaningful human engagement in peer review and beyond (Section 6.2).

注意转载授权 · 阅读原文