研究：LLM 审稿人偏好自信措辞，内容不变也会改变评分与录用判断

   Sumedh Santosh Joshi Affiliation: CISPA Helmholtz Center for Information Security    Philipp Christmann Affiliation: CISPA Helmholtz Center for Information Security    Ivaxi Sheth Affiliation: CISPA Helmholtz Center for Information Security    Ruta Binkyte Affiliation: CISPA Helmholtz Center for Information Security Affiliation: German Research Center for Artificial Intelligence (DFKI)    Koel Dutta Chowdhury Affiliation: University of Technology Nuremberg    Raouf Kerkouche Affiliation: INRIA, University of Lille, CNRS, Centrale Lille    Hui-Po Wang Affiliation: CISPA Helmholtz Center for Information Security    Mario Fritz Affiliation: CISPA Helmholtz Center for Information Security

Abstract

The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload of researchers. At the same time, large language models (LLMs) are increasingly used to support peer review, both by individual reviewers and by conference organizations. While LLM capabilities have improved considerably, their evaluations may be influenced by linguistic characteristics that are unrelated to scientific content. In this work, we introduce a controlled experimental framework for studying such effects by systematically perturbing the linguistic style of research papers. Specifically, we rewrite papers to exhibit (i) linguistic features associated with native or non-native English writing, and (ii) high- or low-confidence linguistic features. To construct these interventions, we derive fine-grained rewriting instructions grounded in the linguistic literature and apply them using LLMs. Crucially, our interventions are designed to preserve the underlying scientific content. We then evaluate the resulting papers using different LLM reviewers. Our experiments expose a critical vulnerability in LLM-based peer review: reviewers systematically reward confident rhetoric independent of scientific content, raising serious concerns about the robustness of their assessments.

^**footnotetext: Equal contribution.

1 Introduction

The peer-review process is an integral part of scientific publishing, but also requires substantial time and effort from researchers. This challenge has become increasingly pronounced with the growing number of submissions to major AI conferences. In parallel, recent advances in large language models (LLMs) have opened up new opportunities to automate or support parts of the reviewing process (Jin et al., 2024; Lin et al., 2025; Dycke and Gurevych, 2026). Individual reviewers use LLMs to assist with the review process, while LLMs are also increasingly seen as systematic tools for generating or supporting peer review (Demetrio et al., 2025; Baumann et al., 2026).

However, the use of LLMs for peer review also introduces new risks. LLM judgments can be influenced by properties that are unrelated to the actual quality of the evaluated content. Prior work on LLM-as-a-judge has, for example, demonstrated biases related to the position of an answer or preferences towards outputs of the same LLM family (Liu et al., 2024; Wataoka et al., 2024; Ye et al., 2025). Similar concerns have been identified specifically for LLM-based peer review as well. LLM reviewers can be susceptible to adversarial modifications of papers (Lin et al., 2025), fail to reliably identify faulty scientific reasoning (Dycke and Gurevych, 2026), and exhibit biases based on author-related information such as affiliation, seniority, or publication history (Vasu et al., 2026). Moreover, recent results suggest that even paraphrasing or stylistic properties of a paper can substantially affect LLM-generated review scores (Kaneko, 2026; Kousha and Thelwall, 2026; Baumann et al., 2026). These findings raise the central question of our paper: ”How do we systematically measure whether LLM reviewers judge the research quality, rather than the linguistic style with which the research content is presented?”

In this work, we specifically study the impact of linguistic style on LLM-based peer review. Researchers have different linguistic backgrounds, and previous work has already found evidence that language proficiency and writing style can influence human peer review (Politzer-Ahles et al., 2020; Saposnik et al., 2014; Lee et al., 2006; Sayegh et al., 2024). Motivated by these human studies, we focus on two complementary dimensions that together signal epistemic authority: (i) the language proficiency reflected in the writing, and (ii) the confidence with which the research is presented. Language proficiency can signal author credibility (Lev-Ari and Keysar, 2010), while confident linguistic framing can influence the perceived scientific credibility (Teigen, 2023). Both dimensions are well characterized in the linguistic literature (Hinkel, 2002; Hyland, 1996), and should not substantially affect the assessment of the underlying research quality.

Systematically studying such effects is challenging, as naturally occurring papers with different linguistic styles also differ in their research content and quality. We therefore propose a controlled framework for perturbing a research paper along a specific linguistic dimension, while preserving its underlying research content. Specifically, we rewrite papers to exhibit (i)  linguistic characteristics associated with native or non-native English writing, and (ii) high-confidence- or low-confidence linguistic characteristics. Rather than relying on generic rewriting instructions, we derive fine-grained prompts from the linguistic literature to systematically control the corresponding stylistic features. Crucially, we design these interventions to avoid distorting the content and thus the research quality of the paper, and verify the content integrity of the resulting rewrites. To assess whether rewrites exhibit the intended stylistic characteristics we use established linguistic measures. The different rewrites are then given to LLM reviewers, that provide us with a full-fledged peer review. While review scores are the immediate quality signal, the arguments raised in reviews play an important role in the final decision process as well (Arous et al., 2026). To this end, we integrate an automated annotation procedure for identifying positive and negative aspects in reviews (Lu et al., 2025).

While related work has used counterfactual modifications to study reasoning capabilities (Dycke and Gurevych, 2026), author-related biases (Vasu et al., 2026), or the general susceptibility of LLM reviewers to rewriting and paraphrasing (Kaneko, 2026; Baumann et al., 2026), framework isolates linguistic cues of epistemic authority and examines their effects on review scores and qualitative arguments. We use this framework to study different state-of-the-art LLM reviewers and investigate whether their reviews systematically favor perceived language proficiency and confidence.

Our experiments reveal that the two linguistic dimensions affect LLM reviewers differently. Across models, expressed confidence has a strong and consistent effect on evaluation even though the underlying scientific content remains unchanged. In contrast, differences between native- and non-native associated writing are substantially smaller and more model-dependent. More importantly, our controlled rewriting framework provides a practical way to stress-test LLM reviewers by varying specific linguistic cues and measuring when these changes alter review outcomes. Our analysis further shows that these effects extend beyond review scores: LLM reviewers raise substantially different arguments across rewrites, focusing on different aspects of the paper and expressing different sentiments toward them.

In summary, our main contributions are:

• ∙

  We devise a controlled experimental framework for stress-testing LLM reviewers. By rewriting research papers along different linguistic dimensions in a content-preserving manner, our framework enables the systematic investigation of linguistic biases in LLM-based peer review.

• ∙

  We design literature-grounded rewriting prompts to manipulate language proficiency and expressed confidence, and validate the resulting rewrites using established linguistic measures.

• ∙

  We conduct experiments across state-of-the-art LLM reviewers. Our results reveal that expressed confidence has a strong and consistent effect on reviewer preferences, while effects associated with language proficiency are substantially smaller and vary across models.

2 Methodology

In this section, we introduce our experimental framework for controlled linguistic interventions to detect stylistic biases in LLM reviewers. Starting with the original paper, our framework proceeds in three stages. First, we generate two versions that differ along a specific linguistic dimension, while preserving the underlying research content. We then validate the content integrity of rewrites, and verify the presence of the respective linguistic style. Finally, we provide the resulting versions to LLM reviewers and compare their reviews. Figure 1 provides an overview of our framework.

2.1 Linguistic Rewriting

A central challenge in designing the rewriting interventions is to obtain linguistically distinguishable versions while changing only their linguistic realization and not the scientific content. This challenge was also highlighted in prior work on linguistically informed LLM-rewriting (Kunilovskaya et al., 2024). A simple solution would be to directly ask an LLM to imitate a particular linguistic style. However, this could surface stereotypical characteristics (Cheng et al., 2023) or perturb the scientific content itself, leading to incomparable rewrites. Hence, we opt for a more principled approach and carefully design rewriting prompts based on specific linguistic properties described in the corresponding literature.

Preserving content integrity. A crucial requirement of our interventions is that the scientific content, and thus the research quality of the paper, is preserved. To this end, we design a set of content integrity rules shared across rewriting prompts (Figures 8 - 11). These rules specify which content must remain unchanged and require preserving the length and structure of the original text. At the same time, all rewrites must remain grammatical and plausible as professional academic English, to avoid actually degrading the scientific quality by introducing grammatical mistakes or typos.

Perturbing language proficiency. Prior work identifies systematic differences between L1 and L2 English academic writing at the lexical, phraseological, and syntactic levels. Crossley and McNamara (2009) identify differences in lexical variation and sophistication, while Ädel and Erman (2012) find a larger and more varied repertoire of recurrent lexical bundles in native English writing. Hinkel (2002) further documents differences across a broader set of lexical, syntactic, and rhetorical features. Finally, L1 and L2 writers differ in the linguistic expressions they use to qualify claims and signal certainty, including their choice of words for conveying different degrees of epistemic commitment (Hyland and Milton, 1997).

Based on these findings, we characterize language proficiency primarily through the breadth and diversity of the available linguistic repertoire. Our prompts therefore target lexical diversity and sophistication, the repertoire of conventional academic expressions, syntactic variation, and the range of epistemic expressions. Importantly, lower proficiency is not represented through grammatical errors or otherwise defective English. Both rewriting directions represent plausible, publication-quality academic writing and are designed to capture subtle tendencies reported in the literature rather than stereotypical native or non-native writing. The prompts are provided in Figures 8 and 9.

Perturbing expressed confidence. Our second dimension captures the degree of expressed confidence expressed in the writing. A central mechanism for expressing epistemic commitment in academic writing is the use of hedges and boosters: hedges qualify commitment and introduce uncertainty, whereas boosters emphasize certainty and stronger commitment (Hyland, 1996; Hyland, 1998). Their use reflects different degrees of commitment and detachment in academic discourse (Vassileva, 2001), and more recent work similarly studies epistemic positioning through such linguistic markers (Yang and Guo, 2025).

Hence, we perturb papers along the confidence dimension by varying the degree of epistemic commitment expressed through hedging, boosting, modality, and qualification. The high-confidence and low-confidence prompts move these properties in opposite directions while avoiding exaggerated certainty or implausible hesitation. A central challenge when designing the prompts was to distinguish expressed confidence from the substantive strength of a scientific claim: its linguistic formulation may become more or less cautious, but its scope, evidence, and underlying scientific meaning must remain unchanged. The rewriting prompts are provided in Figures 10 and 11.

Table 1 provides example rewrites for both dimensions along with the original version.

2.2 Rewrite Validation

The validity of our experimental setup depends on two central properties of the generated rewrites. First, the rewrite should preserve the scientific content of the original paper. Second, the rewriting process should systematically induce the intended linguistic characteristics. We therefore evaluate both content integrity and linguistic change for each rewriting dimension.

Validating the content integrity. First of all, we validate that our systematic rewrites preserve the content integrity of the original paper, and are thus comparable. To this end, we consider two different measures: semantic similarity and lexical preservation. Specifically, we compute the semantic similarity between each rewrite and the original paper using dense embeddings.¹¹ 1 We use BAAI/bge-large-en-v1.5 for embedding passages. For long documents, comparing dense embeddings directly can overestimate similarity scores (Xiao et al., 2023). Hence, each paper is split into passages, and we then compute the average maximum cosine similarity between original and rewritten passage embeddings. This ensures that we capture content integrity on a more fine-grained level than with pure paper- or section-level embeddings. In addition to semantic similarity, we quantify the lexical preservation using cosine similarity between unigram and bigram TF-IDF representations for the original paper and rewrites. This metric complements semantic similarity and validates whether the technical terminology is preserved.

We report the content-integrity scores in Table 2. Across all rewrites, both measures strongly indicate the preservation of scientific content, with scores close to their maximum value of 1.

Validating language proficiency perturbations. We validate the language-proficiency rewrites using established measures for lexical and textual complexity. First, we use Measure of Textual Lexical Diversity (MTLD) to measure lexical diversity (McCarthy and Jarvis, 2010). This directly reflects our instruction to vary the breadth and diversity of the vocabulary. Second, we use Advanced Guiraud (AG) to measure lexical sophistication (Daller et al., 2003), capturing the use of advanced vocabulary. Both measures are motivated by prior work for identifying systematic differences in lexical variation and sophistication between L1 and L2 English writing (Crossley and McNamara, 2009). Finally, we report the Gunning Fog index (GF) as a broader measure of textual complexity (Gunning, 1952), which considers sentence length and the prevalence of complex words.

Table 2 shows the results for this validation step. The linguistic measures demonstrate the viability of our rewriting prompts for perturbing the language proficiency: all metrics show an increase for native-associated rewrites compared to non-native-associated rewrites.

Validating expressed confidence perturbations. For expressed confidence, we directly measure the linguistic properties targeted by our rewriting prompts, and measure the frequency of boosters and hedges per 1000 words.

As shown in Table 2, high-confidence rewrites contain substantially more boosters and fewer hedges than low-confidence rewrites, confirming the successful manipulation of expressed confidence.

2.3 Comparative Reviewing

After systematically rewriting papers and validating the underlying linguistic properties, we can stress-test LLM reviewers along different linguistic dimensions. Our framework enables to determine whether papers of the same research quality receive systematically different evaluations based on their linguistic realization. For each paper and rewriting dimension, the LLM reviewer receives one rewritten version at a time and produces a complete review together with an overall review score. We then compare the scores assigned to the corresponding versions of the same original paper.

Review instructions. We derive a reviewer prompt from standard reviewing instructions at major AI conferences, including explicit review criteria and guidance on the expected output format. The specific prompt is provided in Figure 6. We conducted preliminary experiments with different prompts and found the LLM reviewers to be largely insensitive to such prompt variations: the direction and significance of all effects remain unchanged (details in Appendix C).

Review output. The reviewer output contains the paper’s strengths, weaknesses, and suggestions for improvement. In addition, it includes scores for soundness, significance, evidence, and clarity on a 1–4 scale, a reviewer confidence score on a 1–5 scale, and an overall rating on a 1–10 scale. We explicitly outline the Likert scale for the overall rating in the prompt to avoid any ambiguities.

2.4 Configuration

Research papers. We sample 504 research papers from the 2026 reviewing cycle of ICLR, uniformly across the 21 primary areas. From each primary area, we randomly sample 12 accepted and 12 rejected papers. All papers are anonymized, simulating a double blind peer-review setup.

Rewriting model. For paper rewriting, we use Claude Opus 5 in all experiments to ensure strong adherence to our prompts. Keeping the rewriting model fixed prevents differences across reviewers from being confounded by variation in the rewriting model.

LLM reviewers. For reviewing, we consider LLMs from different model families and capability levels. We choose only LLMs with a knowledge cutoff before 2026, to avoid any confounding factors due to known acceptance decisions for ICLR 2026. As proprietary models, we select Gpt-5.5, Claude Sonnet 4.6 and Gemini 3.5 Flash, as they reflect the default model choices in the respective consumer-facing chat interface as of that cutoff. In addition, we consider the open-weight models Gemma3 (27B), Qwen3 VL (235B) and Glm4.7 (358B).

Reviewer variance. Since LLM-reviewer scores can vary across repeated generations, even for identical inputs, we quantify the reviewer variance for every reviewer-model to help interpret our results. We randomly sample 10 papers and conduct n=10 independent repeated reviews for each LLM. The variance for each model is reported in Table 3. Reviewer variance differs substantially across models: Gpt-5.5, Claude Sonnet 4.6, Gemma3, and Qwen3 VL exhibit pooled standard deviations of approximately 0.5 rating points, while Glm4.7 (0.85) and particularly Gemini 3.5 Flash (1.24) show greater variability. This enables us to contextualize score differences between linguistic rewrites relative to the inherent stochastic variation of the corresponding reviewer.

We provide further technical details, including LLM generation parameters, in Appendix A.

3 Experimental Results

3.1 Main Results

We first investigate whether linguistic rewrites systematically affect the scores assigned by LLM reviewers. Our primary comparisons are native vs. non-native for language proficiency and high-confidence vs. low-confidence for expressed confidence. For each pair, we report the percentage of the first version receiving a higher review score, the second version receiving a higher score, or both versions receiving the same score (i.e., a tie). Table 4 reports our main results.

Preferences along the linguistic dimensions. The first takeaway is that the two linguistic dimensions affect reviewer scores differently. For the language proficiency dimension, preferences are comparatively weak and inconsistent across reviewers. For 5/6 of the reviewers, native rewrites receive higher scores more often, but the corresponding preference ratios remain subtle (between 1.0× and 1.3×). In contrast, we observe a consistent and substantially stronger preference towards high-confidence over low-confidence rewrites across all LLM reviewers. High-confidence rewrites are preferred between 1.3× and 7.0× as often as their low-confidence counterparts. The most notable preference ratios are observed for Claude Sonnet 4.6 (7.0×) and Gemma3 (6.0×).

The average scores show a similar pattern: scores are nearly identical between native and non-native rewrites, but consistently higher for high-confidence than for low-confidence rewrites.

Impact on review decisions. A change in review score becomes more significant as it flips the acceptance decision of the paper. We therefore analyze how frequently our linguistic interventions result in such decision flips. Specifically, we define a decision flip whenever the two rewrites of the same paper fall on opposite sides of the acceptance threshold (4.5 in our case). Table 4 reports both directions separately: non-native rejection → native acceptance (First↑) and native rejection → non-native acceptance (Second↑), and for confidence low-confidence rejection → high-confidence acceptance (First↑) and the reverse direction (Second↑).

For language proficiency, decision flips are largely balanced, which is consistent with the absence of a systematic score preference. For confidence, flips predominantly favor the high-confidence version. We observe the most drastic effect for Claude Sonnet 4.6: in 21.0% of reviews, Claude Sonnet 4.6 accepts the high-confidence version and reject the low-confidence version, with only 2.8% of papers accepted only for the low-confidence rewrite. As the average scores for Gemma3 and Qwen3 VL lie well above the acceptance threshold, decision flips are rare for these models.

3.2 In-depth Analysis

Combined effects of linguistic style. Our main experiments consider language proficiency and confidence in isolation. This analysis investigates how both dimensions interact when they occur simultaneously. For a batch of 126 papers, we generate four additional rewrites, including all combinations of the two dimensions. The corresponding rewriting prompts are merged (see Figures 12-15 in the Appendix). We then investigate (i) the language proficiency dimension given a specific confidence rewrite, (ii) the confidence dimension given a specific proficiency rewrite, (iii) the comparison of the two extremes (native &high-confidence vs. non-native &low-confidence), and (iv) the inverse (native &low-confidence vs. non-native &high-confidence). Table 5 shows the results of this analysis.

The conditional comparisons largely reproduce the effects observed when varying the dimensions in isolation. Holding confidence fixed, differences between native and non-native rewrites are slightly more pronounced but remain relatively small. The preference for high-confidence rewrites persists when combined with the proficiency dimension. The cross-condition comparisons further illustrate the strength of the confidence effect. When native and high-confidence language are combined and compared against non-native &low-confidence rewrites, the former is preferred with preference ratios of up to 9.1×. Reversing the confidence dimension switches this pattern, with all reviewers favoring the non-native but high-confidence rewrite. Across the board, the confidence dimension dominates these combined comparisons.

Preferences along the language proficiency dimension. Our main results discover substantial effects of linguistic rewrites along the confidence dimension, but only subtle changes along the proficiency dimension. A natural question that arises is whether LLM reviewers are indeed not substantially affected by language proficiency, or whether our rewrites are not sufficient for bringing out existing biases (despite validated rewriting effects by linguistic metrics; see Table 2). To answer this question, we simulate biased reviewers by injecting a dedicated instruction into the reviewer prompt: “In addition to assessing the scientific content, place substantial weight on the linguistic style and overall presentation of the paper …” (full prompt in Figure 7). We intentionally keep the prompt general, and avoid mentioning language proficiency directly. We run this analysis on 126 papers, equally drawn from the primary areas (see Table 9 in the Appendix for the full results).

Our framework reveals the simulated biased reviewers, and identifies strong preference ratios towards native-associated writing of 6.8 (Gpt-5.5), 4.3 (Claude Sonnet 4.6), and 2.2 (Gemini 3.5 Flash), all of which are statistically significant under a two-sided sign test (p<0.001 for Gpt-5.5 and Claude Sonnet 4.6, p<0.01 for Gemini 3.5 Flash). These results demonstrate that our framework can reveal both the presence (this experiment) and absence (Table 4) of systematic preferences along stylistic dimensions, supporting its utility for stress-testing LLM reviewers.

Aspects that drive the reviews. Review scores alone do not decide the outcome of the peer review process. Arguments brought up in the review are also decisive in deriving an acceptance decision (Arous et al., 2026). We therefore extend our study beyond the plain review scores, and investigate which aspects of a paper (Lu et al., 2025) drive the reviewers’ assessments and whether this focus changes with linguistic style. In particular, we study whether linguistic variations affect the attention given to the underlying scientific content.

We utilize the taxonomy established by Lu et al. (2025), and consider the 16 coarse-grained aspects they defined, including aspects such as Presentation, Methodology, Novelty or Related Work. The annotation is automated via an LLM²² 2 We use Gemma4 (26B) for these simple annotations., which provides a list of aspects with their connotation (positive or negative). In addition, the annotator provides the exact phrase the aspect was derived from, which we verify via substring matching (all of the derived aspects were valid). The prompt we used for aspect annotations is provided in Figure 16 in the Appendix.

We then investigate the aspect-level polarity under linguistic rewriting. For each review, aspect and polarity, we count positive and negative mentions of aspects, and define the net polarity of an aspect as #positive mention – #negative mentions. Along one dimension, we then compute the within-paper difference between rewrites (native–non-native and high-confidence–low-confidence), and average this difference over all papers. The results are provided in Figure 2.

While proficiency rewrites have only subtle effects on review scores, our aspect-level analysis reveal differences in how the LLM reviewer perceive the rewrites: for native-associated papers Presentation is less often criticized (Δ=−2.4 pp) and more often praised (Δ=+3.0 pp), leading to a net-polarity increase of 0.16. Interestingly, this behavior is similar for the confidence dimension, where Presentation also has the largest net-polarity increase (+0.29). These shifts in perceived Presentation quality do not systematically affect review scores in both cases (r=0.04), which is a desired property in this case.

Finally, we find that the overall rating strongly correlates with the share of positive aspect mentions (r=0.81). For high-confidence rewrites, aspects such as Methodology, Theory and Contribution are perceived more positively, alongside the larger Presentation gain, contributing to the increase in positive mention share, and thus contributing to the preference ratios observed in Tables 4 and 5.

4 Related Work

LLMs for peer review. With increasing paper submissions, conferences and reviewers have explored the use of LLMs to assist with peer review (Jin et al., 2024; Demetrio et al., 2025; Thakkar et al., 2026; Zhu et al., 2025; Huang et al., 2025). While promising, LLM-based reviewing raises reliability and fairness concerns, including biases toward author-related properties such as affiliation, seniority, and publication history (Ye et al., 2024; Zhu et al., 2025; Vasu et al., 2026), as well as sensitivity to textual adversarial attacks and prompt injections. Most closely related, Kousha and Thelwall (2026) find that linguistic complexity and text length are more strongly associated with ChatGPT-based research evaluations than with expert-based evaluations in many fields. We complement this work by applying controlled, linguistically grounded transformations to the same papers, manipulating language proficiency and expressed confidence while preserving research content. This isolates effects of linguistic presentation from differences across papers.

Linguistic biases in human peer review. Prior linguistic research documents systematic differences between L1 and L2 English writing. Crossley and McNamara (2009) identify differences in lexical variation and sophistication, while Ädel and Erman (2012) find that native English writers employ a larger and more varied repertoire of recurrent lexical bundles. Differences also extend to epistemic expression: Hyland and Milton (1997) show that L1 and L2 writers differ in the range and precision with which they express qualification and certainty. Linguistic presentation can also affect human peer review. In a controlled experiment, Politzer-Ahles et al. (2020) find preliminary evidence that identical scientific content receives lower ratings when presented in non-native-like rather than native-like English, although the manual interventions limited the study to paper abstracts. Similarly, an observational study of submissions to Stroke finds lower priority scores and acceptance rates for submissions from non-English-speaking countries, even after controlling for potential confounders (Saposnik et al., 2014). Beyond language proficiency, Sayegh et al. (2024) find a modest association between writing style and publication recommendations in a simulated peer-review experiment. These findings motivate our controlled investigation of whether linguistic presentation similarly affects LLM-based peer review, particularly given evidence of linguistic biases in other LLM applications (Hofmann et al., 2024; Reusens et al., 2025; Deas et al., 2023).

5 Conclusion

In this work, we propose a controlled framework for investigating linguistic biases in LLM-based peer review. Our framework applies interventions to research papers along linguistic markers of epistemic authority, while preserving the underlying scientific content. Our experiments reveal substantial preferences of LLM reviewers toward rewrites using high-confidence language. Crucially, these preferences frequently result in decision flips across the accept↔reject boundary, directly affecting review outcomes. Beyond scores, we find that aspects raised in reviews vary in frequency and sentiment depending on linguistic features of the paper, potentially influencing final decisions.

Our findings raise important concerns for the use of LLMs in peer review. If content-equivalent papers receive different scores and review arguments solely because of linguistic properties, reviewer judgments do not exclusively reflect scientific quality. LLM reviewers should therefore be stress-tested on content-preserving rewrites to determine whether linguistic changes systematically affect their assessments. These tests should cover multiple linguistic dimensions and examine both overall scores and review arguments. In particular, reviewer systems should be checked for whether expressed confidence is treated as a signal of stronger evidence or scientific quality. Our framework enables these tests and comparisons of reviewer configurations by the stability of their assessments under controlled linguistic interventions. Finally, substantial differences in average scores and within-model variance caution against treating raw scores from individual LLM reviewers as directly comparable without calibration. Future work can extend these stress tests to additional linguistic and rhetorical dimensions.

AI use statement

In this work, we used generative AI tools to support the design and refinement of the experiments, to implement and edit software code, and to generate the LLM-based reviews and linguistic rewrites that constitute experimental data in our study. We additionally used generative AI tools for analyzing existing literature, identifying relevant literature, and editing the manuscript to improve readability and presentation. We did not use generative AI tools to formulate mathematical claims or assist in writing mathematical proofs, and tasks involving surveys, interviews, or transcription of research material are not applicable to this work. All AI-assisted research and writing activities were reviewed by the authors. We take responsibility for the final content of this work, including text, claims, code, experimental artifacts, and analyses produced with the aid of generative AI.

Ethics statement

This work does not advocate the use of LLMs as replacements for human peer reviewers. Instead, we develop a controlled framework to identify and characterize biases in LLM-based reviewing. Our study shows that linguistic properties of a paper can affect LLM-generated review outcomes even when the underlying research content is preserved. Such sensitivity raises fairness concerns, as linguistic variation associated with researchers’ backgrounds may influence how their work is evaluated.

Our findings also have potential for dual use. Identifying linguistic features that systematically influence LLM reviewers could be used to deliberately manipulate papers in order to obtain more favorable automated reviews. We do not intend to facilitate such manipulation. Rather, we use these findings to expose potential vulnerabilities of LLM-based reviewing and to motivate their careful evaluation before such systems are used in high-stakes peer-review settings.

Reproducibility Statement

All rewriting prompts are provided verbatim in Figures 8–15, and the reviewer prompts in Figures 6 and 7. The paper processing pipeline, the rewriting and reviewing models, and all generation settings are described in Section 2.4 and Appendix A; the variations of the reviewer prompt used in our robustness analysis are described in Appendix C. The sampling of the 504 ICLR 2026 papers is described in Section 2.4, and the rewrite validation measures in Section 2.2. Since several reviewer models do not support fixed sampling parameters, we quantify the variance of each reviewer across repeated generations (Table 3). We will release our code, the list of sampled papers, all rewrites and all generated reviews upon publication.

References

• Ädel and Erman (2012) A. Ädel and B. Erman Recurrent word combinations in academic writing by native and non-native speakers of english: a lexical bundles approach. English for specific purposes 31 (2), pp. 81–92. External Links: Cited by: §2.1, §4.
• Arous et al. (2026) I. Arous, N. N. Kennard, A. Mircea, E. Kuang, J. C. K. Cheung, and A. McCallum Hands-Off or Hands-On? Variation in Area Chair Practices and Implications for AI Support. arXiv preprint arXiv:2608.07425. Cited by: §1, §3.2.
• Baumann et al. (2026) J. Baumann, J. Pei, S. Koyejo, and D. Hovy Stop Automating Peer Review Without Rigorous Evaluation. In Forty-third International Conference on Machine Learning, Cited by: §1, §1, §1.
• Cheng et al. (2023) M. Cheng, E. Durmus, and D. Jurafsky Marked personas: using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1504–1532. Cited by: §2.1.
• Crossley and McNamara (2009) S. A. Crossley and D. S. McNamara Computational assessment of lexical differences in l1 and l2 writing. Journal of second language writing 18 (2), pp. 119–135. Cited by: §2.1, §2.2, §4.
• Daller et al. (2003) H. Daller, R. Van Hout, and J. Treffers-Daller Lexical richness in the spontaneous speech of bilinguals. Applied linguistics 24 (2), pp. 197–222. Cited by: §2.2.
• Deas et al. (2023) N. Deas, J. Grieser, S. Kleiner, D. Patton, E. Turcan, and K. McKeown Evaluation of african american language bias in natural language generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6805–6824. External Links: Link Cited by: §4.
• Demetrio et al. (2025) L. Demetrio, G. Apruzzese, K. Grosse, P. Laskov, E. Lupu, V. Rimmer, and P. Widmer Gen-Review: A Large-scale Dataset of AI-Generated (and Human-written) Peer Reviews. arXiv preprint arXiv:2510.21192. External Links: Link Cited by: §1, §4.
• Dycke and Gurevych (2026) N. Dycke and I. Gurevych Automatic reviewers fail to detect faulty reasoning in research papers: a new counterfactual evaluation framework. Transactions of the Association for Computational Linguistics 14, pp. 465–488. External Links: Link Cited by: §1, §1, §1.
• Gunning (1952) R. Gunning The technique of clear writing. McGraw-Hill, New York. Cited by: §2.2.
• Hinkel (2002) E. Hinkel Second language writers’ text: linguistic and rhetorical features. ESL & Applied Linguistics Professional Series, Lawrence Erlbaum Associates. Cited by: §1, §2.1.
• Hofmann et al. (2024) V. Hofmann, P. R. Kalluri, D. Jurafsky, and S. King AI generates covertly racist decisions about people based on their dialect. Nature 633 (8028), pp. 147–154. Cited by: §4.
• Huang et al. (2025) S. Huang, Q. Wang, W. Lu, L. Liu, Z. Xu, and Y. Huang PaperEval: a universal, quantitative, and explainable paper evaluation method powered by a multi-agent system. Information Processing & Management 62 (6), pp. 104225. Cited by: §4.
• Hyland and Milton (1997) K. Hyland and J. Milton Qualification and certainty in l1 and l2 students’ writing. Journal of second language writing 6 (2), pp. 183–205. Cited by: §2.1, §4.
• Hyland (1996) K. Hyland Writing without conviction? hedging in science research articles. Applied linguistics 17 (4), pp. 433–454. Cited by: §1, §2.1.
• Hyland (1998) K. Hyland Boosting, hedging and the negotiation of academic knowledge. Text & Talk 18 (3), pp. 349–382. Cited by: §2.1.
• Hyland (2005) K. Hyland Metadiscourse: exploring interaction in writing. Continuum, London & New York. Cited by: Appendix A.
• Jin et al. (2024) Y. Jin, Q. Zhao, Y. Wang, H. Chen, K. Zhu, Y. Xiao, and J. Wang AgentReview: Exploring Peer Review Dynamics with LLM Agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1208–1226. External Links: Link Cited by: §1, §4.
• Kaneko (2026) M. Kaneko Paraphrasing Adversarial Attack on LLM-as-a-Reviewer. arXiv. External Links: Link Cited by: §1, §1.
• Kousha and Thelwall (2026) K. Kousha and M. Thelwall Which stylistic features fool ChatGPT research evaluations?. Scientometrics 131, pp. 6497–6518. Cited by: §1, §4.
• Kunilovskaya et al. (2024) M. Kunilovskaya, K. D. Chowdhury, H. Przybyl, C. España-Bonet, and J. Genabith Mitigating translationese with gpt-4: strategies and performance. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pp. 411–430. External Links: Link Cited by: §2.1.
• Lee et al. (2006) K. P. Lee, E. A. Boyd, J. M. Holroyd-Leduc, P. Bacchetti, and L. A. Bero Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Medical Journal of Australia 184 (12), pp. 621–626. Cited by: §1.
• Lev-Ari and Keysar (2010) S. Lev-Ari and B. Keysar Why don’t we believe non-native speakers? the influence of accent on credibility. Journal of experimental social psychology 46 (6), pp. 1093–1096. Cited by: §1.
• Lin et al. (2025) T. Lin, W. Chen, T. Hsiao, H. Liu, Y. Yeh, Y. K. Chan, W. Lien, P. Kuo, P. S. Yu, and H. Shuai Breaking the reviewer: assessing the vulnerability of large language models in automated peer review under textual adversarial attacks. In Findings of the Association for Computational Linguistics: EMNLP 2025, External Links: Link Cited by: §1, §1.
• Liu et al. (2024) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12, pp. 157–173. External Links: Link Cited by: §1.
• Lu et al. (2025) S. Lu, I. Kuznetsov, and I. Gurevych Identifying aspects in peer reviews. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 6145–6167. External Links: Link Cited by: Appendix D, §1, §3.2, §3.2.
• McCarthy and Jarvis (2010) P. M. McCarthy and S. Jarvis MTLD, vocd-d, and hd-d: a validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods 42 (2), pp. 381–392. Cited by: §2.2.
• Politzer-Ahles et al. (2020) S. Politzer-Ahles, T. Girolamo, and S. Ghali Preliminary Evidence of Linguistic Bias in Academic Reviewing. Journal of English for academic purposes 47, pp. 100895. External Links: Link Cited by: §1, §4.
• Reusens et al. (2025) M. Reusens, P. Borchert, J. De Weerdt, and B. Baesens Native design bias: studying the impact of english nativeness on language model performance. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pp. 1195–1215. External Links: Link Cited by: §4.
• Saposnik et al. (2014) G. Saposnik, B. Ovbiagele, S. Raptis, M. Fisher, and S. C. Johnston Effect of English Proficiency and Research Funding on Acceptance of Submitted Articles to Stroke Journal. Stroke 45 (6), pp. 1862–1868. Cited by: §1, §4.
• Sayegh et al. (2024) G. E. Sayegh, A. E. Mercado, J. E. Thomas, D. Ring, and T. J. Crijns Is writing style associated with peer reviewer recommendations?. University of Pennsylvania Orthopaedic Journal 34, pp. 78–80. Cited by: §1, §4.
• Teigen (2023) K. H. Teigen Dimensions of uncertainty communication: what is conveyed by verbal terms and numeric ranges. Current Psychology 42 (33), pp. 29122–29137. Cited by: §1.
• Thakkar et al. (2026) N. Thakkar, M. Yuksekgonul, J. Silberg, A. Garg, N. Peng, F. Sha, R. Yu, C. Vondrick, and J. Zou A large-scale randomized study of large language model feedback in peer review. Nature Machine Intelligence 8 (3), pp. 326–336. Cited by: §4.
• Vassileva (2001) I. Vassileva Commitment and detachment in english and bulgarian academic writing. English for specific purposes 20 (1), pp. 83–102. Cited by: §2.1.
• Vasu et al. (2026) S. S. M. Vasu, I. Sheth, H. Wang, R. Binkyte, and M. Fritz Justice in Judgment: Unveiling (Hidden) Bias in LLM-Assisted Peer Reviews. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 307–330. External Links: Link Cited by: §1, §1, §4.
• Wataoka et al. (2024) K. Wataoka, T. Takahashi, and R. Ri Self-Preference Bias in LLM-as-a-Judge. In NeurIPS Safe Generative AI Workshop 2024, External Links: Link Cited by: §1.
• Xiao et al. (2023) C. Xiao, Y. Li, G. Hudson, C. Lin, and N. Al Moubayed Length is a Curse and a Blessing for Document-level Semantics. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1385–1396. External Links: Link Cited by: §2.2.
• Yang and Guo (2025) Y. Yang and X. Guo “We find that…” changing patterns of epistemic positioning in research writing. Frontiers in Psychology 16, pp. 1634848. Cited by: §2.1.
• Ye et al. (2025) J. Ye, Y. Wang, Y. Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P. Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. In International Conference on Learning Representations, Vol. 2025. Cited by: §1.
• Ye et al. (2024) R. Ye, X. Pang, J. Chai, J. Chen, Z. Yin, Z. Xiang, X. Dong, J. Shao, and S. Chen Are we there yet? revealing the risks of utilizing large language models in scholarly peer review. arXiv preprint arXiv:2412.01708. Cited by: §4.
• Zhu et al. (2025) M. Zhu, Y. Weng, L. Yang, and Y. Zhang Deepreview: improving llm-based paper review with human-like deep thinking process. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 29330–29355. External Links: Link Cited by: §4.

Appendix A Implementation Details

Paper processing. We convert each paper from PDF to a structured textual representation using MinerU (v3.4.4), which preserves the document layout so that tables and equations remain inline at their original positions. Before conversion, we cut each PDF after the references, so that no appendix reaches any model. We then split the paper into top-level sections and blind it, removing the title block with authors and affiliations, the venue watermark, page numbers and the line-number gutter. Figures are not passed to any model; their captions are kept as text.

To preserve the overall organization of the paper and avoid exceeding model context or output limits, we perform rewriting section by section. When rewriting a section, the model receives no context, to ensure rewrites are only stylistic and no additional content is added.

Confidence Indicators. To quantify expressed confidence, we measure the use of boosters (e.g., “clearly,” “demonstrates,” “certainly”) and hedges (e.g., “may” “suggests”, “likely”) drawn from Hyland’s metadiscourse taxonomy (Hyland, 2005). Each paper was segmented into chunks, normalized (lowercased, punctuation-stripped), and scanned for booster and hedge terms using word-boundary-anchored matching (“\b” in regular expressions) to ensure only whole-word occurrences were counted. We report frequency per 1,000 words for each category, with booster and hedge terms kept mutually exclusive to avoid double-counting.

Generation settings. All rewrites are produced with Claude Opus 5. For all rewriting and reviewing experiments, we use each model’s default sampling configuration; we do not set the temperature, since several of the models used do not accept a custom value, and fixing it for some models but not others would make the reviewers less comparable. For GLM 4.7, we explicitly enable reasoning; all other models run with their default reasoning behaviour. Each review is generated in a single call, with the system prompt in Figure 6 and the blinded paper in Markdown as the user message. We use the same generation settings across conditions.

Appendix B Further Rewrite Examples

Tables 6 and 7 show rewrites of two further passages from this paper, taken from the methodology and the results section, produced in the same way as Table 1.

Appendix C Prompt Sensitivity

To test whether our findings depend on the specific reviewer prompt, we re-ran Claude Sonnet 4.6 on Batch 1 (126 papers) with three neutral variations of our prompt: Brief (without the reviewer responsibilities, review criteria and rating-scale anchors), Minimal (additionally without the sentence on fair and substantive feedback), and Bare (additionally without the four per-criterion sub-scores). As a contrast, we also ran a Biased prompt that explicitly instructs the reviewer to place substantial weight on linguistic style and presentation. Default refers to the standard prompt used throughout this work (Figure 6).

Table 8 shows that the neutral variations leave our conclusions unchanged: the native vs. non-native contrast remains a null effect under every neutral prompt, and the high- vs. low-confidence contrast remains large and significant (p<0.0001). Only the explicitly biased prompt changes the picture, turning the native vs. non-native null result into a clear preference for native writing (4.3×, p<0.0001). Reviewers are thus insensitive to how the review task is worded, but do act on an explicit instruction to weight style.

Appendix D Rewrite Effects on Review Criteria

Apart from the overall rating, we also collect scores for the review criteria Soundness, Significance, Evidence and Clarity, each on a scale from 1 to 4. This allows us to analyze which aspects of a paper’s assessment are affected by the linguistic rewrites.

Preference ratios per review criterion. Figure 3 breaks down the preference ratios by review criterion. For language proficiency, the overall rating remains unaffected for all reviewers, but Clarity shows a small yet significant preference for native-associated writing for four of the six reviewers (Gpt-5.5, Claude Sonnet 4.6, Qwen3 VL and Glm4.7; ratios of 1.3–2.1×), while Soundness, Significance and Evidence remain largely unaffected. LLM reviewers thus register differences in language proficiency, but confine them to the criterion where they belong, without affecting the overall rating. Expressed confidence, in contrast, increases Soundness, Significance and Evidence for all six reviewers, i.e., confident framing is read as stronger science. Its effect on Clarity is model-dependent: Claude Sonnet 4.6 (10×), Glm4.7 (5×) and Gpt-5.5 (2.1×) rate high-confidence writing as clearer, whereas Gemma3, Qwen3 VL and Gemini 3.5 Flash show no significant effect.

Alignment with review aspects. We measure the correlation of scores for review criteria with the presence and sentiment of the review aspects Lu et al. (2025) we extract in Section 3.2. We find that negative Presentation mentions negatively correlate with Clarity (r=−0.42), and positive Result mentions with Evidence. Alignment with other criteria is weaker, but still sensible: Soundness correlates with positive Theory (r=+0.25) and Methodology mentions (r=−0.22), and Significance correlates modestly with positive Novelty (r=+0.19) and Contribution (r=+0.21) mentions.

Magnitude of score differences. The preferences we identify are driven by consistent shifts across many papers rather than by a few extreme score changes. For the overall rating, 88% of native/non-native pairs and 85% of high-confidence/low-confidence pairs differ by at most one point, and on the review criteria (scored on a 1–4 scale) fewer than 4% of pairs differ by two or more points. The confidence effects in Table 4 therefore reflect a systematic, directional shift in how reviewers assess the same paper, rather than occasional outliers. Larger differences of two or more points concentrate on Gemini 3.5 Flash and Glm4.7 (28–37% of pairs, compared to at most 9% for all other reviewers), the two reviewers with the highest variance across repeated generations (Table 3). Our framework cleanly separates this stochastic variation from genuine rewrite effects: the mean absolute score difference of these reviewers is nearly identical for both rewrite dimensions (1.20 vs. 1.23 for Gemini 3.5 Flash, 1.29 vs. 1.36 for Glm4.7), yet only the confidence rewrites induce a systematic preference. Comparing paired rewrites of the same paper thus reveals stylistic biases even for reviewers whose individual scores fluctuate considerably.

Appendix E Polarity of Review Aspects per LLM Reviewer

Figures 4 and 5 show the per-model paired prevalence shift of aspect mentions for native − non-native and high-confidence − low-confidence, respectively.

注意转载授权 · 阅读原文