研究：LLM 同行评审评分与人类评分的对齐随提示词、领域和稿件特征变化

[ BoldFont=texgyretermes-bold.otf, ItalicFont=texgyretermes-italic.otf, BoldItalicFont=texgyretermes-bolditalic.otf ]

   Renlong Jie ^††thanks: Corresponding author: Renlong Jie. Email: jierenlong@nwpu.edu.cn.    Yanjun Qian    Salma Srifi

Abstract

Large language model (LLM) scores are being studied as information for scholarly evaluation. Interpreting these scores requires evidence about agreement with human judgments and its variation across evaluation conditions. Guided by an argument-based approach to validation, we compare four LLMs on 7,083 public ICLR 2024 submissions under a standard review condition and examine ten prompt conditions using DeepSeek-V3.2. We distinguish average score divergence, paper-level absolute error, and rank correlation, then examine domain differences and adjusted manuscript-feature associations. For DeepSeek-V3.2, Area Chair guidance reduces mean rating divergence from 1.460 to 0.199 points and mean absolute error from 1.577 to 0.942, while rank correlation changes from 0.377 to 0.387. Historical references combined with downward scoring instructions yield mean divergence of 0.052 and rank correlation of 0.321. Domain differences remain after global calibration. Longer titles have positive adjusted associations with rating divergence, and formula density has negative associations across all four configurations. These findings connect score interpretation to the scoring procedure, evaluation context, and manuscript information represented in the comparison. They show why aggregate alignment, manuscript ordering, and correspondence within manuscript groups require distinct evidence when evaluating LLM-generated scholarly assessment information.

School of Management, Northwestern Polytechnical University, Xi’an, China

Keywords: scholarly evaluation, large language models, peer review, score interpretation, human judgments, manuscript characteristics.

Introduction

Scores condense evaluations into information for comparing manuscripts. Large language models (LLMs) can produce these scores from manuscript text and a rubric. Interpreting their correspondence with human ratings requires examining score levels, differences between individual papers, and variation across research areas and manuscript characteristics. These questions concern how scholarly assessment information is represented and interpreted.

Recent research has examined how LLMs evaluate scholarly work. Thelwall (2025) discusses the evaluation of research quality as a complex text task. Kousha and Thelwall (2025) compare ChatGPT evaluations of societal impact cases with departmental expert averages and find variation across fields. Wu et al. (2025) combine human review information and LLM knowledge to predict methodological novelty. Zhu et al. (2026) identify model-dependent language bias in academic evaluation. Together, these studies show that model-generated assessment depends on the information supplied and the evaluation procedure.

The link between a numerical result and its intended interpretation provides our theoretical starting point. In an argument-based approach to validation, evidence is evaluated against the inferences required for a particular score interpretation or use (Kane, 2013). Applied here, a close average score, a close score for each paper, and a similar ordering address different claims. Evidence about one property leaves the others to be examined. The research problem is therefore to establish how these properties change together across evaluation conditions.

Public OpenReview records link manuscripts to several human ratings (Tran et al., 2020). Meta-analytic evidence documents limited agreement among reviewers (Bornmann et al., 2010), and experimental evidence shows that author-identity information can affect recommendations (Tomkins et al., 2017). These findings motivate examining model-human correspondence alongside the reliability and conditions of the human reference.

We study 7,083 public ICLR 2024 submissions with three to five human reviews per paper. We compare GPT-5 mini, Gemini 3 Flash Preview, Claude Sonnet 4.5, and DeepSeek-V3.2 under a standard review condition. We also evaluate ten prompt conditions using DeepSeek-V3.2. Mean divergence, mean absolute error (MAE), and Spearman correlation describe complementary agreement properties. Global calibration examines domain variation, and double/debiased machine learning (DML) estimates adjusted manuscript-feature associations.

The results reveal different responses across these properties. Area Chair guidance reduces mean divergence and MAE, with a small change in rank correlation. Historical references combined with downward scoring instructions give the closest mean and a lower rank correlation. Differences among domains persist after global calibration. Title length and formula density have recurring positive and negative adjusted associations with signed rating divergence, respectively.

The study connects scoring procedures, evaluation contexts, and manuscript information within one empirical design. It extends prompt comparisons in AI reviewing (Drori and Te’eni, 2024) by examining score alignment, paper-level deviations, and ranking together. The domain and manuscript-feature analyses show where correspondence varies beneath aggregate agreement.

Theoretical Background and Research Questions

Score Interpretation in Scholarly Evaluation

Validity concerns whether evidence supports a proposed interpretation and the actions based on it. Messick (1995) links assessment evidence to score meaning and its consequences. Kane (2013) develops an argument-based approach that makes the required inferences and assumptions explicit. It also separates support for an interpretation from support for a consequential use. We apply this perspective to model-generated scholarly scores by specifying the claims that each empirical comparison addresses.

We use score correspondence to describe the relationship between a model-generated rating and recorded human ratings of the same manuscript. Level agreement concerns signed differences on a common scale. Paper-level agreement concerns absolute distance from the human mean. Rank correspondence concerns manuscript ordering. Each property supports a different interpretation of correspondence with recorded judgments.

Agreement and correlation address different properties of paired measurements (Bland and Altman, 1986). Positive and negative deviations can cancel in the mean, while similar rankings can coexist with systematic score offsets. A positive affine transformation changes location and spread while preserving ranks. We therefore assess mean divergence, MAE, and rank correlation together when comparing scoring instructions.

The reference also matters. Kousha and Thelwall (2025) uses departmental average impact scores. Thelwall and Jiang (2025) compare citation indicators against ChatGPT scores and departmental expert averages. Zheng et al. (2026) compare human and LLM classifications of critical scholarly tweets. Here, manuscript-level human means and pairwise human disagreement define the comparison.

Scoring Procedures and Evaluation Contexts

Prompt instructions are part of the scoring procedure. They specify the role, evaluation criteria, and rules for translating an assessment into a number. Instructions that request conservative ratings or provide historical score patterns can alter score levels. Changes in ordering depend on how the instructions affect distinctions among papers. This reasoning motivates examining several properties together when comparing prompts.

Drori and Te’eni (2024) study cumulative GPT-4 reviewing instructions. Thelwall (2025) emphasizes systematic testing and repeated outputs. Work on model families and prompting further examines scoring procedures (Thelwall and Mohammadi, 2026). Our paired design compares complete instruction bundles across ten DeepSeek conditions on the same submissions.

Evaluation contexts differ in manuscript evidence and review practices. Domains here are topic areas within one machine-learning conference. Overall calibration applies the same transformation to every paper, leaving domain-level correspondence to be examined. Human-review reliability also varies across studies and review conditions (Bornmann et al., 2010). Reporting within-domain human consistency helps interpret these comparisons.

Zhu et al. (2026) find opposite language preferences across two models evaluating humanities and social-science papers. Our domain comparison examines topic differences among real submissions after aggregate calibration, alongside human-reference consistency.

Manuscript Information and Conditional Correspondence

Document characteristics have an established place in research evaluation. Kousha and Thelwall (2024) review associations of titles, abstracts, document length, references, and author information with citations and expert quality judgments. Many relationships depend on the corpus and field. This makes manuscript features relevant to conditional evaluation, while keeping citation impact, expert judgment, and model-human divergence conceptually distinct.

Wu et al. (2025) combine human review statements and model-derived method summaries for novelty prediction. Research on long contexts and controlled presentation changes examines model responses to supplied text (Liu et al., 2024; Du, 2025; Yang et al., 2026). We examine which manuscript characteristics are associated with differences between model and human ratings after adjustment. This connects feature-based research assessment to correspondence between evaluation procedures.

Titles and formulas carry substantive content and presentation choices. We compare observed manuscript groups after adjustment for covariates and a full-text predictor of human ratings. Overlap weighting emphasizes manuscripts with comparable adjustment variables. The contrast describes signed divergence conditional on this information. A negative contrast indicates lower divergence in the compared group. Its implication for absolute agreement depends on the score offset and individual errors. Recurring signs across models identify groups for further validation.

Research Questions and Evidence Map

We examine three questions derived from these interpretive requirements:

1. RQ1:

   How do score levels, paper-level agreement, and rank correspondence vary across prompt conditions, relative to disagreement among human reviewers?

2. RQ2:

   How does model-human correspondence vary across domains after aggregate scale alignment, and across human-rating groups?

3. RQ3:

   Which adjusted associations between manuscript characteristics and signed rating divergence recur across model configurations?

Table 1 maps the questions to their evidence. Table 2 positions the study relative to the closest research. The framework guides interpretation of the analyses.

Research Design

The design connects paired manuscript scores to the three research questions. Prompt comparisons examine the scoring procedure, domain comparisons examine contextual correspondence, and adjusted feature comparisons examine differences among observed manuscript groups. Figure 1 summarizes this workflow.

Data, Human Reference, and Model Configurations

The study uses retrospective public records as a setting for evaluating model-generated assessment information. We obtained approximately 7,400 public ICLR 2024 submission PDFs and their review records from OpenReview. We retained papers with three to five valid human reviews, yielding 7,083 papers. GROBID parsed the PDFs into structured text. We used this output to extract titles, abstracts, manuscript bodies, formulas, figures, and tables (Lopez, 2009; GROBID contributors, 2008). Human reviews included soundness, presentation, contribution, overall rating, and reviewer confidence. The first four concern manuscript evaluation. Confidence records the reviewer’s self-assessment.

We compared GPT-5 mini, Gemini 3 Flash Preview, Claude Sonnet 4.5, and DeepSeek-V3.2 under the standard review condition. The protocol specified requested temperature 0.1 and top-p 0.95. Appendices A-G describe the prompt design, extended results, diagnostics, and implementation.

We used each paper’s mean human score as the reference for each dimension. Throughout the study, bias denotes directional AI-human divergence. We linked author characteristics to manuscript records after data collection. Author variables enter secondary comparisons and adjustment. The country-based variable labeled Native English is a proxy based on the listed country group, with the coding shown in Table 3.

Measurement of Evaluation Properties

Let Ai​d be the AI score for paper i and dimension d, Hi​d​r the score of human reviewer r, H¯i​d its human mean, and mi​d the number of valid reviews. Directional divergence and absolute error are

We also measure disagreement on an individual-reviewer basis:

Here, Nd is the number of papers with valid scores on dimension d. DdA​H averages disagreement between the AI score and each human score. DdH​H averages disagreement among all pairs of human reviewers. Comparing them places AI-human disagreement on the same individual-reviewer basis as human-human disagreement. We measured relative discrimination using Spearman correlation between Ai​d and H¯i​d. We applied Benjamini-Hochberg adjustment within the reported test families (Benjamini and Hochberg, 1995).

Prompt Comparisons and Contextual Heterogeneity

We tested ten DeepSeek conditions: five cumulative structured prompts and five tone/depth variants. V1 supplied the paper and review form. V2 added reviewer guidelines, and V3 added conduct and ethics guidance. V4 added Area Chair selectivity, including choosing the lower rating when undecided. V5 added historical score patterns and a scoring procedure: form a preliminary rating, then usually lower it by about one point, or up to two for noticeable weaknesses. The tone variants used the same numeric scales with a different section order, including Questions and Ethics flag and details. V4 and V5 each changed several instructions. Appendices A and G document these instruction bundles and their implementation.

We evaluated the same papers under each prompt. We used Friedman tests for bias and absolute error, Cochran’s Q for directional rates, and permutation tests for rank-correlation differences. False discovery rate (FDR) adjustment was applied within each dimension. The standard condition had the highest observed DeepSeek rank correlation and was selected for the cross-model analyses. Selection and evaluation used the same papers, so the reported comparison describes performance within this sample.

We assessed the human intraclass correlation coefficient (ICC(1,k)), mean divergence, MAE, and rank correlation by domain. Human consistency was examined in the 12 primary areas with at least 200 papers (Figure C1). The seven-area model comparison covered a subset of 3,684 papers from the 7,083 submissions. We also calibrated raw model scores to separate differences in score location and scale from remaining domain variation:

For nondegenerate scores, a>0, so this transformation preserves ranks. We estimated the parameters from the analyzed sample. The overall mean and standard deviation are aligned by construction. The transformation is a within-sample diagnostic for remaining domain differences. Area-model cell analyses used sample-size weights, nested model comparisons, a one-degree-of-freedom non-additivity test, and wild-bootstrap inference. This analysis decomposes score-scale and domain differences across 28 area-model cells. The four configurations evaluate overlapping sets of papers within each area. We treat the cell-level inferential decomposition as exploratory and emphasize the reported area-specific means and rank correlations.

We rounded each paper’s mean human rating to the nearest integer and limited it to the 1-10 scale. We then defined low (1-4), middle (5-6), and high (7-10) groups. Figure 3 uses the labels Low, Middle, and High for these groups. Their membership was determined by ratings. For the human mean Hi=H¯i,rating, the identity 𝔼⁡[Δi∣Hi=h]=𝔼⁡[Ai∣Hi=h]−h shows that conditional divergence depends on the reference score itself. We use these strata to describe the sample, with mathematical coupling and within-stratum range restriction considered in interpretation. We tested bias differences using analysis of variance (ANOVA) and Kruskal-Wallis tests. Correlation heterogeneity was assessed using Fisher-z-based Cochran tests and I2.

Adjusted Manuscript-Feature Comparisons

We first used ordinary least squares (OLS) to relate overall rating divergence Δi to author and manuscript characteristics (Table 3). We calculated heteroskedasticity-consistent standard errors using the HC3 estimator. The ex-ante specification included author and manuscript variables. The ex-post specification added human-rating dispersion and reviewer count. Missing continuous values were imputed as zero in the OLS analysis.

DML added adjustment for a hierarchical full-text SciBERT predictor of mean human ratings (Beltagy et al., 2019; Chernozhukov et al., 2018). This predictor summarizes textual information associated with human evaluation. Its target is the mean human rating. We obtained H^i through five-fold out-of-fold prediction at the paper level, using the title, abstract, and main body. We excluded appendices and references. Appendix F describes the architecture and estimation score.

We compared each characteristic in turn. Binary variables retained their coding. For other variables, we compared the upper and lower terciles and excluded the middle tercile. Author-characteristic comparisons adjusted for the proxy, manuscript variables, review-condition controls, and primary-area indicators. Manuscript-characteristic comparisons adjusted for the proxy, author variables, review-condition controls, and area indicators. Thus, each comparison adjusted for characteristics in the other block.

Let T identify the high group for the characteristic being compared, and let X denote its adjustment variables. The propensity e⁡(X)=Pr⁡(T=1∣X) is the probability of belonging to the high group given X. The outcome function mt(X)=𝔼[Δ∣T=t,X] is expected rating divergence for group t. The average treatment effect for the overlap population (ATO) uses overlap weights (Li et al., 2018):

The propensity multiplied by its complement gives greater emphasis to papers whose observed characteristics are compatible with both groups. Each manuscript characteristic defines its own overlap population, so comparisons across rows concern different populations. We used the ATO-specific rate-doubly-robust estimator to account for the dependence of these weights on the propensity (Wang et al., 2025). Ensemble nuisance models used five-fold paper-level cross-fitting, repeated across ten partitions. We reported median estimates with paper-clustered influence-function inference and adjustment for variation across partitions. FDR adjustment was applied separately within each model and author/manuscript family. Appendix F gives the estimating score, training settings, and clustered inference.

We interpret θATO as an adjusted difference in rating divergence between naturally occurring high and low groups. The contrast is expressed in rating points on the 1-10 scale. Adjustment conditions on the observed covariates and human-rating predictor used in each specification. Titles and formulas carry both substantive information and presentation choices, which jointly enter these group comparisons.

Results

Score Divergence Relative to Human Disagreement

Table 4 reports the V1 DeepSeek baseline. AI-human disagreement exceeds human-human disagreement for soundness, contribution, and rating, with ratios of 1.11, 1.31, and 1.31, respectively (q<0.001). Presentation disagreement is comparable to human disagreement (ratio 1.00, q=0.470). Mean divergence is positive on every dimension. Overall rating divergence is 1.460 points, with a standardized effect of 1.26. Confidence scores show less AI-human disagreement than human-human disagreement, describing agreement in reported certainty.

Prompt Conditions and Distinct Agreement Properties

Table 5 compares the ten DeepSeek conditions on the same 7,083 papers. V1-V3 show large positive mean divergence and upward-scoring rates above 85%. V4 has the lowest MAE, 0.942, and mean divergence of 0.199, compared with 1.577 and 1.460 for V1. Its rank correlation is 0.387, close to V1’s 0.377. The Area Chair instructions substantially improve score alignment, alongside a small change in rank agreement.

V5 has the smallest mean divergence, 0.052, and the lowest rank correlation, 0.321. Its MAE is 0.961. These results reflect the combined historical reference values and downward scoring instructions in V5. The standard condition has the highest rank correlation, 0.423, and a larger MAE of 1.464. The critical condition lowers MAE to 1.115 with a correlation of 0.403. Domain-expert and quick instructions yield correlations of 0.415 and 0.407, with MAE near 1.49.

Table 6 summarizes variation across dimensions, with omnibus tests in Table B5. Prompt variation is greater for manuscript evaluation than for self-reported confidence.

The omnibus tests identify differences in bias, MAE, and correlation across prompt conditions. For overall rating, the ranges are 1.408 for bias, 0.635 for MAE, and 0.102 for correlation. Evaluation properties thus respond differently to the instruction bundles.

Correspondence Across Domains and Rating Groups

Human rating consistency varies across the 12 domains with at least 200 papers (Figure C1). Paper-level absolute human disagreement also varies across all five dimensions (Table C1). ICC is 0.725 for Generative Models and 0.720 for Reinforcement Learning, compared with 0.601 for Societal AI Ethics and 0.614 for Datasets and Benchmarks. Domain comparisons therefore need to consider variation in the human reference alongside model performance.

Figure 2 shows model results after global affine calibration. Generative Models and Reinforcement Learning have small absolute mean divergence, at most 0.11. All four models show negative divergence in Learning Theory, ranging from −0.36 to −0.62, alongside relatively strong rank agreement. Societal AI Ethics shows positive divergence and weaker rank agreement. These domain patterns persist after correcting global score location and scale.

Table C2 reports the full area-model cell analysis. The cell-level decomposition summarizes area and model components. Its inferential results are exploratory because the model cells share manuscripts within areas. Rank correlations are unchanged by the positive affine transformation. Global score calibration therefore leaves domain differences in divergence. Table C5 describes the fields’ manuscript characteristics, including formula and appendix use.

Figure 3 shows results by human-rating group. All four configurations have positive divergence of approximately 2.40-3.22 points in the low group and 1.19-1.73 in the middle group. Model-specific directions differ in the high group. Within-group rank correlations decline toward the high group. Table C4 reports heterogeneity tests. Human-score standard deviations also decline, from 0.603 in the low group to 0.555 in the middle group and 0.469 in the high group (Table C3). Group membership uses the human score that also enters rating divergence. The group patterns therefore reflect conditional score comparisons, mathematical coupling, and range restriction alongside model performance.

Manuscript-Feature Associations Across Models

The OLS models explain different shares of rating divergence. R2 is 0.339 for Claude, 0.209 for DeepSeek, 0.092 for Gemini, and 0.106 for GPT. Title length is positively associated with divergence in all four configurations. Formula density, appendix ratio, and abstract length are negatively associated. Adding human-rating dispersion and reviewer count increases R2 by at most 0.002. Tables D1 and D2 report the full specifications.

The full-text SciBERT proxy predicts mean human rating with out-of-fold MAE 0.790, R2=0.369, Pearson r=0.614, and Spearman ρ=0.607. After overlap weighting, maximum absolute standardized mean differences are below 0.06 for every compared characteristic. Figure 4 shows these diagnostics by characteristic. Appendix ratio, citation density, and formula density show greater separation in propensity scores, yet their weighted covariate differences remain below 0.06. Effective sample sizes also vary because each comparison concerns a different overlap population.

Table 7 reports the adjusted ATO contrasts. Title-length estimates are positive: 0.913, 0.828, 0.742, and 0.755 for Claude, DeepSeek, Gemini, and GPT. Formula-density estimates are negative: −0.097, −0.169, −0.139, and −0.291. All eight contrasts remain significant after FDR adjustment. For title length, these values compare the upper and lower terciles of title character counts. Formula-density values compare the corresponding terciles of formulas per 1,000 main-text characters. The differences are conditional associations in each characteristic-specific overlap population.

Appendix ratio and top-institution affiliation have negative estimates in all four configurations and are significant in three. Author count and log body length are negative in all four and significant in two. Industry involvement changes sign between Claude and GPT. Estimates for first-author student status, female-author share, table-information density, and average formula length are statistically indistinguishable from zero after FDR adjustment. These secondary comparisons describe the wider pattern of manuscript-group differences. The main recurring manuscript associations concern title length and formula density.

We repeated the analysis with a 512-token SciBERT proxy and with primary-area adjustment omitted. Title-length and formula-density contrasts retain their signs and FDR significance across all four configurations. Appendix E reports all characteristics under both specifications (Tables E1 and E2) and diagnostics for the specification without area adjustment (Figure E1). Significance varies for secondary characteristics. The two focal contrasts remain consistent across specifications.

Discussion

The findings answer the three research questions through different properties of score correspondence. Prompt comparisons show that closer scoring levels can accompany little change or a decrease in rank correlation. Domain comparisons identify differences that remain after aggregate calibration. Feature comparisons locate recurring signed differences among observed manuscript groups. Together, these results make the interpretation of an LLM score conditional on the scoring procedure, evaluation context, and manuscript information represented by the evidence.

Scoring Rules and the Meaning of Agreement

RQ1 shows why selecting a prompt requires specifying the evaluation property of interest. V4 reduces DeepSeek’s mean divergence from 1.460 to 0.199 and MAE from 1.577 to 0.942. Rank correlation changes from 0.377 to 0.387. V5 brings mean divergence to 0.052 and yields a lower correlation of 0.321. The standard condition produces the highest observed correlation, 0.423, alongside larger absolute deviations. Different prompts therefore offer different combinations of correspondence on the same papers.

These comparisons extend the prompt-based evidence of Drori and Te’eni (2024) and complement the procedural concerns described by Thelwall (2025). V5 contains an explicit downward scoring instruction as well as historical references. Its proximity to the human mean demonstrates the behavior of that scoring rule. V4’s lower MAE supplies additional paper-level evidence. A choice among these procedures should follow the intended interpretation of the scores, with each relevant property evaluated directly.

The prompt results give practical significance to the distinction between agreement and correlation (Bland and Altman, 1986). A prompt chosen for average alignment can differ from one chosen for ordering. Independent validation should therefore evaluate the properties required by the intended assessment task.

Contextual Evidence After Aggregate Alignment

RQ2 concerns whether aggregate correspondence describes the domains represented in the sample. Global affine calibration aligns overall location and spread by construction and preserves ranks. All four configurations nevertheless show negative mean divergence in Learning Theory. Societal AI Ethics combines positive divergence with weaker rank agreement. This pattern identifies contextual variation that a single aggregate adjustment leaves in place.

The result complements field differences in societal-impact evaluation (Kousha and Thelwall, 2025) and model-dependent language differences (Zhu et al., 2026). Our contribution is the joint examination of domain-level score differences, rank correspondence, and human consistency after aggregate alignment.

Domain ICC describes the reliability of the human mean used for comparison. This complements evidence of limited reviewer agreement (Bornmann et al., 2010) and variation in review focus associated with reviewer confidence (Xie et al., 2026). Model-human correlations should therefore be interpreted alongside human consistency in the relevant domain.

Manuscript Representation and Recurring Differences

RQ3 identifies two recurring associations. Longer titles have positive adjusted contrasts in signed rating divergence across all four configurations. Formula density has negative contrasts. Both retain their directions and FDR significance under the alternative text proxy and area-adjustment specifications. They identify manuscript groups for which a single overall agreement summary is incomplete.

The findings connect document-feature research (Kousha and Thelwall, 2024) to differences between evaluation procedures. The outcome here is signed model-human divergence. This complements novelty prediction (Wu et al., 2025) and controlled presentation studies (Du, 2025; Yang et al., 2026). Our comparisons retain the content variation of real submissions, with recurring associations across models identifying manuscript groups for validation.

The contrasts condition on observed variables and text information associated with human ratings. Each feature defines a different comparison and overlap population. Author-identity effects in human review (Tomkins et al., 2017) also show why divergence can reflect differences arising in either evaluation procedure.

Implications for Scholarly Assessment Information

The Leiden Manifesto places quantitative evidence within expert-led, contextual research evaluation (Hicks et al., 2015). Its extension to LLMs emphasizes evaluation purpose and transparency (Thelwall, 2026). Our findings give these principles a concrete application: report signed differences and MAE for score levels and individual distances, rank evidence for ordering, and domain or manuscript-group results for the scope of correspondence. Describing the human reference makes these interpretations explicit.

The implications concern validation design. Studies of authorized assessment settings can test whether displaying model scores improves human judgments or decisions. Such studies should establish the added value of the score, including how people respond to disagreement. Similar error patterns across models also make error dependence relevant before their scores are combined. Operational use requires appropriate authorization, confidentiality protection, and human responsibility (Kankanhalli, 2024; Shmueli and Ray, 2024).

Limitations and Further Research

The evidence concerns public ICLR 2024 submissions and four model configurations. Prompt comparisons use DeepSeek, and V4 and V5 change several instructions together. Standard-prompt selection and affine calibration use the analyzed sample. Independent years, venues, and model-by-prompt comparisons would establish the stability of the reported patterns. Model-generated confidence is a statement of certainty, whose calibration would require separate evaluation.

Human ratings contain noise and possible systematic bias. They define the recorded review-process reference in this study. Human-rating strata also condition on a component of the divergence outcome, introducing mathematical coupling and range restriction. The cell-level domain tests involve overlapping manuscript sets across models. These analyses provide descriptive and exploratory evidence alongside the domain-specific estimates.

Feature comparisons depend on observed adjustment variables, the generated human-rating proxy, and the corresponding overlap populations. The influence-function standard errors are conditional on that proxy and do not separately propagate uncertainty from its training. Dependence between the two learning stages also affects inference. Controlled manuscript changes and appropriately nested estimation would strengthen causal and inferential conclusions.

PDF extraction can affect the mathematical and tabular information supplied to a model. Public records may have been available during model development. Further work can document input coverage, use newly released or explicitly authorized manuscripts, and repeat scoring calls to quantify output variability. The consequences of using scores for editorial or research-assessment decisions require direct studies of those uses.

Conclusion

LLM-human score correspondence changes across prompts, domains, and manuscript groups. Area Chair guidance reduces average divergence and absolute error, while historical references with downward scoring instructions produce closer mean scores and lower rank correlation. Domain differences persist after global calibration. Title length and formula density have recurring adjusted associations with signed divergence across four model configurations. These findings connect the interpretation of scholarly assessment information to the procedure that generates it and the context in which it is compared. Evaluating LLM scores therefore requires evidence suited to the intended claim about score levels, manuscript ordering, or correspondence within manuscript groups.

References

• Beltagy et al. (2019) I. Beltagy, K. Lo, and A. Cohan SciBERT: a pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3615–3620. External Links: Document, Link Cited by: Adjusted Manuscript-Feature Comparisons.
• Benjamini and Hochberg (1995) Y. Benjamini and Y. Hochberg Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological) 57 (1), pp. 289–300. External Links: Document, Link Cited by: Measurement of Evaluation Properties.
• Bland and Altman (1986) J. M. Bland and D. G. Altman Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 327 (8476), pp. 307–310. External Links: Document Cited by: Score Interpretation in Scholarly Evaluation, Scoring Rules and the Meaning of Agreement.
• Bornmann et al. (2010) L. Bornmann, R. Mutz, and H. Daniel A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLOS ONE 5 (12). External Links: Document Cited by: Introduction, Scoring Procedures and Evaluation Contexts, Contextual Evidence After Aggregate Alignment.
• Chernozhukov et al. (2018) V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal 21 (1), pp. C1–C68. External Links: Document, Link Cited by: Adjusted Manuscript-Feature Comparisons.
• Drori and Te’eni (2024) I. Drori and D. Te’eni Human-in-the-loop AI reviewing: feasibility, opportunities, and risks. Journal of the Association for Information Systems 25 (1), pp. 98–109. External Links: Document, Link Cited by: Introduction, Scoring Procedures and Evaluation Contexts, Table 2, Scoring Rules and the Meaning of Agreement.
• Du (2025) S. Du TitleTrap: probing presentation bias in LLM-based scientific reviewing. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems, pp. 119–125. External Links: Document, Link Cited by: Manuscript Information and Conditional Correspondence, Manuscript Representation and Recurring Differences.
• GROBID contributors (2008) GROBID contributors GROBID. GitHub. Note: https://github.com/grobidOrg/grobid External Links: Link, 1:dir:dab86b296e3c3216e2241968f0d63b68e8209d3c Cited by: Data, Human Reference, and Model Configurations.
• Hicks et al. (2015) D. Hicks, P. Wouters, L. Waltman, S. de Rijcke, and I. Rafols Bibliometrics: the Leiden Manifesto for research metrics. Nature 520 (7548), pp. 429–431. External Links: Document Cited by: Implications for Scholarly Assessment Information.
• Kane (2013) M. T. Kane Validating the interpretations and uses of test scores. Journal of Educational Measurement 50 (1), pp. 1–73. External Links: Document Cited by: Introduction, Score Interpretation in Scholarly Evaluation.
• Kankanhalli (2024) A. Kankanhalli Peer review in the age of generative AI. Journal of the Association for Information Systems 25 (1), pp. 76–84. External Links: Document, Link Cited by: Implications for Scholarly Assessment Information.
• Kousha and Thelwall (2024) K. Kousha and M. Thelwall Factors associating with or predicting more cited or higher quality journal articles: an Annual Review of Information Science and Technology (ARIST) paper. Journal of the Association for Information Science and Technology 75 (3), pp. 215–244. External Links: Document Cited by: Manuscript Information and Conditional Correspondence, Table 2, Manuscript Representation and Recurring Differences.
• Kousha and Thelwall (2025) K. Kousha and M. Thelwall Assessing the societal influence of academic research with ChatGPT: impact case study evaluations. Journal of the Association for Information Science and Technology 76 (10), pp. 1357–1373. External Links: Document Cited by: Introduction, Score Interpretation in Scholarly Evaluation, Table 2, Contextual Evidence After Aggregate Alignment.
• Li et al. (2018) F. Li, K. L. Morgan, and A. M. Zaslavsky Balancing covariates via propensity score weighting. Journal of the American Statistical Association 113 (521), pp. 390–400. External Links: Document Cited by: Adjusted Manuscript-Feature Comparisons.
• Liu et al. (2024) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, pp. 157–173. External Links: Document, Link Cited by: Manuscript Information and Conditional Correspondence.
• Lopez (2009) P. Lopez GROBID: combining automatic bibliographic data recognition and term extraction for scholarship publications. In Research and Advanced Technology for Digital Libraries, M. Agosti, J. Borbinha, S. Kapidakis, C. Papatheodorou, and G. Tsakonas (Eds.), Vol. 5714, pp. 473–474. External Links: Document Cited by: Data, Human Reference, and Model Configurations.
• Messick (1995) S. Messick Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist 50 (9), pp. 741–749. External Links: Document Cited by: Score Interpretation in Scholarly Evaluation.
• Shmueli and Ray (2024) G. Shmueli and S. Ray Reimagining the journal editorial process: an AI-augmented versus an AI-driven future. Journal of the Association for Information Systems 25 (1), pp. 60–75. External Links: Document, Link Cited by: Implications for Scholarly Assessment Information.
• Thelwall and Jiang (2025) M. Thelwall and X. Jiang Is OpenAlex suitable for research quality evaluation and which citation indicator is best?. Journal of the Association for Information Science and Technology 76 (12), pp. 1660–1681. External Links: Document Cited by: Score Interpretation in Scholarly Evaluation.
• Thelwall and Mohammadi (2026) M. Thelwall and E. Mohammadi Can small and reasoning large language models score journal articles for research quality and do averaging and few-shot help?. Scientometrics. External Links: Document, Link Cited by: Scoring Procedures and Evaluation Contexts.
• Thelwall (2025) M. Thelwall ChatGPT for complex text evaluation tasks. Journal of the Association for Information Science and Technology 76 (4), pp. 645–648. External Links: Document Cited by: Introduction, Scoring Procedures and Evaluation Contexts, Scoring Rules and the Meaning of Agreement.
• Thelwall (2026) M. Thelwall Large language models and responsible research evaluation: an extension of the leiden manifesto. Scientometrics. External Links: Document, Link Cited by: Implications for Scholarly Assessment Information.
• Tomkins et al. (2017) A. Tomkins, M. Zhang, and W. D. Heavlin Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences 114 (48), pp. 12708–12713. External Links: Document, Link Cited by: Introduction, Manuscript Representation and Recurring Differences.
• Tran et al. (2020) D. Tran, A. Valtchanov, K. Ganapathy, R. Feng, E. Slud, M. Goldblum, and T. Goldstein An open review of OpenReview: a critical analysis of the machine learning conference review process. Note: arXivPreprint External Links: 2010.05137, Link Cited by: Introduction.
• Wang et al. (2025) Y. Wang, Y. Liu, and S. Yang Rate doubly robust estimation for weighted average treatment effects. Journal of Causal Inference 13 (1), pp. 20240073. External Links: Document Cited by: Appendix F, Adjusted Manuscript-Feature Comparisons.
• Wu et al. (2025) W. Wu, C. Zhang, and Y. Zhao Automated novelty evaluation of academic paper: a collaborative approach integrating human and large language model knowledge. Journal of the Association for Information Science and Technology 76 (11), pp. 1452–1469. External Links: Document Cited by: Introduction, Manuscript Information and Conditional Correspondence, Table 2, Manuscript Representation and Recurring Differences.
• Xie et al. (2026) W. Xie, P. Jia, and X. Wang Reviewer self-reported confidence and evaluation breadth/depth in open peer review. Journal of the Association for Information Science and Technology 77 (6), pp. 863–885. External Links: Document Cited by: Table 2, Contextual Evidence After Aggregate Alignment.
• Yang et al. (2026) X. Yang, Z. Sha, J. Li, J. Yu, Y. Sun, M. Zhao, J. Fang, X. Guo, Y. Wu, X. Hu, Y. Luo, Q. Liu, and Z. Wang No hidden prompts needed! you can game AI peer review with presentation-only revisions. External Links: 2606.13044, Document, Link Cited by: Manuscript Information and Conditional Correspondence, Manuscript Representation and Recurring Differences.
• Zheng et al. (2026) E. Zheng, H. Fu, M. Thelwall, and Z. Fang Can social media provide early warning of retraction? evidence from critical tweets identified by human annotation and large language models. Journal of the Association for Information Science and Technology 77 (4), pp. 624–639. External Links: Document Cited by: Score Interpretation in Scholarly Evaluation.
• Zhu et al. (2026) Y. Zhu, Y. Jia, Y. Zhu, and J. Ye Does language bias GenAI academic evaluation in humanities and social sciences? a mixed-methods study based on Chinese-language HSS papers. Journal of the Association for Information Science and Technology 77 (7), pp. 940–958. External Links: Document Cited by: Introduction, Scoring Procedures and Evaluation Contexts, Table 2, Contextual Evidence After Aggregate Alignment.

Appendices

Interpreting LLM Scores in Peer Review: Alignment and Divergence Across Prompts, Domains, and Manuscript Features

These appendices report the prompt specifications, extended results, human-reference comparisons, regression estimates, sensitivity analyses, and estimation details. Citations refer to the preceding reference list.

Appendix A Prompt Design

The V1-V5 conditions combine structured review instructions with reviewer guidelines, ethics guidance, selectivity, and score adjustment. The tone and depth conditions use the same numerical scales in a separate review form. Table A1 summarizes the ten conditions. Appendix G specifies the scales, review forms, and distinctive instructions.

Appendix B Extended Prompt Comparisons

Tables B1-B4 report DeepSeek-V3.2 comparisons for Soundness, Presentation, Contribution, and Confidence. Table 5 reports Rating and exact agreement.

Appendix C Human Reference and Contextual Heterogeneity

The area-model cell decomposition is reported as exploratory. Model-specific cells within an area use overlapping manuscripts. The descriptive domain means and correlations are interpreted separately from the cell-level significance tests.

Figure C1 reports human rating consistency in the 12 areas with at least 200 papers. Table C1 compares absolute disagreement among human reviewers across these areas. Table C2 decomposes model and area variation in 28 cells covering 3,684 papers and 14,736 model-paper scores. Its partial effect sizes summarize weighted cell variation. Tables C3 and C4 report dispersion and heterogeneity within the three rating bins. Table C5 describes manuscript structure in the seven displayed areas. Its appendix/main-text ratio uses a different denominator from the appendix/total ratio in the manuscript-level regressions.

Appendix D Complete OLS Associations

Table D1 reports the ex-ante OLS specification. Table D2 adds human-rating dispersion and reviewer count. Both use 7,083 papers per model, and their R2 values differ by at most 0.002. OLS coefficients describe changes per unit of each variable. DML estimates compare high and low groups within a characteristic-specific overlap population. These different scales and populations govern the interpretation of each estimate. Standard errors are HC3 robust, and OLS significance marks use unadjusted p-values.

Appendix E Complete DML Sensitivity Analyses

Table E1 omits primary-area indicators. Table E2 uses a 512-token SciBERT proxy based on the title, abstract, and introduction. Both tables report all manuscript and author characteristics. Title-length and formula-density contrasts retain their directions and FDR significance across the four models under both specifications. The remaining contrasts show varying sensitivity to the controls. Changing the proxy changes the information used for adjustment and may change the overlap population. Figure E1 reports diagnostics for the specification without area indicators.

Appendix F Human-Rating Proxy and Estimation Implementation

Full-Text Human-Rating Proxy

Table F1 specifies the hierarchical human-rating proxy. SciBERT encodes overlapping text segments. The final-layer representation of each segment’s first token receives a learned position embedding and enters a document-level Transformer. Masked mean pooling combines these representations for the regression head. Epoch selection and target scaling use the outer-training sample. Predictions summarize full-text information associated with human ratings.

Orthogonal Score and Clustered Inference

For each characteristic, Ti identifies the two comparison groups and Xi contains the adjustment variables. Let e⁡(Xi)=Pr⁡(Ti=1∣Xi) and mt(Xi)=𝔼[Yi∣Ti=t,Xi], where Yi=Δi is AI-human rating divergence. Hats denote cross-fitted predictions. With λ⁡(e)=e⁡(1−e), λ˙​(e)=1−2​e, and τ^i=m^1​i−m^0​i, the estimating score is

Here ℙn denotes the sample average, and s indexes a sample partition. The first equation adjusts the outcome contrast using the propensity and outcome predictions. The derivative terms in the numerator and denominator account for the dependence of overlap weights on the estimated propensity (Wang et al., 2025). Outer and inner folds group all observations from a paper together. Table F2 specifies the nuisance learners, preprocessing, and weighting.

Inference and Diagnostics

For split s, J^s=ℙn​(D^i​s) and

Group ℐg contains the observations from paper g, and Φ^g​s sums their influence scores. With G papers and n observations, the paper-clustered standard error is

Ten partitions yield median estimate θ~=medians⁡θ^s and

Two-sided tests use the normal reference distribution. The corresponding 95% interval is θ~±1.96​SE~. This standard error incorporates sampling uncertainty and variation across partitions.

Diagnostic weights are wi=Ti​(1−e^i)+(1−Ti)​e^i. Effective sample size is (∑iwi)2/∑iwi2, which describes how much information remains after weighting. Diagnostics report the share of propensities outside [0.05,0.95] and weighted standardized mean differences. They also report propensity AUC and log loss, and cross-fitted outcome MAE, root mean squared error (RMSE), and R2. Median predictions across partitions summarize the fitted models. These diagnostics assess observed overlap, covariate balance, and prediction performance.

Appendix G Prompt Implementation

This appendix specifies the prompt inputs, scoring scales, review forms, and condition-specific instructions.

Inputs, Scales, and Output Structures

All ten conditions use integer scores for the supplied manuscript. Soundness, Presentation, and Contribution range from 1 to 4 (poor, fair, good, excellent). Rating ranges from 1 to 10, and Confidence ranges from 1 to 5. Rating distinguishes strong reject (1-2), reject (3-4), borderline reject (5), borderline accept (6), accept (7-8), and strong accept or award-level work (9-10). Confidence ranges from very low to very high. Soundness assesses technical rigor, Presentation clarity, and Contribution novelty and significance. Rating is a holistic judgment.

The V1-V5 form orders Summary, Strengths, Weaknesses, Soundness, Presentation, Contribution, Ethical Considerations, Rating, and Confidence. Scored headings follow the pattern “Soundness: X: ¡short label¿,” with an integer and explanation. Tone and depth templates order Summary, Soundness, Presentation, Contribution, Strengths, Weaknesses, Questions, Ethics flag and details, Rating, and Confidence. The forms share scales and differ in section order. The tone templates require every section and scored dimension, including Questions, even when the narrative is brief.

Structured Conditions: V1-V5

V1: baseline. The role is “You are an ICLR reviewer.” It requests a complete, structured review of the title, abstract, main text, and supplied appendices. It asks for specific strengths and weaknesses, ethical assessment, and evidence-linked scores. The grounding instruction is: “Do not invent data, experiments, or citations that are not present in the paper.”

V2: reviewer guidelines. V2 retains the form and adds checks on evidence, methods, clarity, reproducibility, and balanced assessment. It asks reviewers to check proofs, experiments, ablations, strong baselines, unrealistic settings, and leakage. Missing support should lower Soundness and Rating. Missing implementation details should reduce Presentation and Soundness. Its distinctive instruction includes: “Do not give high ratings based on vague impressions such as ‘interesting idea’ without solid empirical or theoretical support.”

V3: conduct and ethics. V3 adds objectivity, score discipline, explicit weaknesses, uncertainty, and ethics to the reviewer guidelines. Its conduct rule states: “Do not speculate about the authors’ identity, affiliation, seniority, or intentions.” Unresolved uncertainty is a negative factor for Soundness and Rating. Unaddressed misuse, fairness, privacy, security, or safety risks count as weaknesses. Adequate discussion and mitigation receive positive recognition.

V4: Area Chair selectivity. V4 changes the role to Area Chair and requests a rigorous, conservative review. Ratings of 6 or higher require clear novelty, sound methodology, strong evaluation, and reasonable clarity and reproducibility. Ratings of 7 or higher require comprehensive evidence, non-trivial contributions, and no major unresolved methodological or ethical concern. The tie rule is explicit: “If you are undecided between two adjacent scores (for example, 6 vs 7), choose the lower score.” It also prohibits topic-based leniency.

V5: historical reference and explicit adjustment. V5 combines conservative assessment, modified review guidelines, and a calibration procedure. The reference states: “Typical mean reviewer rating in large conferences is around 5.0–5.5.” Scores of 8 or higher are described as rare and appropriate for exceptional work. The material procedure is:

1. 1.

   “First, form an internal preliminary overall rating based purely on your qualitative assessment (this is not yet reported).”

2. 2.

   “Then, adjust this preliminary rating downward:” “In most cases, reduce it by approximately 1 point.” For noticeable weaknesses, the instruction says to “reduce it by up to 2 points.”

3. 3.

   Avoid the adjustment only when novelty, technical strength, empirical validation, clarity, and reproducibility are all strong. Use 7 or higher only across a high bar in all major dimensions, and 8 or higher only for exceptional, award-level or field-changing work.

The practical guidance places promising but imperfect papers typically in the 4-6 range and requires the reported Rating and rationale to reflect calibration. V4-V5 therefore compare instruction bundles. Their differences combine historical reference values, downward adjustment, and conservative assessment.

Tone and Depth Conditions

The five tone/depth templates use the separate form and integer scores. They prohibit systematic score shifting by style. Standard, Constructive, Critical, and Quick restrict judgment to the supplied paper and instructions. Domain Expert also permits generally accepted domain knowledge. It excludes external experimental results, unpublished work, and author-specific assumptions.

Standard requests balanced, evidence-based, decision-relevant strengths and weaknesses, concise comments, targeted Questions, and agreement between text and scores. Its constraint states: “Standard style must NOT systematically shift scores.”

Constructive requests prioritized, actionable suggestions and respectful, improvement-oriented criticism while disclosing serious problems. Its core instruction is to “Frame weaknesses as opportunities for improvement.”

Critical requests skeptical examination of assumptions, methods, missing evidence, baselines, ablations, and overclaims while acknowledging strengths. It states: “Critical style must NOT mechanically downshift scores.”

Domain Expert requests technical depth, field-aware novelty assessment, hidden assumptions and edge cases, appropriate baselines, and accessible explanations for nonspecialists. Expertise changes commentary emphasis while retaining the scoring scales.

Quick preserves all sections and scores while shortening narrative: Summary uses 2-4 sentences. Strengths and Weaknesses use 2-5 points or short sentences. Scores still require a rationale, especially at extremes. The template states: “Quick style must NOT systematically change scores.”

注意转载授权 · 阅读原文