阿里团队发布 SciTaste-Bench：六学科评估大语言模型判断论文科研品味的基准

   Xiaogang Li Affiliation: Alibaba Group Email: xiongdao@alibaba-inc.com    Jialu Nie Affiliation: Alibaba Group    Yuting Fu Affiliation: Alibaba Group    Jingye Xu Affiliation: Alibaba Group    Xinyan Liu Affiliation: Alibaba Group    Muxi Zheng Affiliation: Tsinghua University    Xiao Zhou Affiliation: Alibaba Group    Wenbo Li Affiliation: Alibaba Group    Xuan Ren Affiliation: Alibaba Group    Hu Wei Affiliation: Alibaba Group    Bing Zhao Affiliation: Alibaba Group

October 9, 2026

Abstract

Whether large language models can assess the innovativeness of academic publications is important in literature reviews, research funding screening, and academic recommendations. We introduce the Scientific Research Taste Benchmark (SciTaste-Bench), a gold-standard benchmark covering six disciplines: physics, mathematics, chemistry, biology, medicine, and finance. SciTaste-Bench measures research taste, that is, the ability to judge the scientific value and potential of a paper, of which innovativeness is only one dimension among the seven assessed. SciTaste-Bench integrates metadata from NASA ADS, OpenAlex, and arXiv, aligns papers to field-specific classification systems, and evaluates each paper using the seven-dimensional Academic Taste Index. The weights for each dimension are derived from a bi-objective machine-learning derivation (five regressors, including random forests) and are shared across all six disciplines. This benchmark includes 300 selected papers (50 per discipline) and a backup pool of 5,700 papers (950 papers per field) covering 1996–2023. To allow external knowledge to influence model decisions, SciTaste-Bench supports four evaluation modes: pure_qa, native_search, our self-built structured tool-calling agent DimAgent-v3 (9 bibliometric tools in ADS/OpenAlex), and dimagent_v3_native. Model outputs are evaluated with two complementary schemes: one combines correlation with dimension-level accuracy, and the other is based on the WAPE relative error. In the evaluation of ten state-of-the-art large language models, the tool-based structured evaluation generally outperformed the toolless and raw web search modes, but the improvement was model-dependent. Physics is the easiest subject, while chemistry, biology, and medicine are still more challenging. By measuring how well large language models judge the value potential of academic work across fields, SciTaste-Bench supplies the value-judgment substrate that auto-research systems need to screen, rank, and prioritize candidate ideas. We release the dataset, the evaluation harness, and the scoring code to support reproducible research on scientific-innovation assessment; all data and code are publicly available on Hugging Face at https://huggingface.co/datasets/skylenage/SciTaste-benchmark.

^††footnotetext: ^†These authors contributed equally to this work. ^*Corresponding authors.

Keywords: large language models; research-paper innovativeness; scientific benchmark; scientometrics; multi-disciplinary evaluation

1 Introduction

The evaluation of the innovativeness of research papers is located at the intersection of scientometrics and natural language processing. Citation-based metrics have a long history [3, 7] , and recent studies have shown that large language models are already being utilized to produce scientific feedback [4] , generate and rank research ideas [10] , and act as reviewers or judges [12, 6] . However, most existing evaluations focus on a single discipline, often physics or NLP, and tend to use small or loosely structured samples. Furthermore, they rarely control whether the model has access to external knowledge. This leaves three questions open:

1. 1.

   Does a model’s ability to judge innovativeness vary significantly across disciplines?

2. 2.

   How does the model’s performance change when it is equipped with retrieval tools, native web search, or structured bibliometric database queries?

3. 3.

   Do models show systematic bias in data-sparse or sensitive domains such as biomedicine and the social sciences?

SciTaste-Bench is designed to address these questions. The main contributions are:

1. 1.

   A unified and reproducible construction pipeline integrates multi- source metadata, classifies papers according to discipline-specific taxonomies, and scores them using a seven-dimensional Academic Taste Index. The index weights are derived by machine learning and are uniform across all six disciplines (Sec. 3).

2. 2.

   Cross-discipline comparability under one fixed weight vector covers the fields of physics, mathematics, chemistry, biology, medicine, and finance, producing 300 selected papers and 5,700 backup papers (950 per discipline) with unified metadata (Sec. 4).

3. 3.

   DimAgent-v3, a self-built structured tool-calling agent exposes nine bibliometric tools over NASA ADS and OpenAlex, allowing the isolation and analysis of the effect of structured external knowledge (Sec. 5).

4. 4.

   A dual scoring protocol combines a correlation/dimension composite with WAPE relative-error grades, alongside an empirical study of ten LLMs across four modes (Sec. 6, 7).

5. 5.

   A foundation for auto-research. Because SciTaste-Bench provides fixed, gold-standard judgments of scientific value across six disciplines, it can serve as the value-prioritization substrate for autonomous research systems, laying groundwork for future auto-research in science.

2 Related Work

Scientometric measures of impact and disruption.

Citation dynamics has a heavy-tailed distribution, often described as a discretised lognormal or a hooked power-law distribution depending on the field and age [11]. Beyond simple citation counts, the CD index and its variants measure the extent to which a paper consolidates versus disrupts existing knowledge [3]. Large-scale analysis shows that papers and patents become less disruptive over time [7].

LLMs for scientific evaluation.

Large-scale language models can generate plausible peer review feedback [4] and novel research ideas [10], and LLM-as-a- judge paradigms are now common in open-ended evaluation [12]. A growing literature surveys LLM-based peer review and its reliability challenges [6]. This study complements this line of research by providing a structured, multi-discipline gold-standard set with fixed ground-truth scores rather than free-text review judgments.

Tool-augmented LLMs.

Tool calls targeting a large number of real-world APIs have been studied at scale [9]. DimAgent-v3 is an example of this idea applied to bibliometrics. The method does not rely on open web searches, but instead uses nine deterministic tools to retrieve raw metadata from curated academic databases. It must then reason from that evidence to produce a score.

3 Construction of the Six-Discipline Evaluation Set

3.1 Overall architecture

Figure 1 illustrates the data processing pipeline. The indexing layer collects records from three sources. The fusion layer resolves conflicts, the classifier assigns discipline and sub-discipline labels, and the index engine computes the seven-dimensional index. The final quality checker separates papers into the selected evaluation set (300 papers) and the backup evaluation set (5,700 papers).

3.2 Multi-source data collection

Three sources feed the fusion layer, each with a distinct role across disciplines. NASA ADS serves as the backbone for physics, OpenAlex is the backbone for the five non-physics disciplines (Sec. 4), and arXiv supplies preprint metadata.

NASA ADS.

ADS is the authoritative database for physics literature [1]. It contains detailed information on all related fields, such as bibcode, title, abstract, doi, bibstem, year, citation_count, authors, references, and orcid_pub.

OpenAlex.

OpenAlex supplies metadata that ADS may lack [8]. For the dimension of disruptiveness, the calculation of the CD index uses the platform’s full citing and referenced-work lists. For the dimension of conceptual innovation, its Concepts/Topics taxonomy is utilized, and for the dimension of team collaboration, authorship records containing institution and country information are used.

arXiv (preprints).

The arXiv XML API provides preprint metadata, and its subject-class field serves as the highest-priority signal for discipline classification.

Conflict resolution.

When the three sources disagree, we apply fixed precedence rules. Title, publication year, and citation count follow ADS. Abstracts follow the precedence ADS > OpenAlex > arXiv, and discipline labels follow arXiv > journal inference > keyword matching > title matching.

3.3 Discipline classification

Each academic discipline is organized according to a second-level sub- discipline taxonomy (Table 2). This taxonomy is aligned with the graduate-education discipline catalogue while being granular enough to provide each sub-discipline with a roughly balanced pool of papers. Because physics and the other five disciplines rely on different backbone databases, they use two different labeling mechanisms.

Physics: a four-stage cascade over ADS/arXiv metadata.

Eight categories are defined in physics: classical physics, condensed matter physics, quantum physics, applied physics, high energy physics, computational physics, interdisciplinary physics and astrophysics. A paper’s label is determined by a four-stage cascade (Table 1). arXiv subject classification is used where possible. If this is not available, the publication venue is matched against discipline-characteristic journals. If a decision still cannot be reached, controlled keywords are matched in the title and abstract; the final fallback is title matching. A paper receives the label from the first stage that fires. For example, quant-ph maps directly to quantum physics. At the journal stage, bibstems such as PRB or NATMAT indicate condensed matter, whereas APJ or MNRAS indicate astrophysics. At the keyword stage, two or more controlled-vocabulary hits, such as superconductivity and graphene for condensed matter, determine the label. Papers that fail all four stages remain unclassified and are routed to the backup evaluation set.

The other five disciplines: taxonomy-driven collection via OpenAlex.

The fields of mathematics, chemistry, biology, medicine, and finance do not have a system equivalent to arXiv’s subject classes. Therefore, the labels for these fields are assigned by construction rather than post hoc. Each sub-discipline in Table 2 is anchored to an OpenAlex subfield identifier and a set of controlled keywords that describe the area. Candidate papers from OpenAlex are collected by selecting article-type records whose primary topic belongs to the relevant subfield and whose text matches the keywords, sorted by citation count in descending order. If the keyword search returns too few records, the subfield filter alone is used to back-fill the candidate pool. After cross-sub-discipline deduplication, where each paper belongs to only one sub-discipline, roughly 125 papers per sub-discipline are retained through stratified sampling across composite-score quartiles. Each paper is labeled by the collection cell from which it was drawn. This design avoids between-sub-discipline misclassification, while the subfield filter and keyword query preserve topical purity within each cell. The result is eight sub-disciplines for mathematics, chemistry, biology, and finance, and seven for medicine (Table 2).

3.4 Seven-dimensional Academic Taste Index

The core scoring engine assigns each paper a 0–100 innovation score as a weighted sum over seven interpretable dimensions.

where each DimensionScorei∈[0,100]. The same seven dimensions and the same weight vector are applied identically to every discipline, and this is what makes cross-discipline comparison possible. Table 3 lists the dimensions, what they measure, and their final weights.

The weight vector is itself data-driven. The training corpus is the 998-paper quantum-physics V5 evaluation dataset from an earlier single-discipline study. Two regression analyses were run: one predicting citation impact from the other six dimensions, and one predicting journal impact from the other six. Each analysis trained five regressors—linear, ridge (α=1.0), lasso (α=0.1), random forest (100 trees), and gradient boosting (100 trees)—on standardized dimension scores with an 80/20 train–test split (seed 42) and five-fold cross-validation. Per-model feature contributions (absolute standardized coefficients for linear models, feature importances for tree ensembles) were averaged across the five models; dimensions shared by the two analyses received the mean of their two contributions, and the raw weights were normalized to sum to one, yielding the final vector in Table 3. The random forest attained a test R2 of 0.736 for citation impact but only 0.170 for journal impact, indicating that citation behavior is largely predictable from the remaining dimensions whereas journal placement is comparatively independent.

Weight derivation (dual-target Random Forest).

Which dimensions deserve the largest weights? In everyday scholarly practice, two community-sanctioned outcomes often shape perceptions of a paper’s value: how often it is cited and where it is published. Therefore, we used citation count and journal impact as two independent supervision targets, allowing the data to assess how strongly each dimension influences these outcomes. Specifically, we ran two independent Random-Forest analyses [2] and interpreted feature importances as weights. Analysis 1 predicts the citation impact (A) from the six other dimensions, and Analysis 2 predicts the journal impact (G) from the six other dimensions. Because A and G each serve as a target in one analysis, their own weights are taken from the other analysis: A’s weight comes from its contribution to predicting Journal Impact (37.99%), and G’s from predicting Citation Impact (19.73%). Dimensions B–F use the arithmetic mean of their importances across the two analyses. The raw weights sum to 128.86% and are renormalized to 100% (Table 4). Both forests use 100 trees (seed 42) with an 80/20 train–test split. The fit of the citation model is good (test R2=0.74), indicating that the citation impact is largely predictable from the other six dimensions. On the other hand, the journal model is much weaker (R2=0.17), suggesting that the journal impact has relatively independent information. The two targets therefore provide complementary evidence for the final weight vector.

3.5 Dimension-to-score mapping

Each dimension’s raw sub-parameters are mapped onto a 0–100 score.

A. Citation Dynamics (29.5%).

From total_citations we blend a log-normalized score with an empirical percentile:

The logarithmic term handles the heavy-tailed citation distribution [11]. The percentile term counts the fraction of papers that are strictly below the target, anchoring the score in the empirical distribution.

B–F. PCA + percentile rank.

The set of secondary parameters is as follows. The breadth of knowledge uses the number of references, the diversity of reference fields, the diversity of reference journals, and the age range of references as indicators. Semantic novelty uses nearest neighbor and mean neighborhood semantic distances, number of preferred fields, field Shannon entropy, SPECTER distance to the reference list, SPECTER bridge score, and reference semantic dispersion. Disruptiveness uses the CD index and disruptive citation count. Conceptual Innovation uses core-concept score, concept count, core-topic score, and topic span. Team Collaboration is measured by the country count and the author count. Missing entries are imputed first by the column mean and then by zero. The resulting matrix is standardized and projected onto the first principal component. If PC1 is negatively correlated with the standardized variables, the sign is reversed. The projection is then mapped to [0,100] by mean-rank normalization:

G. Journal Impact (15.3%).

This score is a fixed-weight blend of the impact factor (70%) and two partition schemes (15% each):

The coefficients 0.70/0.15/0.15 were set from expert experience and validated through expert inspection of the final paper scores.

The impact factor score is the percentile rank of the journal’s impact factor among observed values; missing entries receive 0 points. Both partition schemes enter the score: JCR (Journal Citation Reports) quartiles and CAS (Chinese Academy of Sciences zones) map Q1/zone-1→100, Q2/zone-2→75, Q3/zone-3→50, and Q4/zone-4→25, with missing partitions again scoring 0.

3.6 Dataset record format

Each record contains identifiers, text, year, and the ground-truth total plus seven per-dimension scores ( 2).

{
  "bibcode": "1998PhRvA..58.3983S",
  "title": "Nonadiabatic dynamics in evaporative cooling ...",
  "abstract": "Magnetically trapped neutral atoms can be cooled ...",
  "year": 1998,
  "ground_truth_score_100": 46.36,
  "ground_truth_dimensions": {
    "citation_impact": 51.55,
    "knowledge_breadth": 35.33,
    "semantic_novelty": 34.39,
    "disruptiveness": 79.22,
    "conceptual_innovation": 44.51,
    "team_collaboration": 13.77,
    "journal_impact": 57.58
  }
}

4 Six-Discipline Coverage and Unification

4.1 Coverage and collection

SciTaste-Bench covers six academic fields: physics, mathematics, chemistry, biology, medicine, and finance. Physics is collected through NASA ADS and organized into eight sub-disciplines (Sec. 3). The five non-physics disciplines are collected and aligned to the same framework in five stages: (1) define a second-level taxonomy aligned with the graduate-education discipline catalogue; (2) collect candidate papers through SCI replacement and OpenAlex retrieval, align them by title/DOI/bibcode, and assign an internal work identifier linked to the corresponding OpenAlex record when no ADS bibcode exists; (3) recompute the seven-dimensional scores with the unified scoring pipeline; (4) apply stratified sampling and manual verification; and (5) produce unified output for cross-discipline merging.

4.2 Unification principles

This unified set follows two principles. Weight consistency: all six disciplines share the same weight vector (Table 3), making their scores comparable. Field mapping: Chinese dimension names are mapped to stable English keys (citation_impact, journal_impact, etc.), raw dimension scores are scaled to 0–100, and the composite score is recomputed.

4.3 Sampling criteria

For each discipline, the 50 selected papers cover the major second-level sub- disciplines, balance journal tiers Q1–Q4 under a combined CAS-2025 and JCR-2025 tier map, emphasize 2000–2020 papers with relatively stable citation data, and span the score range so that the set retains discriminative power. Each sample is then manually checked for title, abstract, discipline, and journal consistency.

4.4 Dataset statistics

Table 5 summarizes SciTaste-Bench. The released data set consists of a selected split (300 papers, 50 per discipline) and a backup split (5,700 papers, 950 per discipline) that supports refresh and re-sampling.

Each selected record is additionally assigned discipline, subcategory_id/name, journal_tier, journal_name, and openalex_id to ensure reproducibility.

5 Tool-Calling Agent DimAgent-v3

5.1 Design

In agent mode, the model does not evaluate papers based solely on parametric memory. Instead, it issues tool calls to nine special tools. DimAgent-v3 is not an off-the-shelf component: the agent framework was designed and implemented in-house by the authors—the nine-tool interface, the two-round call-then-score loop, and the reviewer system prompt are all our own—rather than adapted from an existing agent codebase. Each tool queries NASA ADS or OpenAlex for non-physics domains in real-time and returns raw evidence rather than precomputed results. The model must integrate this evidence and decide the score itself based on it. Since the dataset provides bibcode, a separate search step is not required.

5.2 The nine tools

Table 6 lists each tool, the raw data it returns, and the corresponding dimensions. Each tool receives one bibcode argument.

5.3 Agent loop

This loop is deliberately designed to be compact (see Figure 3). The system prompt casts the model as a senior academic reviewer and prescribes three steps: run all nine tools in parallel, analyze the returned raw data, then score each dimension with 0–100 and calculate a weighted composite score. Given a bibcode, nine simultaneous calls are sent in Round 1. After receiving the raw data payload, Round 2 outputs an exact JSON object containing the composite score, seven dimension scores, a brief reasoning trace, and self-reported confidence. This design isolates structured bibliometric data as the only external input.

5.4 Backend routing

Physics papers are requested directly from NASA ADS. Non-physics papers are routed through OpenAlex: openalex_bibcode_map converts the internal identifier to an OpenAlex work ID before calling the OpenAlex API. To comply with OpenAlex’s rate limits, we enforce a minimum request interval and include a mailto field in the User-Agent to join the polite pool.

6 Evaluation Protocol and Scoring

6.1 Four evaluation modes

SciTaste-Bench separates the data set from the evaluation harness and supports four complementary modes (Table 7). The two dimagent modes are driven by DimAgent-v3, the agent we designed and built for this benchmark (Sec. 5), so the tool interface and the scoring prompt are held fixed across models. The verbatim prompt templates and a fully worked example are given in Appendix C.

6.2 Scoring scheme 1: correlation + dimension accuracy

This comprehensive assessment covers both an overall assessment and a detailed assessment. In part A (50%), the Spearman coefficient ρ (15%), Pearson coefficient r (10%), mean absolute error and mean square error (15%), and |Bias| (10%) are measured. In Part B (50%), the seven dimensions are estimated using the weighted mean Spearman coefficient ρ (30%) and the weighted mean absolute error (20%), based on the weighting in Table 3. The final score is Part A×50%+Part B×50%, and the grade classification is as follows: ≥80 is excellent, ≥60 is moderate, ≥40 is poor, and <40 is unusable.

6.3 Scoring scheme 2: relative error (WAPE)

The simple mean absolute percentage error 1n​∑i|pi−gi|/gi is dominated by papers with low true values. We therefore adopt the weighted absolute percentage error, following forecasting-evaluation practice [5]:

WAPE aggregates the total error using the true values as weights, preventing poorly scored papers from dominating the overall results. Based on the grade interval, this maps to a 0–100 score, and the grades are A (<5%), B (5–10%), C (10–20%), D (20–30%), E (30–50%), and F (≥50%) (e.g., <5%→90–100).

7 Experiments and Results

7.1 Setup

We evaluated ten frontier large-scale language models (qwen-3.8-max, qwen3.7-plus, qwen3.7-max, claude-fable-5, claude-opus-4-8, gpt-5.6-sol, gemini-3.5-flash, kimi-k3, Doubao-Seed-2.1-Pro, and DeepSeek-V4-Pro) through a unified API gateway. Where reasoning or thinking was supported, we enabled the setting: high for the Qwen and GPT models, adaptive for the Claude models, and standard for Gemini, Kimi, Doubao, and DeepSeek. The main experiments were conducted under all four conditions using a dataset of 300 articles from six academic disciplines. For each model and mode combination, the percentage of successfully scored papers was recorded.

7.2 Main results

Table 8 reports the Scheme-1 and Scheme-2 scores for all ten models. gpt-5.6-sol was fully re-run after a parameter-conflict fix (Sec. 7.5), and gemini-3.5-flash is newly added in this round. Structured tool grounding (dimagent_v3) remains the strongest single intervention. For the three Qwen models and claude-fable-5, Scheme-2 scores rise from roughly 20–33 in the tool-free modes to 47–52 (grade E → D). claude-opus-4-8 improves only to 34.5 in the pure tool mode, still grade E, with the combined mode becoming its best configuration (40.7). There are two exceptions. In the dimagent_v3 run, gpt-5.6-sol’s Scheme-2 score collapses to 10.6 (grade F) but recovers to 28.7 (E) when native search is added. gemini-3.5-flash is the weakest overall, scoring grade F in all four modes, and structured tools yield only a marginal gain. The three newly added models refine this picture. kimi-k3 stays at grade E in all four modes, with structured tools lifting its Scheme-2 score from 25.0 to 33.4. Doubao-Seed-2.1-Pro is the strongest of the newcomers, reaching grade D (44.0) in the combined mode, which is also its best configuration. DeepSeek-V4-Pro shows the weakest sensitivity to external grounding: its tool-free pure_qa score (30.2) is already its best, and structured tools add little (34.5), indicating that its parametric prior is comparatively well matched to the task while bibliometric grounding fails to compound.

7.3 Cross-discipline analysis

Table 9 breaks down the dimagent_v3 mode by discipline for qwen-3.8-max. Physics is by far the easiest (ρ=0.78, WAPE 13.1%, Scheme-2 70.4, grade C), reflecting richer parametric coverage and cleaner ADS grounding. Finance is the strongest non-physics discipline. Medicine has the weakest correlation (ρ=0.31), while chemistry and biology have the largest relative error (WAPE ≈29%). The cross-model breakdown (Table 10) shows that the physics advantage is shared by most models: the three Qwen models, claude-fable-5, and kimi-k3 all reach Scheme-2 between 46.9 and 70.7 on physics. Doubao-Seed-2.1-Pro inverts this pattern, scoring its best in finance (58.6) and medicine (49.8) and its worst in physics (15.8), a reminder that the discipline gap is partly a property of each model’s training mix rather than of the benchmark. The claude-opus-4-8 physics cell (6.3) is an instability of that model’s tool run (ρ=−0.16, WAPE 74.4%), not a reflection of discipline difficulty.

7.4 Findings

Structured tools beat raw web search.

Across models, dimagent_v3 is usually the strongest or near-strongest mode. Native search alone barely improves pure_qa and is sometimes worse: unstructured web results mix signal with noise, which often causes the model to alternate between the paper’s own abstract and distracting external hits. Structured tools return more precise bibliometric evidence and are better matched to this task. However, this advantage is not universal. gpt-5.6-sol collapses in the pure tool setting, and gemini-3.5-flash improves only marginally (Sec. 7.5).

Adding native search to tools rarely helps.

When analyzing the three Qwen models, dimagent_v3_native tracks dimagent_v3 closely, with performance differences staying within about one point on both schemes. The pattern diverges for the Claude models: claude-fable-5 drops from 47.0 to 39.3 on Scheme 2, while claude-opus-4-8 rises from 34.5 to 40.7, making the combined mode its best configuration. The most extreme case is gpt-5.6-sol: native search pulls it back from a collapsed pure- tool run (Scheme-2 10.6) to 28.7, its only grade-E configuration. For gemini-3.5-flash, the two tool modes are essentially tied. Among the newly added models, Doubao-Seed-2.1-Pro is the clearest beneficiary of stacking search on tools (39.2→44.0, E → D), whereas kimi-k3 (33.4→30.1) and DeepSeek-V4-Pro (34.5→33.7) gain nothing from the combination.

Discipline gap persists even with tools.

Physics holds a large advantage in most models and modes, while biology and medicine remain difficult. This points to uneven training coverage and to the limits of ADS-style grounding outside physics. The collapsed physics sub-run of gpt-5.6-sol’s combined mode is a notable exception (Sec. 7.5).

Systematic over- and under-estimation.

When using the tool-free mode, a large positive bias was observed, indicating that models tend to over-rate papers. The structured tool reduces this bias and, in physics, can even make it slightly negative. The direction of the bias can still reverse: in the pure tool run of gpt-5.6-sol, a strong negative bias occurred (−20.2). The median per-paper relative error reached 100%. In other words, it was confirmed that more than half of the papers are severely under-rated in this mode. The mechanism behind this reversal is examined in Section 7.5. Overall, Scheme-1 scores remain at a low level because the weight-averaged dimensional correlation (see Part B) is heavily dependent on three challenging dimensions: conceptual_innovation, semantic_novelty, and disruptiveness. The per-dimension correlation in each of these dimensions is near zero in all evaluation modes.

7.5 Engineering notes and the gpt-5.6-sol case

Several practical issues arose. Gateway issues: the gateway sometimes returned empty or non-JSON responses, or rate-limited requests until the retry budget was exhausted (HTTP 429), as for claude-fable-5 on a subset of biology, medicine, and chemistry titles and for kimi-k3 under dimagent_v3. Such records were dropped from the metrics rather than scored as zero, so the affected success counts remain under 300; they are reported separately rather than treated as a model weakness. Mass failures: gemini-3.5-flash completed only 113/300 items under native_search, with failures spread across all six disciplines; in those runs the model never emitted a parseable scoring JSON. Its other three modes completed 300/300. Parameter compatibility: some models reject certain temperature, reasoning, and tool combinations, which the client handles per- model. Claude thinking also required a larger max_tokens value to avoid truncation. Resumption: the harness supports --resume keyed on bibcode, plus a retry script for transient failures.

gpt-5.6-sol requires a caveat; this is the same case as the strong negative bias reported in Section 7.4. The early tool-calling runs for this model were affected by a parameter-conflict bug. Specifically, combining temperature and reasoning_effort with tools triggered API errors. To resolve this, these two parameters were disabled for this model’s tool calls on the client side, and a full re-run was performed. As a result, all four modes achieved 300/300. Five transient pure_qa failures during testing were back-filled by retry. The results of this re-run are included in Table 8. The pure_qa and native_search results for this model are broadly on par with other models, both receiving a grade of F, but dimagent_v3 shows systematic severe underestimation. Scheme-2 scores were only 10.6, with ρ≈0.17, bias value −20.2, and all disciplines received a grade of F. Enabling native search recovered performance to 28.7 (grade E), but the physics sub-run destabilized (ρ is undefined, Scheme-2 scores are 0.0). Therefore, this is not a technical issue but indicates a genuine model-level difference. The lesson is that tool-calling configurations must be validated per model before large runs, and a technically valid run does not guarantee behaviorally effective tool use.

8 Limitations and Future Work

Since each discipline offers only 50 selected papers, this benchmark cannot cover every sub-discipline in depth. Some runs on biomedical and chemistry titles failed at the gateway—empty or non-JSON responses and HTTP 429 rate limiting—leaving gaps that cannot always be back-filled. The dataset is English-dominant and may under-represent non-English scholarship. Since citation counts and journal indicators drift over time, the backup pool needs to be refreshed and re-sampled periodically. The next natural steps would be to expand the benchmark to computer science, materials science, earth science, and the social sciences, and to add human-expert annotations to complement the bibliometric ground truth.

A further limitation concerns the weight vector itself. The seven dimension weights are obtained from a single dual-target derivation trained on 998 quantum-physics papers with five regressors (including a random forest) and are then shared across all six disciplines. We have not re-run the derivation separately within each discipline to obtain field-specific weight vectors, so the transferability of one common weight vector across fields remains an assumption of the current benchmark. We regard this as one of the principal limitations of this work; per-discipline re-derivation and comparison are left to future work.

9 Conclusion

We present SciTaste-Bench, a six-discipline gold-standard benchmark for judging research innovativeness. The benchmark is constructed by (i) integrating metadata from ADS/OpenAlex/arXiv and scoring each paper with a seven- dimensional, machine-learning-weighted Academic Taste Index under a single weight vector shared across physics, mathematics, chemistry, biology, medicine, and finance; (ii) designing and implementing in-house a structured tool-calling agent, DimAgent-v3, together with its nine bibliometric tools, two-round loop, and reviewer prompt; and (iii) evaluating models with correlation- and relative-error-based methods. Across ten large language models, structured tool-based evaluation generally outperforms tool-free and raw web-search modes, although this advantage is model-dependent, with gpt-5.6-sol’s collapsed pure-tool run as the main counterexample. Physics is the easiest field, while chemistry, biology, and medicine are the most difficult, and dimension-level evaluation remains the key bottleneck. The dataset, the harness, and the agent—all designed and built by the authors—are released to support reproducible research on scientific-innovation assessment, and are available on Hugging Face at https://huggingface.co/datasets/skylenage/SciTaste-benchmark. Beyond benchmarking, we view SciTaste-Bench as infrastructure for future auto-research: autonomous systems that propose, evaluate, and prioritize scientific ideas require exactly this ability to judge the value potential of research papers.

Statement on AI Assistance

Large language models were used in preparing this manuscript solely to aid and polish the writing. A multi-hop LLM paraphrasing pipeline was applied segment-wise to the prose for language-level editing, style polishing, and readability, and an LLM assistant helped rephrase and reorganize author-written sentences and paragraphs—including the exposition of the results analysis—so that findings already established by the authors were expressed more clearly. In every case the underlying content, analytical points, tables, and figures were supplied by the authors. Large language models were not used to generate or label data, design experiments, run the evaluations, derive results, search the literature, or prove mathematical claims, and no model-generated text was accepted without human verification. All text was reviewed, corrected, and approved by the human authors, who take full responsibility for the content of this paper.

Appendix A Citation-count distributions

The distribution of citation counts is heavy-tailed. Two prevailing models are the discretised lognormal distribution and the hooked or shifted power-law distribution. The former is often favoured for the arts, humanities, social sciences, engineering fields, single disciplines, and papers published ≥3 years ago. The latter is more commonly favoured for the natural sciences, life sciences, recent papers, and datasets with many zero-cited items; this model naturally supports zero citations [11]. There is no universal distribution. For cross-field comparison, percentile normalization (e.g., the FWCI indicator) combined with non-parametric tests is preferable. For modeling within a specific field, AIC/BIC criteria can be used to choose between candidate distributions. For regression analysis, log⁡(c+1) combined with ordinary least squares is usually the most stable choice. All zero-cited papers should be included, heterogeneity across fields should be controlled by stratification, and goodness-of-fit should not be interpreted as evidence for a generative mechanism.

Appendix B Artifact inventory

The main outcomes published in this study include the merged dataset with its selected and backup splits, the identifier-to-OpenAlex alignment map, the report generator, the agent task configuration defining the nine tools and four evaluation modes, the scoring-metrics module implementing both scoring schemes, and the per-model run scripts, and the weight-derivation package containing the trained regression models (including the random forest) together with their hyperparameters and feature-importance outputs. All components are published together with the dataset, ensuring full replication. The detailed dataset and the full code are open-sourced on Hugging Face at https://huggingface.co/datasets/skylenage/SciTaste-benchmark.

Appendix C Evaluation prompts and a worked example

The original prompts were written in Chinese; the blocks below are condensed translations. All four modes share the same user template.

Please assess the innovativeness of the
following {discipline} paper.
Discipline: {discipline}
Title: {title}
Abstract: {abstract}
Identifier: {bibcode}

is a mode-specific termination instruction. The system prompt of pure_qa casts the model as a senior academic reviewer, requiring it to score the seven dimensions from its own scholarly knowledge, list the seven- dimension rubric using the shared weight vector, and conclude with “output the JSON directly, no explanation”. Its user template closes with “score the seven dimensions from your scholarly knowledge and output JSON.”. The system prompt of native_search retains the same skeleton, but requires the model to first retrieve the paper’s citation count, venue reputation, authorship, and field information through native web search, and to ground every score in the retrieved data. The DimAgent-v3 mode uses the system prompt summarized in Sec. 5: it consists of the nine ADS tools, the three-step parallel-call workflow, the rule that all nine tools must be called and that scores must be grounded in the returned payloads, and the same strict-JSON contract (composite score, seven dimension scores, reasoning trace, and self- reported confidence). Verbatim prompts ship with the released harness.

Worked example.

Instantiating the template on a real item gives:

Discipline: physics
Title: Nonadiabatic dynamics in evaporative
  cooling of trapped atoms by a
  radio-frequency field
Abstract: Magnetically trapped neutral atoms can
  be cooled with the evaporation technique. ...
  [truncated]
Identifier: 1998PhRvA..58.3983S

The model must reply with the strict JSON; the ground truth of this item is

{"innovation_score": 46.36,
 "dimension_scores": {
  "citation_impact": 51.55,
  "journal_impact": 57.58,
  "conceptual_innovation": 44.51,
  "knowledge_breadth": 35.33,
  "semantic_novelty": 34.39,
  "team_collaboration": 13.77,
  "disruptiveness": 79.22}}

References

• [1] A. Accomazzi (2024) Decades of transformation: evolution of the NASA astrophysics data system’s infrastructure. arXiv preprint arXiv:2401.09685. Cited by: §3.2.
• [2] L. Breiman (2001) Random forests. Machine Learning 45 (1), pp. 5–32. Cited by: §3.4.
• [3] K. Funk and J. Owen-Smith (2017) A dynamic network measure of technological change. Management Science 63 (3), pp. 791–817. Cited by: §1, §2.
• [4] W. Liang Y. Zhang et al. (2024) Can large language models provide useful feedback on research papers? a large-scale empirical analysis. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2310.01783 Cited by: §1, §2.
• [5] S. Makridakis, E. Spiliotis, and V. Assimakopoulos (2022) The M5 accuracy competition: results, findings and conclusions. International Journal of Forecasting 38 (4), pp. 1346–1364. Cited by: §6.3.
• [6] T. H. Nguyen and Z. Ahmadi (2026) LLM-based scientific peer review: methods, benchmarks, and reliability challenges. arXiv preprint arXiv:2606.25057. Cited by: §1, §2.
• [7] M. Park, E. Leahey, and R. J. Funk (2023) Papers and patents are becoming less disruptive over time. Nature 613, pp. 138–144. Cited by: §1, §2.
• [8] J. Priem, H. Piwowar, and R. Orr (2022) OpenAlex: a fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv preprint arXiv:2205.01833. Cited by: §3.2.
• [9] Y. Qin et al. (2024) ToolLLM: facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations (ICLR), Note: arXiv:2307.16789 Cited by: §2.
• [10] C. Si, D. Yang, and T. Hashimoto (2025) Can LLMs generate novel research ideas? a large-scale human study with 100 NLP researchers. In International Conference on Learning Representations (ICLR), Note: arXiv:2409.04109 Cited by: §1, §2.
• [11] M. Thelwall (2016) Are the discretised lognormal and hooked power law distributions plausible for citation data?. Journal of Informetrics. Note: arXiv:1603.05078 Cited by: Appendix A, §2, §3.5.
• [12] L. Zheng W. Chiang et al. (2023) Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Note: arXiv:2306.05685 Cited by: §1, §2.

注意转载授权 · 阅读原文