PersonaScienceBench：用多样化模拟研究者评测科学智能体的协作能力

PersonaScienceBench

Siyang Wu¹, Weizhi Zhang², Chen Yang³, Esther Xu Fei, Jorin Overwiening³, Xu Pan³, Jing Tang⁴, Yixuan He^5,‡, Kelvin C. Mo³, Peiwen Li⁶, Defu Cao⁷, Xu Hu⁸, Haixin Wang⁹, Taoran Li¹⁰, Haoming Li⁸, Zheyuan Deng¹¹, Jingwen Zhang³, Pengsong Zhang¹², Haichao Zhang¹³, Guangze Luo³, Yi Daniel Lu³, Zihuai He^14,‡, Chenxi Qiu^3,‡, Dianzhuo Wang³, Zhejian Peng¹⁵, Shushu Liang³, Wenyi Lu¹¹, Yifan Wang¹⁶, Jintao Huang¹⁷, Yijun Wang³, Xiaomin Li^18,∗, Yuexing Hao^18,∗
1University of Chicago  ²University of Illinois Chicago  ³Harvard University  ⁴Boston University  ⁵Arizona State University  ⁶Yale University  ⁷University of Southern California  ⁸The University of Texas at Dallas  ⁹University of California, Los Angeles  ¹⁰Texas A&M University  ¹¹Brown University  ¹²Department of Mechanical and Industrial Engineering, University of Toronto  ¹³Northeastern University  ¹⁴Departments of Neurology and Medicine, Stanford University  ¹⁵University of Illinois Urbana-Champaign  ¹⁶Purdue University  ¹⁷The Ohio State University  ¹⁸MatrAIx
Bold: core team.  ^‡Faculty.  ^∗Corresponding authors

October 23, 2026

Abstract

Scientific agents collaborate with researchers whose expertise and preferences differ, but benchmarks usually hold the collaborator fixed. We introduce PersonaScienceBench: 200 tasks from five benchmarks across four scientific domains. Simulated researchers provide context and feedback according to persona profiles, while task objectives and grading remain fixed. Agents encounter researcher preferences only through conversation. Across eighteen model configurations, task performance, fact recovery, and self-repetition reveal different strengths. Detailed GPT-6 Astra analyses show distinct agent responses across researcher profiles; interaction traces illustrate targeted clarification and workflow revision. These behaviors can differ even when final scores match. PersonaScienceBench therefore evaluates how agents collaborate with diverse researchers alongside whether they complete the task.

1 Introduction

Scientific work is collaborative.

Language-model agents now perform substantial parts of the scientific workflow. They search literature, propose hypotheses, write and execute analysis code, and assemble results into reports Lu et al. (2024); Yamada et al. (2025); Gottweis et al. (2026); Mitchener et al. (2025b). The benchmarks used to measure this progress have followed the same trajectory, moving from recall and reasoning questions toward executable analysis tasks drawn from real studies Majumder et al. (2025); Mitchener et al. (2025a); Kon et al. (2026); Ye et al. (2025). These capabilities do not remove the need for human collaboration, and recent work questions their suitability for unsupervised discovery Bisht et al. (2026). Researchers often hold context that determines whether an analysis is valid: why samples were excluded, whether a covariate is confounded with batch, or which normalization a laboratory uses. Such information may be absent from the data directory. An agent’s scientific output can therefore depend on how it consults a researcher who may explain readily, answer only the literal question, or misremember a detail.

The evaluation gap.

Scientific agent benchmarks typically provide complete task specifications. Interactive benchmarks instead test escalation, clarification, and tool-agent interaction Trinh et al. (2026); Somasekharan et al. (2026); Yao et al. (2024); Chen et al. (2026); Ta et al. (2026), but usually hold the collaborator fixed. Non-collaborative simulators and persona policies vary user behavior in service domains Shim et al. (2026); Chopra et al. (2026); they do not test its consequences for scientific correctness. Varying the researcher while holding the task and agent fixed makes the collaborator’s influence observable. Figure 2 illustrates why interaction also matters: episodes contribute to final performance at different completion lengths across researcher profiles. Evaluation should therefore capture how agents clarify requirements, respond to feedback, and revise their work alongside whether they succeed.

Our approach.

We introduce PersonaScienceBench, which varies the simulated researcher on fixed scientific tasks (Figure 1). We convert 200 tasks from DiscoveryBench Majumder et al. (2025), BixBench Mitchener et al. (2025a), EXP-Bench Kon et al. (2026), SciConvBench Somasekharan et al. (2026), and ASI-Bench Zhou et al. (2026) into collaborative tasks with clarification or review while preserving source-specific deliverables and grading. The researcher receives a complete rendered MatrAIx profile Li et al. (2026): an auditable attribute–value listing, typically containing roughly 200 populated attributes from a schema of 1,290 categorical dimensions. Only the researcher simulator receives this identity block; the actor observes researcher preferences through conversation, and evaluator references remain private. In the persona-off condition, the simulator receives no persona profile; all other instructions remain unchanged. Section 4.1 specifies the single-trial protocol and uncertainty estimates for these comparisons.

What we measure.

We assess scientific outcomes and interaction behavior separately (Figure 1c). Source-specific graders measure task performance; interaction metrics capture review rounds, revisions, clarification, message length, and language adaptation. Together, they distinguish successful completion from intermediate progress, such as recovering missing information in an episode that still ends unsuccessfully. They also reveal changes in collaboration that a final score alone can miss.

Contributions.

Our contributions are threefold:

• •

  A benchmark for scientific collaboration with diverse researchers. We introduce PersonaScienceBench, adapting 200 tasks from five benchmarks across four domains into interactions involving clarification, review, and revision. Researcher personas vary while task objectives and grading criteria remain fixed. Only the simulator receives the profile; the actor encounters researcher preferences through interaction.

• •

  Evidence of persona diversity and agent adaptation. We show distinct simulator and actor responses and interaction trajectories across researcher personas. Behavioral analyses and case studies reveal differences in communication volume, targeted follow-up questions, and workflow revision.

• •

  Evaluation beyond final scores. We evaluate eighteen model configurations using scientific performance, fact recovery, and self-repetition. Detailed persona analyses show how different collaboration processes and information-gathering progress can be obscured by final scores, making these behaviors observable alongside task outcomes.

2 Related Work

Language Models with Personas. Conditioning language models on persona descriptions can produce varied responses and simulated user behavior Argyle et al. (2023); Ge et al. (2024). Recent work develops persona generators for user coverage and applies persona-based simulation to interactive-system evaluation Paglieri et al. (2026); Liu et al. (2026). Non-collaborative user simulators and Persona Policies further show that varying user behavior can change tool-agent performance in service domains Shim et al. (2026); Chopra et al. (2026). MatrAIx provides structured profiles spanning background, capability, and behavior Li et al. (2026). We use these profiles to study simulated scientific collaborators, comparing persona-conditioned interactions with a persona-off control while holding scientific tasks and interaction protocols fixed.

Interactive Agent Evaluation. Interactive benchmarks increasingly test whether agents ask for missing information and adapt to user feedback. τ-bench evaluates tool-agent-user interaction under domain rules Yao et al. (2024); HiL-Bench targets escalation and help-seeking Trinh et al. (2026); AppWorld-UL and regret-based clarification benchmarks study multi-turn interaction and underspecification Chen et al. (2026); Ta et al. (2026). These settings establish that interaction policy is itself a capability, but they generally hold the collaborator fixed or study non-scientific service domains. PersonaScienceBench instead varies the researcher while retaining a graded scientific objective, allowing collaborator-conditioned behavior and scientific outcome to be analyzed together.

Agents for Scientific Discovery. Language-model agents increasingly support hypothesis generation, experimental implementation, and data analysis, as illustrated by the AI Scientist, AI co-scientist, and Kosmos Lu et al. (2024); Yamada et al. (2025); Gottweis et al. (2026); Mitchener et al. (2025b). Benchmarks evaluate these capabilities through data-driven discovery and computational biology Majumder et al. (2025); Mitchener et al. (2025a), experimental design and execution Kon et al. (2026), and scientific implementation under varying methodological guidance Zhou et al. (2026). SciConvBench extends evaluation to multi-turn clarification of underspecified computational-science requests Somasekharan et al. (2026). These benchmarks assess scientific task performance and information recovery, but generally leave collaborator behavior fixed. PersonaScienceBench builds on their task settings by varying the simulated researcher on matched scientific tasks and measuring interaction patterns alongside source-specific outcomes. This design makes differences in the collaborative process visible alongside differences in scientific outcomes.

3 PersonaScienceBench

3.1 Persona Construction

Source personas.

We draw simulated researcher profiles from the public Persona8B coreset released by MatrAIx Li et al. (2026). Its underlying population uses a schema of 1,290 categorical dimensions spanning background, psychology, capability, interaction, and lifestyle. MatrAIx constructs records through two pathways. Synthetic records are sampled in topological order from a directed acyclic dependency graph: source-informed categorical priors and conditional adjustments preserve selected correlations, while compatibility rules exclude specified contradictions. Human-grounded records map biographies, review histories, coded surveys, and consented self-reports into the same schema; fields without source support remain unassigned. Both paths undergo source-appropriate quality checks and deduplication. The released coreset contains 599,847 human-grounded and 400,000 synthetic records. This construction provides varied, inspectable identity prompts; it does not establish that a simulated collaborator represents a particular real researcher or a population subgroup.

From records to simulated researchers.

We select four coreset profiles using a coverage-based strategy over communication, decision-making, interaction, tool-collaboration, and workflow attributes. Together with persona off, they define the five settings used throughout the evaluation. For each selected record, the pinned MatrAIx renderer emits every retained populated attribute as a category-grouped attribute–value identity block; null, placeholder, schema-default, and designated source-dump fields are omitted. We inject this complete rendered block into the researcher simulator’s system prompt, without task-specific pruning or summarization. The actor and evaluator never receive it. The persona-off condition uses the same simulator and interaction protocol with the identity block removed. Scientific task facts and grading criteria are held fixed across conditions (Section 3.2).

Table 1 summarizes the backgrounds and interaction preferences of the four profiles used in the detailed persona analyses. The same four profiles are used for every model and task. Their comparisons characterize sensitivity to these collaborator settings, rather than an average effect over a population of researchers. The case studies in Section 6 and Appendix E additionally use persona 127, an English-language coreset profile with a background in education and curriculum design (Appendix D.2). It was run with GPT-6 Astra on the same tasks, under the same protocol and against the same persona-off conversations, so its examples need no translation. It is not one of the five evaluation settings and does not enter any reported result.

3.2 From Static Benchmarks to Interactive Simulations

We adapt scientific tasks so that an actor must work with a simulated researcher, exposing decisions that final-answer evaluation alone does not capture. Conversion separates public inputs, researcher context, and private evaluation, and defines when the agents interact to share information (Algorithm 1). The aim is to make consultation and review observable while preserving a well-defined scientific task. Required checkpoints are distinguished from voluntary clarification and revision; longer conversations are not rewarded.

The benchmark contains 200 tasks, with 40 each from DiscoveryBench Majumder et al. (2025), BixBench Mitchener et al. (2025a), EXP-Bench Kon et al. (2026), SciConvBench Somasekharan et al. (2026), and ASI-Bench Zhou et al. (2026). Table 2 gives their distribution across four domains. The selection covers numerical scientific workflows and diagnostic plots; tasks involving image processing are outside the scope of this study.

0:  Source task t; persona p (or persona off)

1:  (b,k,e,π)←Convert⁡(t)

2:  Initialize actor A with public inputs b.

3:  Initialize simulator S with context k and p.

4:  h←[] {shared conversation}

5:  while protocol π permits another turn do

6:   if π assigns the next turn to A then

7:    m←A.ActAndReport⁡(h)

8:   else

9:    m←S.Respond⁡(h)

10:   end if

11:   Append m to h.

12:  end while

13:  y←SealSubmission⁡(A,h)

14:  return e.Evaluate⁡(y,h)

DiscoveryBench, BixBench, EXP-Bench, and ASI-Bench use approach and result reviews: the researcher approves a proposed approach, then requests an identifiable revision or concludes after reviewing the reported result. SciConvBench retains its native specification-clarification dialogue. Evaluator references remain private, and only the researcher receives the persona profile. Source-specific deliverables and grading are retained within declared adaptations, including EXP-Bench’s design-only track and ASI-Bench’s objective-only actor with researcher-held methodological guidance. Appendix A.1 details the shared conversion rules, information boundaries, interaction protocol, and acceptance rubric; Appendices A.2–A.6 describe the benchmark-specific implementations.

4 Experimental Setup

4.1 Models, Persona Settings, and Evaluation Protocol

We evaluate eighteen model configurations (Table 3). The Anthropic models are Claude Opus 5.5 Anthropic (2026b), Claude Sonnet 5.5 Anthropic (2026d), Claude Sonnet 5 Anthropic (2026c), Claude Haiku 5.5 Anthropic (2026a), and Claude Haiku 4.5 Anthropic (2025). From Google DeepMind, we evaluate Gemini 3.7 Flash Google DeepMind (2026) and Gemini 3 Flash Google DeepMind (2025). The OpenAI models are GPT-6 Astra, GPT-6 Sol, and GPT-6 Luna OpenAI (2026a); GPT-5.6 Terra OpenAI (2026c); GPT-5.5 OpenAI (2026b); and GPT-5 mini and GPT-5 nano OpenAI (2025). We also evaluate Grok 4.6 from xAI xAI (2026), Muse Glimmer 30B from Meta Meta (2026), GLM 5.3 Flash from Z.ai Z.ai (2026), and Qwen 3.8 27B from Alibaba’s Qwen team Qwen Team (2026). Each configuration uses the same model for the actor and researcher simulator. All model-based evaluations use GPT-6 Astra with low reasoning effort. GPT-6 Astra also uses low reasoning effort when acting as the actor and simulator. The detailed persona analyses in Sections 5.2–5.5 use the set of runs with GPT-6 Astra.

We evaluate five settings: the four profiles in Table 1 and persona off. Only the researcher simulator receives the profile; persona off omits it while retaining the same interaction protocol. Every model is evaluated on the same 200 questions, comprising 40 from each benchmark, with one trial per question and setting. Thus each model has 1,000 task–setting episodes. For GPT-6 Astra, persona off and personas 155 and 001 were each run up to three times, and Astra’s performance estimates average the available trials for each task (Appendix B). Tasks, scientific requirements, and grading criteria remain fixed across settings.

For Astra performance and self-repetition, 95% confidence intervals use 20,000 study-cluster bootstrap resamples within benchmarks Field and Welsh (2007), with shared resampling draws across settings. These intervals describe variation across sampled studies, conditional on the recorded trials; they do not estimate variability across repeated rollouts. Appendix B details aggregation and metric eligibility.

4.2 Evaluation Metrics

Scientific performance. We normalize each benchmark’s score to a 0–100 scale. Performance for setting p is Sp=|ℬ|−1​∑b∈ℬ|Qb|−1​∑q∈Qbsq​p, where sq​p is the normalized score of question q under setting p (averaged over available trials for GPT-6 Astra), Qb is the set of questions from benchmark b, and ℬ contains the five benchmarks. The design uses the same 40 questions per benchmark for every model and setting; Appendix B describes the questions available to each estimate. Mean Persona Performance is Spersona=|𝒫|−1​∑p∈𝒫Sp for 𝒫={155,001,003,046}; persona off is reported separately.

Fact recovery. For SciConvBench disambiguation cases, fact recovery is Fe=100​|He∩Re|/|He|, where He is the nonempty set of required withheld facts and Re is the set recovered during episode e.

Actor self-repetition. We measure lexical reuse across actor messages using word windows. For window w in eligible message m, the match indicator is Ie​m(w)=𝟏{maxv∈We,<m∑k=16𝟏{wk=vk}≥3}, where We,<m contains windows from earlier eligible actor messages and 𝟏 denotes the indicator function. Episode-level repetition is Repe=100Me−1​∑m=2Me|We​m|−1​∑w∈We​mIe​m​(w), where We​m contains the windows in message m and Me≥2 is the number of eligible messages. Messages must contain at least nine word tokens, matching the fixed eligibility rule used in the sensitivity grid (Appendix B).

5 Results

5.1 Model-Level Scientific Collaboration

Across the eighteen reported model configurations, the difference between mean persona performance and persona-off performance varied in both direction and magnitude (Table 3). GPT-5 mini showed the largest observed increase, from 26.37 under persona off to 31.63 across the four personas, a difference of 5.26 points. Gemini 3.7 Flash had nearly identical means under the two settings (48.21 and 48.17, respectively). Evaluating multiple researcher settings therefore reveals variation in scientific performance that a single persona-off score cannot capture.

The model configurations also showed contrasting strengths in task performance and interaction behavior. Gemini 3.7 Flash had higher mean persona performance than Gemini 3 Flash (48.17 versus 31.98), whereas Gemini 3 Flash had higher fact recovery (90.01% versus 75.24%). GPT-5 mini had the lowest observed self-repetition (10.52%), further distinguishing interaction behavior from the ordering by performance. Together, these measures characterize task success, recovery of missing information, and reuse of earlier language, providing a fuller account of how the coupled actor–simulator configurations participate in scientific collaboration. The following analyses examine how these behaviors vary across researcher profiles.

5.2 Performance across Scientific Domains

The persona profiles showed different relative strengths across scientific domains (Table 4). Although persona 003 had the highest observed mean throughout, the ordering of the other profiles changed with the domain. Persona 001 ranked second in physical sciences and engineering and in mathematics and computer science, whereas persona 046 ranked second in social sciences and humanities. Their relative performance reversed: 001 scored above 046 in mathematics and computer science (54.15 versus 50.76), but below it in social sciences and humanities (16.86 versus 18.35). Thus, an overall persona ranking can obscure domain-specific strengths.

The profiles provide useful context for these differences. Persona 001 combines business and management with curriculum design and brief, direct communication; persona 046 combines finance and economics with formal, indirect communication. Because background and communication traits vary together, these comparisons cannot identify which attributes explain a score difference. Persona 155, whose agronomy background accompanies detailed, accuracy-oriented guidance, ranked second in life sciences, although means were closely grouped in that domain. Persona 003’s engineering background and concrete, analytical style accompanied its largest difference from persona off in physical sciences and engineering (76.91 versus 66.73). Together, these patterns motivate evaluating agents across combinations of researcher profiles and scientific domains to reveal strengths that an aggregate score can conceal. Within each domain, every setting is evaluated on the same questions.

5.3 Semantic Diversity in Responses and Interaction Paths

Researcher personas were associated with distinct final responses from both the simulator and the actor (Figure 3A). The analysis compares final role messages from the single-trial Astra evaluation, matching questions across all five settings. We compared Qwen3-Embedding-0.6B embeddings Zhang et al. (2025) within domains, measured separation relative to within-persona variation across questions, and weighted domains equally. All ten persona pairs showed positive separation in each role (Holm-adjusted p<0.05, correcting across 20 question-wise label-swap tests Phipson and Smyth (2010); Holm (1979)). Actor responses were separated even though actors encountered profiles only through conversation. These exploratory comparisons include possible language effects; Appendix C gives methods and coverage.

Two manually selected examples illustrate variation across intermediate messages and interaction paths (Figure 3B). In BixBench, both simulator and actor trajectories followed distinct paths with similar overall movement and partial convergence in the UMAP McInnes et al. (2018) projection. The shared task thus elicited a recognizable progression across personas, while the messages produced along the way remained different. In SciConvBench, the simulator trajectories differed in their overall patterns, and the corresponding actor trajectories were distinct as well. These local projections illustrate message sequences; their axes are not comparable across plots, and projected distances are not effect sizes.

The selected episodes also show targeted follow-up by the actor. The BixBench example involved more interaction iterations under every persona than under persona off. In SciConvBench, personas 155, 003, and 046 involved three or four interaction iterations, compared with one iteration for persona off. During these conversations, the actor pursued specific questions: under persona 046, it successively clarified the starting structure, magnetic configuration, external pressure, and convergence criteria. This progression from a brief interaction to targeted follow-up questions makes the agent an active participant in specifying how the task should proceed. Such participation is valuable in scientific collaboration, where initial requests can leave consequential assumptions and preferences implicit. Continued dialogue gives the researcher and agent opportunities to make those choices explicit, resolve uncertainty, and incorporate feedback before committing to an approach.

5.4 Communication Patterns across Researcher Profiles

Profiles that favor detailed communication were accompanied by more extensive output from both the simulator and the actor (Figure 4A,B). Relative to persona off, personas 155 and 003 were associated with an additional 1.12 and 0.61 thousand actor tokens per episode, respectively. Simulator output also increased under both profiles; all four comparisons had Holm-adjusted p<0.05. These comparisons matched questions across settings before averaging across benchmarks. The contrast also appeared between personas: actor output under 155 and 003 exceeded that under persona 001, whose profile favors brief, direct communication. Researcher preferences were thus reflected in how much the agent produced in response, extending the behavioral variation beyond the simulator itself.

The clearest differences appeared in communication volume. Clarification recall varied by −0.28 to 4.55 percentage points across persona pairs, and tool use by −0.54 to 0.28 calls per episode (Figure 4C,D); neither metric showed a pairwise difference at Holm-adjusted p<0.05. These patterns give the benchmark contrasting communication demands: detailed, accuracy-oriented guidance under persona 155, concrete explanations under persona 003, and concise responses under persona 001. Evaluating these settings together makes responsiveness to researcher preferences an observable part of scientific collaboration.

5.5 Interaction Behavior beyond Final Scores

The actor returned to its earlier wording to different degrees depending on the researcher persona (Figure 5A). Self-repetition was 8.03% under persona 003, compared with 13.50% for persona off and 15.94% under persona 046. These differences are relevant to conversational mode collapse, where interactions become concentrated in a few recurring response patterns. CORE evaluates this phenomenon jointly through mode diversity, lexical repetition, and semantic stagnation Pandey et al. (2026). Extended closed-loop LLM studies also report semantic collapse, with meanings converging even as surface wording varies Kong et al. (2026). This motivates assessing self-repetition alongside semantic diversity (Section 5.3). The observed interaction lengths further motivate evaluating multi-turn scientific work (Figure 2). Every setting received score contributions from episodes completing after the second iteration. For persona 003, the cumulative contribution rose from 37.49 at iteration two to 48.69 at iteration six as later-completing episodes were included.

Information recovery revealed progress during multi-turn scientific work, including in episodes that ultimately failed (Figure 5B). Within each setting’s failed SciConvBench disambiguation episodes, the case-averaged rate of recovering all required facts ranged from 20.0% to 41.7% across the four personas, compared with 0% for persona off. The persona-conditioned episodes with complete recovery contained four to ten interaction iterations, during which the actor elicited every annotated missing fact before termination. This identifies a concrete intermediate accomplishment within unsuccessful tasks: recovering the annotated context needed to formulate a scientific response. Final scores alone would leave this information-gathering progress unreported.

6 Discussion

In BixBench, researcher feedback helped the actor replace an unavailable R/limma Ritchie et al. (2015) workflow with Python while preserving the scientific objective (Figure 6). The researcher, persona 127 (an additional English-language profile; Section 3.1), treated the runtime limitation as grounds for an explicit methodological change and required the replacement tests to be labelled as unmoderated rather than limma-trend. The matched persona-off conversation reached the same answer through a different analysis that met no runtime blocker, so this collaboration is visible in the interaction rather than in the final score. Appendix D.2 describes persona 127; Appendices E.1 and E.2 provide both interactions. A complementary clarification case and its persona-off comparison appear in Appendices E.3 and E.4.

注意转载授权 · 阅读原文