面向长周期科研智能体的动作构建：阅读集框架评估实验设计的信息价值

Alkut Arxidin, Shaoning Zeng^*, Jianhang Zhou, Jianping Gou, Xinzhong Zhu

Abstract

Large language model agents support automated scientific research by connecting literature analysis and planning with experimentation. Evaluating which action an agent selects does not establish which experiment its instruction specifies, because omitted interventions or replication counts leave consequential choices to the executor. We introduce a reading-set framework for evaluating action construction under a common cost rule. Across 30 synthetic states, complete-design requests increase affordable efficiency by 465.98% and 327.91% relative to the action-name baseline in the two model families, respectively. An author-constructed research-action evaluation dataset provides 60 scenarios for complementary judgment evaluation. On this dataset, the scenario-level information-per-cost measure has a mean within-scenario correlation of 0.733 with judgments from 18 human participants. The framework makes design specification and execution ambiguity measurable, providing a reproducible basis for assessing how research agents translate scientific intent into informative, budget-feasible experiments.

Index Terms: 

Research agents, action construction, experimental design, information gain, budget-aware evaluation.

I Introduction

Large language model agents are expanding the scope of automated scientific research. Surveys chart their growing role in literature analysis, hypothesis generation, and experimentation [1, 2]. Systems including the AI Scientist, AI co-scientist, Agent Laboratory, and Kosmos connect these capabilities into integrated research workflows [3, 4, 5, 6].

Research-agent benchmarks assess experimental performance, scientific rediscovery, and innovation, while workflow and process evaluations examine intermediate decisions and verification targets [7, 8, 9, 10, 11, 12]. Bayesian experimental design values specified experiments by expected information gain, and cost-aware learning incorporates acquisition expense into resource allocation [13, 14, 15, 16]. Tool-using scientific agents connect language-model plans to executable operations, while studies of prompt sensitivity and structured outputs examine the interfaces through which those plans are expressed [17, 18, 19, 20, 21, 22, 23]. Yet selecting an appropriate research action does not establish which experiment its instruction specifies. Language-grounding and clarification studies show why execution can require choices beyond the instruction itself [24, 25]. Experimental utility depends on the specified design and its outcome model [13, 14, 15], so different executable interpretations of the same instruction can have different information values. The unresolved question is how to evaluate the information value and affordability of an instruction when it leaves experimental choices to the executor.

The central idea is therefore to evaluate the designs admitted by the instruction, making this dependence visible. We propose a reading-set framework that scores those designs under a common cost rule with per-design budget gating. Across 30 synthetic states, complete-design requests increase affordable efficiency by 465.98% and 327.91% relative to action-name requests in the two model families. Complementary evidence comes from an author-constructed dataset of 60 scenarios and 240 candidate actions with traceable source records. On this dataset, the scenario-level information-per-cost measure shows positive agreement with judgments from a six-model panel and 18 human participants.

II Related Work

II-A Research Agents and Scientific Evaluation

Research-agent systems span hypothesis generation, literature analysis, implementation, and experimentation [3, 4, 5, 6]. General agent benchmarks and evaluation surveys organise assessment around interactive task performance and evaluation objectives [26, 27]. Benchmarks make these capabilities measurable through tasks involving experimental performance, information organisation, rediscovery, and innovation [7, 28, 8, 9]. At the workflow level, systematizations map research workflows to evaluation and verification targets [10]. At a finer scale, trace datasets and process reward models examine intermediate behaviour and reasoning [11, 12]. In controlled agent environments, exploration and exploitation errors can also be measured separately from observed actions [29]. These approaches establish complementary views of research performance, but task success and step-level correctness do not by themselves resolve the experimental choices left unspecified in an action. A scientifically relevant instruction can still admit several executable designs with different information values and costs.

II-B Information- and Cost-Aware Experimental Design

Bayesian experimental design uses expected information gain (IG) to identify experiments that reduce uncertainty about competing hypotheses [13, 14, 15]. Bayesian optimisation supports efficient search over expensive evaluations [30, 31, 32], while cost-aware active learning incorporates the expense of acquiring additional evidence [16]. Although these objectives provide a principled basis for allocating experimental resources, applying them to a natural-language action still requires identifying the executable design or designs it denotes. When interventions or replication counts are omitted, a single utility value depends on how those omissions are completed. The reading-set framework addresses this dependence by making the admitted designs explicit before aggregating their budget-gated information-per-cost contributions.

II-C From Natural-Language Instructions to Executable Actions

Autonomous chemistry systems connect language-model planning with scientific tools and laboratory operations, demonstrating how tool interfaces can carry research intentions into executable workflows [17, 18]. In robotic planning, SayCan grounds action selection in skill feasibility [24], while KnowNo uses calibrated candidate sets to determine when an ambiguous instruction requires clarification [25]. Procedural skill libraries encode reusable research workflows as instructions and supporting resources [33]. Research on prompting shows that ordering, formatting, and contextual features can influence model responses [19, 20, 21, 22], and work on schema evolution examines the reliability of structured outputs [23]. However, the reliability of an action interface does not by itself establish whether the proposed experiment is fully specified, informative, and affordable. A parsable instruction can still omit interventions or replication counts, leaving the executor to determine the design that is ultimately evaluated.

III Method

The framework evaluates the experimental choices communicated through an action interface. Given a research state and a model response, the parser extracts the design fields and the resolver maps them to a set of executable designs. We then apply a budget gate to each design, average its information-per-cost contribution, and normalise the result by the best affordable information efficiency. This flow connects action specification, executable interpretation, and scoring, as shown in Fig. 1. The common cost rule and aggregation are defined in Eqs. (3)–(5).

III-A States, Actions, and Executable Designs

A research state s contains a scientific question, a method description, partial results, candidate mechanisms, a belief distribution p⁡(h∣s) over mechanisms, and a budget B. An executable design is d=(A,n), where A is a nonempty set of ablation interventions and n is the number of seeds per condition. The unablated baseline is always run. The synthetic benchmark uses six interventions and n∈{2,…,12}, giving 63×11=693 candidate designs per state.

We represent an action as a=(τ,t,A,n,c,π), comprising action type τ, target t, intervention set A, replication count n, success criterion c, and optional prediction π. The four interfaces specify different design information. The action-name interface (L1) requests an action name and a reason, but its parser extracts only the action name. The target interface (L2) requests a target and its justification, leaving interventions and replication to the executor. By contrast, the complete-design interface (L3) requests the target, intervention set, replication count, and success criterion. The design-with-prediction interface (L4) adds a prediction to these complete-design fields. Target, arms, and seed count determine the reading set, whereas success criteria and predictions remain recorded text. Thus, two outputs with the same scored design parameters receive the same design score even if their explanations differ. Appendix A specifies the exact fields and parser interpretations.

III-B Reading Sets and Executor Sensitivity

Let Rs​(a) denote the designs produced by the fixed executor for action a in state s. The default evaluator weights its members uniformly. For an action without a stated target, the component-scan reading set contains four designs. These use one, two, or four interventions at two seeds, or one intervention at four seeds. With a target but no design, a fixed target-focus rule selects interventions and considers two replication levels. Complete arms and seed fields identify one design. Appendix A records the resolver details. Appendix C illustrates how individual responses are resolved and scored.

Two additional executor modes examine completion sensitivity. The conservative mode keeps the cheaper half of the candidate set, whereas the permissive mode keeps the more expensive half. Because neither mode changes a singleton set, a fully specified design remains the same in all modes. We also record the ambiguity index g⁡(a)=1−1/|Rs​(a)|. Its value is zero for a singleton and increases with the number of admitted readings. This index measures set cardinality, rather than the fraction of experimental parameters invented by an executor.

III-C Information, Cost, and the Primary Score

We use the information-theoretic utility of Bayesian experimental design [13, 14, 15]. A design is informative when its possible outcomes reduce uncertainty about competing mechanisms. Accordingly, its information value is the expected reduction in mechanism entropy, defined as follows.

Entropy uses natural logarithms, giving information in nats. The observation model is a normalised Gaussian kernel on the discrete grid 𝒴={0,0.25,0.5,0.75,1}. For intervention j, the probability of an observation is given by the following normalised kernel.

Here σn=0.55​5/n and μh​j∈{0,1} encodes whether the intervention removes the hypothesised source of the improvement. Outcomes are conditionally independent given h. The hypothesis-independent baseline factor cancels from the posterior. The evaluator sums over the joint outcome grid in log space, omitting outcome probabilities at or below 10−10. This is a finite observation model with numerical expected information gain.

The cost model accounts for experiment setup, interventions, and replication. The prompt exposes the same rule used by the evaluator, so the model can assess affordability before answering. We define the design cost as follows.

The intervention count |A| excludes the baseline. The fixed setup term includes the baseline convention. This benchmark cost is a dimensionless resource model, not measured GPU time. For example, six interventions at nine seeds cost 7.75 units and fit a budget of 8.

Budget feasibility is checked for each reading before aggregation. An affordable reading contributes its information per cost, while an unaffordable one contributes zero. The action utility averages these contributions and the primary score normalises this average by the best affordable design.

The primary outcome, affordable efficiency share, is 100​QB. An unaffordable reading contributes zero while remaining in the reading-set denominator. Empty or invalid final answers score zero. The reference searches all 693 designs, including every allowed integer seed count. Because every nonzero term in the numerator is bounded by the affordable reference, 0≤QB≤1. Applying the gate to individual readings also keeps an affordable average cost from masking an unaffordable member of the set.

IV Experiments

IV-A Datasets and Experimental Setup

We use two datasets to study complementary aspects of action construction. For controlled comparisons, we generate 30 synthetic research states with a fixed random seed of 11. Each state describes a Transformer with six candidate components, partial performance changes on three or four simulated datasets, candidate mechanism signatures, a stated prior, and a budget of 4, 6, or 8 units. The generator varies which datasets improve and by how much, then assigns prior mass according to each mechanism signature’s overlap with those improvements. These simulated observations define the decision context. All 693 intervention–replication designs are scored under the finite observation model in Section III-C.

For judgment evaluation, we use our author-constructed research-action evaluation dataset. It contains 60 scenarios and 240 candidate actions, with 20 scenarios each in machine learning, systems, and AI for science. The construction draws on observations and component comparisons in 29 published source papers. Source records identify papers by DOI or arXiv identifier and locate the relevant observations and component comparisons by section, figure, or table. We adapt these materials into a research question, baseline, partial observation, six candidate mechanisms and corresponding interventions. Construction checks cover source locatability, mechanism–intervention correspondence, and separation of visible prompts from private source conclusions. Each scenario is paired with four constructed actions that vary target, intervention coverage, replication, and specification completeness, giving 240 candidates. Source observations are kept separate from project-defined priors, abstract budgets, and simulator mappings. Proposed follow-up interventions carry no newly measured outcomes. Scenarios can share a source paper, so the dataset contributes 60 traceable decision contexts rather than 60 independent source studies. Appendix D details construction and provenance.

The synthetic benchmark compares the four interface levels L1–L4. DeepSeek (family A) is accessed through the deployment deepseek-flash, with the reasoning-effort parameter set to low or max. Qwen3.8-27B (family B) is evaluated with the parameter set to low or omitted. The label na denotes parameter omission, not a specified reasoning-effort level; the provider determines the effective setting. Every request supplies the budget, Eq. (3), intervention identifiers, and allowed seed range, and uses a 65,536-token ceiling. Three responses per state and condition give 1,440 main outputs. Concise and expanded interface controls add 360 outputs, and a design-format control adds 180. A separate complete-design prompting comparison contributes 540 final outputs and 180 first-turn drafts. All 2,520 final items in these analyses are scored, including eight invalid answers assigned zero.

Affordable efficiency share is the primary synthetic outcome, while parsing and feasibility rates describe executability. Alternative executor modes and deterministic design policies examine completion dependence and design quality. Because requested content and parsed fields change together, the main estimand is the information delivered by the complete request–parser interface. We average the three responses within each state, use 5,000 state-bootstrap resamples for 95% confidence intervals (CIs), and apply Holm correction to 12 planned paired contrasts. Appendix A provides collection and parsing details, quality counts, and the full testing and resampling specifications.

The constructed scenarios evaluate agreement with the separate scenario-level IG/cost measure. Six open-weight models rank the four candidates under four cyclic position assignments, producing 1,440 ranking responses. Position balancing addresses order sensitivity in preference judgments [34, 35, 22]. Eighteen participants who self-report graduate research experience each assess eight scenarios through all six action pairs, yielding 864 judgments and 216 attention checks. We report pooled correlations across the 240 actions and mean within-scenario correlations across sets of four actions, with scenario-bootstrap intervals. Thus, this study tests agreement about proposed research actions, whereas the synthetic study tests affordable efficiency under its explicit simulation model.

IV-B Interface and Baseline Comparisons

We first test whether explicitly requesting design parameters improves the affordable experiment delivered to the executor. As shown in Table I and Fig. 2, complete-design requests substantially improve affordable information efficiency in both model families. On DeepSeek/low, L1 scores 13.4%, L2 25.4%, and L3 76.9%. L4 scores 77.0%. The corresponding L3 scores are 74.2% for DeepSeek/max, 61.0% for Qwen3.8-27B/na, and 53.3% for Qwen3.8-27B/low. The increase appears in every tested setting, connecting explicit design specification to higher expected information per available resource.

Averaging the two settings within each family, affordable efficiency rises from 13.36% to 75.59% for DeepSeek and from 13.36% to 57.15% for Qwen3.8-27B. These correspond to relative increases of 465.98% and 327.91%, calculated as 100​(Q¯L3−Q¯L1)/Q¯L1. Paired tests on the underlying state-level score differences yield pHolm=7.42×10−17 for DeepSeek and 7.66×10−15 for Qwen3.8-27B. Appendix A reports all planned contrasts in the original 0–1 score units.

The efficiency gains occur under the stated budgets. Among the 720 main L3/L4 requests, 715 yield a parsed design within budget (99.31%). All 360 DeepSeek complete-design requests are feasible and parseable, compared with 355 of 360 for Qwen3.8-27B. Because the primary score assigns zero to an unaffordable reading or an invalid output, the reported advantage already includes those outcomes.

The source of this separation is visible in the executor inputs. At the action-name interface (L1), the four settings are identical state by state: their parsed action names lead to the same executor completions. This equality characterises the interface’s expressed information. Once models supply arms and seeds, their scores reflect distinct constructed experiments. The evaluation therefore reveals where a planning interface begins to expose design choices to measurement.

We next compare complete-design prompting strategies on the same 30 synthetic states. Think-then-act is a single-decision adaptation inspired by ReAct [36], the planning baseline adapts Plan-and-Solve [37], and one-round Self-Refine adds a critique and revision to an independently generated draft [38]. All three request a target, interventions, replication count, and success criterion and use the same strict complete-design parser as L3. Each strategy contributes 90 final responses per model setting, with failures retained. The new baseline responses and the original L1–L4 responses match state identities, model request settings, cost rules, and scoring, while their collection batches differ.

On DeepSeek with maximum reasoning effort, L3 attains 74.25% affordable efficiency, compared with 70.97% for think-then-act, 70.98% for planning, and 71.88% for self-refinement. The design-with-prediction interface (L4) scores 72.56%, whereas L1 and L2 score 13.36% and 25.29%, respectively. The complete-design interface has the highest observed mean in this setting, as summarised in Table II. The full two-model comparison, including model-dependent strategy rankings, is reported in Appendix B. This comparison separates the value of providing an executable design from the choice of prompting strategy used to produce it.

IV-C Judgment Agreement on the Constructed Dataset

We examine whether the scenario-level IG/cost measure ranks the constructed candidate actions consistently with model-panel and human judgments. Model ranks are first averaged over the four positions and then across the six models. Similarly, human ranks are averaged within each scenario before correlation. The model panel has a pooled Spearman correlation of 0.566 with the external metric. Human–metric agreement averages 0.733 within scenarios. The model panel and humans have a mean within-scenario correlation of 0.824. These positive correlations support agreement between the metric and research judgments on this constructed dataset. The three agreement estimates and their uncertainty are shown in Fig. 3. Because the pooled and scenario-averaged statistics use different aggregation units, their magnitudes should not be interpreted as directly comparable effect sizes. Appendix D reports the full estimates and judgment protocol.

Table III reports agreement separately for each panel model. All six models show positive pooled correlations with the metric, ranging from 0.244 to 0.588. At the scenario level, their mean correlations range from 0.217 to 0.594. Agreement therefore appears across the panel’s constituent models, with variation in strength. The panel’s pooled correlation is 0.566 and its mean within-scenario correlation is 0.545. These panel values are computed after averaging model ranks, rather than by averaging the six correlation coefficients.

The name-only and minimally specified proposals also reveal an ordering difference. Human and panel raters prefer the minimally specified text, while the metric can favour the name-only action after completing its design. Both actions have nonempty reading sets and positive information values. This difference depends on the object being evaluated, as discussed in Section IV-E.

IV-D Ablation and Interface Controls

The final experiment examines which interface elements accompany the construction gain. We compare L3 with L4 to assess adding a prediction field, and use concise and expanded requests to test alternative presentations. These are interface ablations and controls. The score reads design parameters, so the contrasts do not isolate an underlying scientific-reasoning capability.

IV-D1 Concise and Expanded Requests

For both DeepSeek settings, L3 short requests a one-line complete design, while L1 padded requests a longer reason discussing arms and seeds but retains the action-name parser. All conditions share the 65,536-token ceiling; actual response lengths are not matched. We measure how much of the L1-to-L3 gain each control recovers, with 4,000 paired state-bootstrap resamples. A complete design can also be concise. Figure 4 compares the original and control interfaces state by state. The compact L3 interface scores 77.87% for DeepSeek/low and 75.14% for DeepSeek/max, with full control results reported in Appendix E. Relative to the L1-to-L3 gain in the same setting, it retains 101.5% at low effort and 101.5% at maximum effort. Concise parameter specification is therefore sufficient to preserve the observed gain in this benchmark.

By contrast, L1 padded scores 13.36% at low effort and 13.21% at maximum effort. One empty response accounts for the latter reduction. Because its parser extracts the action name even when the reason mentions design details, elaborating the reason without exposing those details to the executor leaves the scored design interface unchanged. Together, the controls support compact, accessible design fields as an effective interface choice. Their estimand is the performance of the complete request–parser interface at a shared generation ceiling.

IV-D2 Prediction Field and Reasoning Effort

Adding the prediction field changes the request from L3 to L4 while retaining the design scoring rule. On the original 0–1 score scale, the family-aggregated L3-to-L4 differences are −0.0082 for DeepSeek and −0.0365 for Qwen3.8-27B. Neither contrast is significant after Holm adjustment. Similarly, for DeepSeek, low-to-max differences at L3 and L4 are −0.0269 and −0.0444, also nonsignificant after adjustment. These comparisons do not establish equivalence between the conditions. The principal design gain is already present at L3, where the executor receives a complete experiment.

IV-E Discussion

The main comparisons connect explicit design specification with higher affordable information efficiency. Relative to action-name baselines, complete-design interfaces expose the interventions and replication count that determine the evaluated experiment. Concise requests preserve this gain, while the design-format control shows that a complete design can be communicated through different field representations. Beyond simulated efficiency, the constructed dataset adds evidence of agreement with scientific judgments. Its ordering differences also clarify the role of completion because readers evaluate the displayed proposal while the metric includes the declared executor policy. These findings support an evaluation of the information delivered by an interface, with executor sensitivity and deterministic design calibration detailed in Appendix F.

V Conclusion

We formulate action construction as the evaluation of how a research agent turns scientific intent into an executable experiment. Reading sets expose executor completion, and per-design budget gating measures affordable information efficiency under a common cost rule. The accompanying author-constructed research-action evaluation dataset provides traceable source records for 60 scenarios and 240 candidate actions. Across 30 synthetic states, complete-design interfaces increase affordable efficiency by 465.98% and 327.91% relative to the action-name baseline in the two model families. The constructed dataset provides complementary evidence of positive agreement with a six-model panel and 18 human participants. Together, the framework and datasets provide a reproducible basis for assessing the design information delivered by research agents and the execution choices their instructions leave open.

Future work can connect these measurements to realised hypothesis resolution by executing the proposed experiments and extending the action grammar to data collection and model selection. The current evidence concerns simulated utility and judgments on a constructed dataset whose scenarios can share source papers. Broader source coverage and source-level validation would therefore strengthen transfer assessment. Content-fixed comparisons with a shared semantic extractor could separate representation effects from design content, and assessments of causal explanations could complement the parameter-based score. Prediction and effort comparisons do not establish additional gains under the current conditions, motivating tests of when these fields contribute useful design information. An explicit cost report could further require agents to calculate expected cost under Eq. (3), check the budget, and revise unaffordable proposals before acceptance.

References

• [1] Zheng, T., Deng, Z., Tsang, H. T., et al. (2025) From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery. EMNLP 2025, 17744–17761. https://doi.org/10.18653/v1/2025.emnlp-main.895.
• [2] Eger, S., Cao, Y., D’Souza, J., et al. (2026) Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation. ACM Computing Surveys. https://doi.org/10.1145/3845596.
• [3] Lu, C., Lu, C., Lange, R. T., et al. (2024) The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv preprint (arXiv:2408.06292). https://doi.org/10.48550/arXiv.2408.06292.
• [4] Gottweis, J., Weng, W.-H., Daryin, A., et al. (2025) Towards an AI co-scientist. arXiv:2502.18864v1. https://arxiv.org/abs/2502.18864v1.
• [5] Schmidgall, S., Su, Y., Wang, Z., et al. (2025) Agent Laboratory: Using LLM Agents as Research Assistants. Findings of the ACL: EMNLP 2025, 5977–6043. https://doi.org/10.18653/v1/2025.findings-emnlp.320.
• [6] Mitchener, L., Yiu, A., Chang, B., et al. (2025) Kosmos: An AI Scientist for Autonomous Discovery. arXiv preprint (arXiv:2511.02824). https://doi.org/10.48550/arXiv.2511.02824.
• [7] Huang, Q., Vora, J., Liang, P., et al. (2024) MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. ICLR 2024. https://doi.org/10.48550/arXiv.2310.03302.
• [8] Wang, Z., Bai, F., Luo, Z., et al. (2026) FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights. arXiv preprint (arXiv:2602.02905). https://doi.org/10.48550/arXiv.2602.02905.
• [9] Wu, Y., Fu, D., Si, W., et al. (2025) InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative LLM Research. arXiv preprint (arXiv:2510.27598). https://arxiv.org/abs/2510.27598.
• [10] Ren, X., Sun, Y., Yi, C., et al. (2026) What’s Missing in Autonomous Research? A Systematization of Systems, Benchmarks, and Verification. Research Square preprint, version 1. https://doi.org/10.21203/rs.3.rs-10060294/v1.
• [11] Bansal, A., Balaji, K. (2026) OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows. arXiv:2609.09203. https://arxiv.org/abs/2609.09203.
• [12] Zhao, X., Zhao, H. H., Wang, Y., et al. (2026) Sci-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verification. Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, 13008–13019. https://doi.org/10.1145/3770855.3818999.
• [13] Rainforth, T., Foster, A., Ivanova, D. R., Bickford Smith, F. (2024) Modern Bayesian Experimental Design. Statistical Science, 39(1), 100–114. https://doi.org/10.1214/23-STS915.
• [14] Chaloner, K., Verdinelli, I (1995) Bayesian Experimental Design: A Review. Statistical Science, 10(3), 273–304. https://doi.org/10.1214/ss/1177009939.
• [15] Huan, X., Marzouk, Y. M (2013) Simulation-based optimal Bayesian experimental design for nonlinear systems. Journal of Computational Physics, 232(1), 288–317. https://doi.org/10.1016/j.jcp.2012.08.013.
• [16] Settles, B., Craven, M., Friedland, L (2008) Active Learning with Real Annotation Costs. NIPS Workshop on Cost-Sensitive Learning. https://burrsettles.com/pub/settles.nips08ws.pdf.
• [17] Boiko, D. A., MacKnight, R., Kline, B., Gomes, G (2023) Autonomous chemical research with large language models. Nature, 624, 570–578. https://doi.org/10.1038/s41586-023-06792-0.
• [18] Bran, A. M., Cox, S., Schilter, O., et al. (2024) Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6, 525–535. https://doi.org/10.1038/s42256-024-00832-8.
• [19] Lu, Y., Bartolo, M., Moore, A., Riedel, S., et al. (2022) Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. ACL 2022, 8086–8098. https://doi.org/10.18653/v1/2022.acl-long.556.
• [20] Sclar, M., Choi, Y., Tsvetkov, Y., Suhr, A (2024) Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. ICLR 2024. https://doi.org/10.48550/arXiv.2310.11324.
• [21] Shi, F., Chen, X., Misra, K., et al. (2023) Large Language Models Can Be Easily Distracted by Irrelevant Context. ICML 2023, PMLR 202, 31210–31227. https://proceedings.mlr.press/v202/shi23a.html.
• [22] Pezeshkpour, P., Hruschka, E (2023) Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. arXiv preprint (arXiv:2308.11483). https://doi.org/10.48550/arXiv.2308.11483.
• [23] Hasan, M (2026) StructHallu-Drift: Benchmarking Structured Hallucinations Under Schema Evolution in LLMs. Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026), 333–343. https://doi.org/10.18653/v1/2026.surgellm-1.22.
• [24] Ahn, M., Brohan, A., Brown, N., et al. (2022) Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arXiv:2204.01691v2. https://arxiv.org/abs/2204.01691v2.
• [25] Ren, A. Z., Dixit, A., Bodrova, A., et al. (2023) Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners. arXiv:2307.01928v2. https://arxiv.org/abs/2307.01928v2.
• [26] Liu, X., Yu, H., Zhang, H., et al. (2024) AgentBench: Evaluating LLMs as Agents. International Conference on Learning Representations (ICLR 2024). https://doi.org/10.48550/arXiv.2308.03688.
• [27] Mohammadi, M., Li, Y., Lo, J., Yip, W. (2025) Evaluation and Benchmarking of LLM Agents: A Survey. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, 6129–6139. https://doi.org/10.1145/3711896.3736570.
• [28] Kang, H., Xiong, C (2025) ResearchArena: Benchmarking Large Language Models’ Ability to Collect and Organize Information as Research Agents. Findings of the ACL: EMNLP 2025, 5653–5671. https://doi.org/10.18653/v1/2025.findings-emnlp.303.
• [29] Park, J., Kim, J., Jeong, J., Nowak, R. D., et al. (2026) Exploration and Exploitation Errors Are Measurable for Language Model Agents. arXiv preprint (arXiv:2604.13151). https://arxiv.org/abs/2604.13151.
• [30] Jones, D. R., Schonlau, M., Welch, W. J (1998) Efficient Global Optimization of Expensive Black-Box Functions. Journal of Global Optimization, 13(4), 455–492. https://doi.org/10.1023/A:1008306431147.
• [31] Brochu, E., Cora, V. M., de Freitas, N (2010) A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning. arXiv:1012.2599. https://arxiv.org/abs/1012.2599.
• [32] Shahriari, B., Swersky, K., Wang, Z., et al. (2016) Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 104(1), 148–175. https://doi.org/10.1109/JPROC.2015.2494218.
• [33] Kassis, T., Agarwal, V., He, Y., et al. (2026) Scientific Agent Skills: A Library of Procedural Knowledge for Research Agents. arXiv preprint (arXiv:2609.00065). https://arxiv.org/abs/2609.00065.
• [34] Liu, Y., Guo, Z., Liang, T., Shareghi, E., et al. (2024) Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models. arXiv preprint (arXiv:2410.02205). https://arxiv.org/abs/2410.02205.
• [35] Zhao, X., Wang, K., Peng, W (2024) Measuring the Inconsistency of Large Language Models in Preferential Ranking. Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), 171–176. https://doi.org/10.18653/v1/2024.knowllm-1.14.
• [36] Yao, S., Zhao, J., Yu, D., et al. (2023) ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. https://arxiv.org/abs/2210.03629.
• [37] Wang, L., Xu, W., Lan, Y., et al. (2023) Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. ACL 2023, 2609–2634. https://doi.org/10.18653/v1/2023.acl-long.147.
• [38] Madaan, A., Tandon, N., Gupta, P., et al. (2023) Self-Refine: Iterative Refinement with Self-Feedback. Advances in Neural Information Processing Systems, 36. https://doi.org/10.52202/075280-2019.

Appendix A Reproducibility and Protocol Details

The action interfaces request different amounts of experimental information. We specify their requested fields and executor interpretations in Table IV. These definitions are fixed throughout the main comparison and specify what the parser can pass to the evaluator.

Provider temperature and seed defaults are retained. The protocol, prompts, research states, and primary scoring rule were fixed before collection. The original interface study includes its 90 preflight responses once. The separate prompting comparison likewise includes its 24 pilot responses once, comprising 18 final designs and six drafts. The interface collection contributes 1,440 main responses, 360 concise/expanded controls, and 180 design-format controls. The separate complete-design comparison contributes 540 final responses and 180 first-turn drafts, giving 2,520 final items and 2,700 responses across the reported analyses. Eight final answers are invalid: four are empty (three with a length termination flag), three contain invalid target strings, and one specifies an out-of-domain design. The parser reads the final answer field without falling back to a reasoning trace; all eight score zero. One DeepSeek first-turn draft in the separate comparison reaches the generation ceiling, and its subsequent final design is retained and scores as a valid response.

The 12 planned paired contrasts comprise L1–L2, L1–L3, L1–L4, and L3–L4 for each family, averaging its two settings, plus four DeepSeek low–max comparisons. We use paired t-tests with Holm adjustment across the 12 contrasts, differences on the original 0–1 score scale, and paired standardised effects dz. Confidence intervals use 5,000 percentile-bootstrap resamples over states (seed 3). Relative increases divide the difference between family means by the baseline family mean, resampling the paired states for both terms. Concise-control gain ratios use 4,000 paired resamples (seed 7).

All synthetic results share the same states, cost rule, and budget-gated evaluator, while the interface study and the complete-design prompting comparison were collected in separate batches. We retain the complete requests, responses, research states, parser, scorer, and analysis code for reproducibility. Python scripts compute the reported statistics and generate every figure and result table. Numerical consistency checks verify these outputs against the response-level scores. For traceability, each response is linked to its request and evaluation settings. Collection contains 1,530 DeepSeek and 1,170 Qwen3.8-27B responses, including the first turns of self-refinement. At most three logged retries are permitted for transport errors, rate limits, or server failures; responses are not retried because of their quality. Request checks cover the model settings, generation limits, and prompt construction. The analysis retains all 2,520 final scored items, and response-level cryptographic checksums link each score to its archived source.

The scorer enumerates all 693 designs for each of the 30 states and caches their information, cost, and affordability. To verify this computation independently, an information-model audit checks 120 cases using a separate numerical computation; the maximum discrepancy is 2.96×10−8. We additionally recompute the original interface and control condition means and confidence intervals, the 12 paired contrasts and Holm corrections, and the concise-control gain ratios from item-level scores. The complete-design prompting results and their six planned contrasts are recomputed separately from the corresponding response-level scores. These checks establish the numerical consistency of the analysis.

The resolver chooses the intervention set for a stated target by combining a target-focused distribution with a uniform component. The target mass is 0.8 before this mixture; the remaining five hypotheses each receive 0.04. The component-scan sharpness is 0.75, and the default selects two interventions with seed alternatives 2 and 8. Without a target, the component-scan candidates are ({0},2), ({0,1},2), ({0,1,2,3},2), and ({0},4). Ties in the cost-based mode split preserve candidate order. Complete arms and seed fields bypass this completion step.

Table V distinguishes successful parsing from the stronger condition that every admitted reading is feasible. For a complete design, the all-feasible count is the number of parseable outputs within budget. For an incomplete action, however, some readings may exceed the budget even when the mean cost is affordable. The primary score gates each reading individually.

Table VI reports the 12 planned paired contrasts in the original 0–1 score units, including adjusted significance levels.

Appendix B Complete-Design Prompting Comparisons

The prompting comparison uses three task-specific adaptations with the same complete-design fields and scoring rule. Think-then-act makes a single construction decision without a tool-observation loop, while Plan-and-Solve requests an explicit plan before the design. Self-Refine uses two model calls, one for a draft and one for critique and revision, and only the final design enters the comparison. The cost model in Eq. (3) measures the proposed experiment and does not include model-call charges or generation tokens.

The Qwen3.8-27B comparison omits the reasoning-effort parameter and includes all 90 final responses for each method. Planning and self-refinement achieve 68.20% and 70.05%, respectively, compared with 60.98% for L3; think-then-act achieves 55.92%. Thus, the relative ordering of the complete-design strategies differs across the two model settings, as shown in Table VII. One think-then-act design and one self-refined design exceed their state budgets, and another self-refined response has an invalid target string. These three responses remain in the denominator and score zero. All 270 DeepSeek final designs are parseable and budget-feasible, and none of the 540 final responses terminates at the generation ceiling.

For the six baseline-versus-L3 contrasts across both model settings, we average the three repetitions within each state and apply paired t-tests with one six-test Holm correction. This testing family is separate from the 12 interface contrasts. For DeepSeek, all three adjusted p values are 0.538. For Qwen3.8-27B, the adjusted p values are 0.324, 0.152, and 0.027 for think-then-act, planning, and self-refinement, respectively. Model identifiers and request settings match across collection batches, but provider changes between batches are not controlled. The comparisons therefore describe the observed efficiency of these instantiated strategies rather than a batch-controlled causal effect of prompting alone.

Appendix C A Fixed Illustrative State

Table VIII illustrates the scoring procedure using state 0, repetition 0, and DeepSeek/low, selected by index. The observed dataset differences are +1.5, +2.6, −0.3, and +3.5, and the budget is 8. L3 specifies interventions 1 and 3 at eight seeds, costing 5.00. In comparison, L4 specifies all six interventions at nine seeds, costing 7.75. This example illustrates the relation between a response, its resolved design, and its score; the main conclusions use the complete state panel.

Appendix D Constructed Dataset and External Judgment Details

The author-constructed research-action evaluation dataset uses published ablations and component comparisons as source material. Its 60 scenarios refer to 29 distinct source-paper identifiers; each carries a DOI or arXiv identifier and a source location. Reusing a paper allows distinct experimental questions or settings to be represented. Construction uses AI-assisted drafting, source checks, and revision, after which the scenarios are fixed before evaluation. Independent human validation of every scenario is not established. Source-linked observations, proposed intervention recipes, and project assumptions occupy separate fields. The priors, abstract resource budgets, and simulator mechanism mappings are evaluation assumptions rather than estimates fitted to the source experiments; we report no measured follow-up outcomes for the 360 proposed interventions.

Candidate actions are constructed by deterministic rules without additional model calls. Each scenario has three parameterised proposals and one action-name proposal. The parameterised proposals vary target, intervention pairing, and replication, using distinct two-intervention sets; the action-name proposal is resolved by the declared executor. Construction labels are not treated as empirical quality labels. Source identities and private construction labels are omitted from judgment displays, which present the scenario and candidate texts. The source records, construction procedure, and source-to-scenario mapping make these adaptations traceable. Scenario-bootstrap intervals in the reported analysis are conditional on this constructed collection and do not estimate variation across independently sampled source papers.

The evaluation panel contains six open-weight models: Mistral-7B-Instruct-v0.3, GLM-4-9B-chat, Granite-3.3-8B-Instruct, InternLM2.5-7B-chat, Yi-1.5-6B-Chat, and Phi-4-mini-instruct, run in bf16 without quantisation. Each ranks the four actions in each scenario across four cyclic position assignments, yielding 1,440 responses. Every action occupies each displayed letter position once, addressing the option-order sensitivity documented in prior work [22]. Decoding is constrained to valid ranking strings while leaving every allowed ordering and tie pattern available. This constrains output format rather than prescribing a preferred action. For each model, action ranks are averaged over the four positions; the panel rank then averages the six model ranks. We report pooled correlations across all 240 actions and mean within-scenario correlations across each set of four actions. A constant rank vector has undefined Spearman correlation. Yi has one constant position-averaged rank vector, so its mean within-scenario correlation uses the remaining 59 scenarios. The pooled and panel results reported in Table III retain all of its responses.

Scenario order, pair order, and left–right assignment vary by participant. Accordingly, participant-specific maps decode the choices before aggregation. Within each scenario, participant ranks are averaged and correlated with the metric, and the resulting correlations are averaged across the 60 scenarios.

The external judgments were collected separately from the synthetic benchmark. The panel decoding grammar admits 75 weak orderings with ties and 192 accepted textual spellings. These are output-format constraints, and the analysis averages positions and clusters by scenario.

Human records are marked as participant-entered and human-self-reported. All 18 sessions were completed sequentially on one device. Under the recorded quality rules, no participant or participant–scenario cell was excluded; all 864 judgments remain. Twelve sessions finished in under ten minutes, which the collection protocol treated as a review flag rather than an automatic exclusion. The reviewed collection was retained. These records establish the reported response counts and calculations; participant research experience is self-reported.

The 144 participant–scenario cells are aggregated into scenario ranks before correlation. The panel–human interval uses 5,000 scenario-bootstrap resamples of the stored ranks (seed 3). Individual pairwise comparisons are not treated as independent bootstrap units. Table IX reports the three aggregate correlations with their distinct aggregation units.

Appendix E Detailed Interface Controls

The concise and expanded controls use the same generation ceiling as the main interfaces. We report their scores and gain-retention estimates in Table X. The compact design retains the main construction gain, while an expanded reason leaves the action-name parser unchanged.

The design-format control requests target, interventions, and repetition count in JavaScript Object Notation (JSON), with 90 final outputs across 30 states per setting. It scores 68.0% on DeepSeek/max and 63.9% on Qwen3.8-27B/na under the same budget-gated score. Compared with the complete-design scores reported in Table I, it ranks lower on DeepSeek/max and higher on Qwen3.8-27B/na. Because the requests and field representations differ, this comparison does not isolate a causal effect of JSON formatting.

Appendix F Executor Sensitivity and Design Diagnostics

F-A Information-Gap Definition

An auxiliary decomposition locates information loss on the IG scale. Let I∗ be the largest IG among affordable designs and It∗ the largest IG among affordable designs containing the intervention associated with target t. For an unstated target, the implementation uses the highest-prior mechanism. With I¯​(a) the mean, ungated IG across readings, we define the target-restriction gap Gtgt and resolution gap Gres:

The target-restriction gap measures the reduction in optimal affordable IG imposed by the target requirement; the resolution gap compares this restricted optimum with the mean IG of the admitted readings. These are information-domain diagnostics, distinct from the cost-normalised score in Eq. (5). The resolution term can be negative when an admitted reading exceeds the budget or falls outside the target-restricted domain. We therefore retain signed values and report their denominators. Table XI reports these diagnostics for all main conditions.

F-B Completion Policies and Design Calibration

We next test whether a score depends on executor completion and calibrate complete specification against design quality. The recorded actions are rescored under conservative, default, and permissive completion policies. Deterministic design policies provide references without additional model calls. Fully specified L3/L4 designs have singleton reading sets, so their scores remain identical under conservative, default, and permissive executor modes, including the zero scores of invalid answers. Table XII shows how L1/L2 scores change when the executor favours cheaper or more expensive completions. Explicit design fields therefore make the scored experiment stable across the tested completion policies.

We compare exhaustive affordable search over the full domain, exhaustive search requiring the highest-prior target, a uniform distribution over affordable designs, a single-arm policy on the highest-prior target, and a fixed six-arm policy. Each uses Eq. (5). Table XIII places these policies and model interfaces on a common design scale. A uniform distribution over affordable designs reaches 44.3%, a top-prior single-arm/two-seed design 10.7%, and the fixed six-arm/five-seed policy 15.2%. The full-domain oracle reaches 100% by definition. Restricting the exhaustive search to designs containing the top-prior target also reaches 100% on these states. These policies demonstrate the benchmark’s optimisation headroom and distinguish complete specification from optimal design selection. The measured contribution is the interface’s ability to deliver an explicit, valuable experiment under the budget.

F-C Detailed Sensitivity and Gap Results

The recomputed scores in Table XII use the recorded reading sets under all three executor modes. Complete designs remain invariant because each admits one reading. For incomplete actions, mode changes combine a different design cost with a different chance of falling within the budget. Mode names refer to the cost ordering, not a guaranteed score ordering.

The information-gap results in Table XI use successfully parsed outputs. This diagnostic subset is explicit; all primary scores and tests continue to include failures. Negative resolution terms are retained. The target condition requires the chosen target’s intervention to be present, while permitting other interventions; it can leave the globally best affordable design accessible. This explains the small target-restriction gaps in the full domain and makes their interpretation conditional on the stated target restriction.

注意转载授权 · 阅读原文