反思式提示优化提升 Deep-Research 智能体引用奖励：主题不相交多随机种子协议揭示可迁移增益与提示级过拟合

Abstract

Prompt optimization is an attractive alternative to reinforcement learning for adapting language-model agents, since it is cheap and produces human-readable artifacts. On factual tasks, however, it carries a failure mode that scalar-reward reinforcement learning does not. An optimizer that reads the training examples can write their answers into the prompt, so an in-distribution gain may be memorization rather than transferable behaviour. We apply reflective prompt evolution (GEPA) to the two system prompts of a multi-agent deep-research agent and evaluate it under a topic-disjoint split in which no topic is shared across train, validation, and test. Held-out citation reward rises by 0.079, from 0.702 to 0.781, with a bootstrap 95% confidence interval of [0.051, 0.107], and the gain replicates across three seeds (mean 0.107). Nearly half of the winning prompt is a memorized answer key, yet a controlled ablation shows this content earns no reward even on the exact training questions it encodes. The large validation-minus-held-out gap is therefore small-sample selection under a noisy judge, not memorization. Varying the verifier isolates what makes the gain transfer. Decomposed per-dimension feedback is the causal ingredient, and collapsing it to a single scalar reward makes held-out transfer negative. The gain persists under an out-of-family judge and is largely reproduced by a hand-written version of the discovered recipe. Credible evaluation of prompt optimizers therefore requires held-out topics, multiple seeds, and verifier ablations. We release the candidate prompts, lineage diffs, figures, tables, and analysis.

1 Introduction

Language-model agents are increasingly adapted to downstream tasks with reinforcement learning methods such as Group Relative Policy Optimization [Shao et al., 2024], which typically need thousands of rollouts and collapse a rich execution trace into a single scalar reward. Prompt optimization is an attractive alternative because it is cheap, needs no training infrastructure, and produces human-readable artifacts that practitioners can audit. Frameworks such as DSPy [Khattab et al., 2024] have made this practical by treating a system as a program of language-model calls and optimizing its prompts automatically, with instruction-and-demonstration search in MIPROv2 [Opsahl-Ong et al., 2024] and reflective prompt evolution in GEPA [Agrawal et al., 2025].

On factual tasks, prompt optimization has a failure mode that scalar-reward reinforcement learning does not. An optimizer that sees the training examples can copy their specific answers into the prompt, so the reported gain then reflects memorized lookups on seen topics rather than a transferable change in how the agent behaves. This is a form of reward hacking [Skalse et al., 2022, Amodei et al., 2016], in which the optimizer raises the measured reward without improving the behaviour that the reward is meant to track. It is the primary risk when a reflective optimizer is allowed to read and rewrite prompts against a training signal. The risk is sharpened when that signal is a rubric-style reward scored by language-model judges, which are themselves prone to hacking in reinforcement learning with verifiable rewards [Mahmoud et al., 2026, Tyagi et al., 2026, Wang et al., 2026b], as ours is (Section 3.2). A study that reports only an in-distribution gain therefore cannot tell the reader how much of that gain will survive on data the optimizer never saw, and closing that gap is the measurement problem we address.

Our contribution is a generalization-first protocol that separates the two. We use a topic-disjoint split in which no article family is shared across train, validation, and test, and a held-out test set large enough for tight confidence intervals. Under this design the held-out gain reports only transferable behaviour, and the difference between the validation gain and the held-out gain lets us test whether the apparent gain is memorization. The optimizer we use as the instrument is GEPA, the Genetic-Pareto reflective prompt evolution method of Agrawal et al. [2025]. GEPA feeds a language-model reflector the full trajectory (reasoning, tool calls, and tool outputs) together with diagnostic textual feedback, so the reflector can diagnose why a candidate failed and propose a targeted rewrite. It keeps a Pareto frontier of candidates so the search does not collapse to a local optimum. Because GEPA reads execution traces, it can both discover transferable rules and copy specific facts into the prompt, which is exactly the mixture our protocol is designed to separate. Our configuration follows recent practice. Decagon [2026] report that a frontier reflection model is needed, that the optimization sample size has a sweet spot around 20 to 100 examples, and that prompts bloat without length regularization, and GEPA Team [2026] note that enlarging the reflection batch causes context overload. We therefore use a strong reflector and keep the train and validation sets at 24 examples each, and we observe the bloat effect when GEPA’s Pareto selection rejects its two longest candidates.

The object we optimize is a source-attribution reward for deep-research agents [Shi et al., 2025]. Onweller et al. [2026] show that deep-research agents routinely emit citations that are present but not verified, where the cited page exists yet fails to support the specific claim. They propose parsing and evaluation machinery for source attribution. Our gated composite reward (Section 3.2) operationalizes those failure modes through the levers link_works, relevant_content, and fact_check, so optimizing against it targets a documented attribution-quality gap. The agent’s search behaviour is in turn shaped by its harness [Lumer et al., 2026], and Sen et al. [2026a] show that harness design materially reshapes agentic search quality, which is why prompt edits that change how the researcher searches and cites can move the reward at all. More broadly, structured-retrieval agent systems [Sen et al., 2026b] situate this work in the line of retrieval-augmented agents [Lumer et al., 2025], though our task is single-shot cited reporting rather than long-horizon conversational memory.

This paper makes the following contributions.

• •

  A generalization-first evaluation protocol, combining topic-disjoint held-out testing, a controlled topic-shared ablation, and multi-seed replication, that tests whether the validation-minus-test gap is memorization and finds that it is not. The memorized answer key the optimizer writes does not improve the reward, and the gap is consistent with small-sample selection under a noisy judge.

• •

  A rigorous held-out evaluation of reflective prompt optimization on a multi-agent research system with a language-model-judge citation reward, with bootstrap confidence intervals and a paired significance test on 500 tasks, replicated across 3 optimization seeds (mean held-out gain 0.107, significant in every run).

• •

  A controlled verifier ablation identifying decomposed per-dimension feedback as the causal ingredient behind transfer, since collapsing the reward to a single scalar drives held-out transfer negative, while finer resolution and repeated scoring do not raise it and reward gaming tracks the weakest verifiers rather than the strongest.

• •

  A per-lever, error-analyzed account of which prompt edits drove the transferable gain and where the residual failures concentrate, including a prompt autopsy showing that nearly half of the winning prompt by word count is a memorized answer key that does not improve the reward, even on the exact training questions it encodes.

2 Related work

Automatic prompt optimization.

Automatic prompt optimization searches the natural-language input space of a fixed model rather than its weights. Early methods treat the language model as a black-box optimizer that proposes and scores instructions, as in APE [Zhou et al., 2023] and OPRO [Yang et al., 2024], and a parallel line borrows from evolutionary computation by mutating and recombining prompts, as in Promptbreeder [Fernando et al., 2024] and EvoPrompt [Guo et al., 2024]. Later work makes the search more structured and sample-efficient, including PromptWizard [Agarwal et al., 2024], the study of instructions and exemplars by Wan et al. [2024], and the joint instruction-and-demonstration search of MIPROv2 [Opsahl-Ong et al., 2024]. GEPA [Agrawal et al., 2025] extends this line by reflecting on full execution traces in natural language, and we use its reference implementation [GEPA-AI, 2025] as the optimizer in this study. A related text-space optimizer, SkillOpt [Yang et al., 2026c], trains a persistent skill document with bounded, validation-gated edits and an update rule that explicitly forbids hardcoding task-specific values, a design we return to in Section 6.

Optimizing multi-module language-model programs.

Modern agents are multi-module language-model programs, so optimizing them is more than tuning a single instruction [Bai and Shi, 2026]. The DSPy line frames these systems as declarative programs that can be compiled and optimized [Khattab et al., 2022, Khattab et al., 2024], and LangProbe [Tan et al., 2025] provides a benchmark for such programs. Recent methods optimize across modules, whether by combining fine-tuning with prompt search [Soylu et al., 2024], aligning local rewards to a global objective in Optimas [Wu et al., 2025b], or composing policy gradients with prompt optimization in multi-module GRPO [Ziems et al., 2025]. Our system is a two-module program, an orchestrator and a researcher, and we optimize both.

Reflection and natural-language feedback.

A growing body of work treats natural-language feedback, rather than a scalar, as the learning signal. TextGrad backpropagates textual critiques through a system of language-model calls [Yuksekgonul et al., 2025], and Xu et al. [2025] give conditions under which such feedback provably improves a policy. Related test-time methods accumulate reusable lessons in an external memory, as in dynamic cheatsheets [Suzgun et al., 2025] and agent workflow memory [Wang et al., 2024]. These build on prompting strategies that externalize reasoning and action, including chain-of-thought [Wei et al., 2023] and ReAct [Yao et al., 2023]. Reflective prompt evolution inherits the premise that a written diagnosis carries more information than a single reward number.

Reinforcement learning and verifiable rewards for agents.

The alternative to prompt optimization is reinforcement learning on the model weights. GRPO and its relatives are effective but sample-hungry, often needing thousands of rollouts [Shao et al., 2024], and a line of work applies them to retrieval and search agents [Xu et al., 2026], including Search-R1 [Jin et al., 2025], ReSearch [Chen et al., 2025], R1-Searcher [Song et al., 2025], and FrugalRAG [Java et al., 2025], with Wu et al. [2025a] reducing wasteful searches. Because these methods need a reward, verifiable-reward engineering has become its own topic, from generalizing verifiable instruction following [Pyatkin et al., 2025] to verification pipelines such as VerIF [Peng et al., 2025]. Rubric-style rewards are especially prone to hacking, which has motivated policy-aware rubric design [Tyagi et al., 2026, Yang et al., 2026a], direct characterizations of rubric reward hacking [Mahmoud et al., 2026, Wang et al., 2026b], and rubric-free distillation alternatives [Rezaei et al., 2026]. Our reward is a language-model-judged rubric rather than a programmatic verifier, and we optimize the prompt rather than the weights, which avoids rollout cost but introduces the leakage risk we measure.

Citation faithfulness and source attribution.

Our task is citation quality for deep-research agents, where the concern is whether a cited source actually supports its claim. Onweller et al. [2026] document that such agents routinely emit citations that are present but unverified, and they provide parsing and evaluation machinery for source attribution that our reward operationalizes. Faithfulness has also been pursued through training, whether by teaching contextual faithfulness with synthetic tasks and reinforcement learning [Si et al., 2025] or by detecting and mitigating hallucination in reasoning models [Sun et al., 2025], and it is measured by multi-hop verification benchmarks such as HotpotQA [Yang et al., 2018] and HoVer [Jiang et al., 2020]. The reliability of language-model rubric verifiers for this exact setting, deep-research source attribution, has itself been benchmarked against human labels [Leung et al., 2026], which informs the per-dimension judge choices in our reward. Because agent behaviour depends heavily on the surrounding harness [Sen et al., 2026a, Lumer et al., 2026] and retrieval structure [Sen et al., 2026b, Lumer et al., 2025], prompt edits that change how an agent searches and cites can move citation quality without any weight update. We differ from all of this work in what we measure. Prior prompt-optimization studies report in-distribution gains, but on factual tasks such a gain conflates a transferable behavioural change with answers the optimizer memorized from the training examples, and our protocol separates the two, testing directly whether the reported gain survives on topics the optimizer never saw.

3 Method

3.1 System under optimization

The agent is a multi-agent research system in which an orchestrator plans and delegates to a research sub-agent [Lumer et al., 2026] that searches a frozen corpus with a search tool, consolidates citations, and writes a cited report. We optimize the two system prompts.

• •

  The orchestrator prompt, covering planning, delegation, synthesis, report structure, and citation rules.

• •

  The researcher prompt, covering search strategy, evidence standards, and response format.

3.2 Reward

The reward is a gated composite in [0,1], unchanged by this study, designed around the source-attribution failure modes documented for deep-research agents [Onweller et al., 2026].

The answer_gate is a correctness gate, so a wrong answer zeroes the whole reward regardless of citation quality, and it is evaluated by a tiered check that escalates from an exact substring match to a coverage test to a language-model judgment (Appendix C). The citation levers are per-citation scores averaged over a task.

• •

  link_works is 1 when a cited URL exists in the corpus.

• •

  relevant_content measures whether the cited source is about the claim.

• •

  fact_check measures whether the cited source supports the specific claim and figures.

• •

  citation_numbering checks that inline references are numbered and matched to a source list.

The 4 weights and the verbatim judge prompts are given in Appendix C.

3.3 GEPA configuration

We optimize both components with the open-source GEPA implementation [GEPA-AI, 2025], using Pareto candidate selection, a reflection minibatch size of 3, and a rollout budget of 240 metric calls. GEPA is wired to the agent through a standard adapter with three functions. A rollout function runs the agent with the candidate prompts, an evaluation function scores with the composite reward and renders textual feedback that names each lever and surfaces the weakest citations’ judge reasons, and a record function logs traces for the reflector.

3.4 Topic-disjoint split

Each task identifier encodes its source article, and questions from the same article share a slug. We group tasks by slug and assign whole topic groups to test first, then validation, then train, which guarantees zero topic overlap across the 3 splits (Table 1). Train and validation are 24 tasks over 12 topics each, and test is 500 tasks over 254 topics. The small train and validation sets sit within the 20 to 100 window reported for GEPA [Decagon, 2026]. The 500-task set is held-out evaluation data, not optimization data.

3.5 Ablation protocols

Beyond the main held-out evaluation we run two sets of controlled experiments that each vary a single axis of the setup while holding the rest fixed. The first varies the data split and the optimizer seed to test whether the gain is real and whether the validation-minus-held-out gap is memorization, and the second varies the verifier to test which property of the reward signal makes the learned rules transfer.

Split and seed.

A multi-seed protocol repeats the optimization with an identical configuration under 3 independent seeds and compares the held-out gains, which separates the direction and significance of the effect from run-to-run variance in which candidate the optimizer selects. A topic-shared evaluation then measures whether the memorized answer key actually earns reward. We hold the seed and the selected best prompt fixed, run no further search, and score both on questions drawn from the 12 training article families. These questions are the 24 exact training questions the prompt could have memorized, plus 96 fresh in-corpus questions on the same families (8 per family, verbatim-quote verified and deduplicated). Because the frozen corpus is unchanged, link resolution and search behave identically. We add no documents, only new questions over documents the agent can already retrieve. Six families whose answers appear verbatim in the winning prompt are flagged, so the topic-shared set decomposes into memorized and non-memorized families (leaky_split_manifest.json).

Verifier configurations.

The second set of experiments holds the data, policy, reflector, and rollout budget fixed and varies only the verifier GEPA optimizes against. The canonical verifier uses a fine-grained reward, a single scoring pass, and per-dimension decomposed textual feedback that names each citation lever. From it we form 3 single-knob variants: a coarse 3-level reward resolution, 3x repeated scoring with judge averaging and a majority answer gate, and a scalar variant that collapses the per-dimension feedback into one aggregate number. We run each of the 4 configurations at 2 seeds, for 8 optimization runs. We then re-score every optimized prompt on a fixed held-out set of 120 topic-disjoint tasks with a single canonical verifier, so that transfer is comparable across configurations regardless of the verifier each was optimized under. Per configuration we report the validation gain under its own verifier, the canonical held-out gain (transfer), and their difference as a gaming gap.

3.6 Models

The policy agent listed in Table 2 is OpenAI GPT-5.4, the deployment target, and the reflector is OpenAI GPT-5.5 at high reasoning effort, consistent with the frontier reflector requirement reported by Decagon [2026]. The reward’s citation judges are GPT-5-mini for source relevance and GPT-5.4-mini for factual support, and the answer gate uses a language-model judge in its final tier (Appendix C). These per-dimension judge choices follow Leung et al. [2026], who benchmark rubric language-model verifiers against human labels for deep-research source attribution and report that a small model suffices for source relevance while a stronger small model leads on factual support.

4 Experiments

4.1 Experimental setup

After searching on the 24-task validation set, we evaluate the seed prompt and the best prompt on the full 500-task held-out test set with a checkpointed and resumable runner, scoring every task with the same gated composite reward used during optimization. We report the mean paired reward change with a bootstrap 95% confidence interval over tasks [Efron, 1979], a Wilcoxon signed-rank test [Wilcoxon, 1945] and a paired t-test for significance, the effect size dz, and the win, tie, and loss counts. For the lever-level analysis we re-score every candidate on validation, and we re-score the seed and best prompts on a 200-task held-out subset with the reward’s per-citation reasons enabled, which lets us decompose the reward and collect the material for the error analysis.

4.2 Results

Optimization trajectory.

Over 240 rollouts GEPA produced 7 candidates (Table 3), and validation reward climbs along the accepted frontier from 0.672 at the seed to 0.874 at the best candidate, discovered at rollout 156 (Figure 1). Both prompt components contributed, with the first gain coming from the researcher prompt and the two largest coming from the orchestrator prompt. The last two candidates grew the prompts further and both regressed on validation, so Pareto selection kept the shorter candidate 4. The rejection of the longer candidates 5 and 6 is in-run evidence for the bloat effect that production users mitigate with length regularization [Decagon, 2026], supplied here by the Pareto objective without an explicit length penalty.

Held-out generalization.

On the 500-task, 254-topic held-out test set the optimized prompt beats the seed by 0.079 with a tight bootstrap confidence interval and a highly significant Wilcoxon test (Table 4). Because this test set shares no topic with anything the optimizer saw, it is our headline held-out estimate. The gain is a net improvement rather than a clean sweep, with 287 tasks improving, 147 regressing, and 66 unchanged. The per-task reward distribution shifts tasks into the top bucket (Figure 3), the per-topic change is right-shifted with a positive mean (Figure 3), and the per-topic scatter is denser above the line of equality (Figure 4).

Reward decomposition.

The held-out decomposition on 200 topic-disjoint tasks (Table 5, Figure 5) shows the improvement is broad across the 3 citation-quality levers, each near 0.09 with confidence-interval-separated bars, rather than concentrated in one. Factual support is the largest gain and, at 0.53 rising to 0.62, still the lowest lever and the binding constraint, while the answer gate moves little and numbering is saturated. The 24-task validation set is too noisy to decompose reliably, so we report the decomposition on the 200-task held-out set.

4.3 Robustness across seeds

Repeating the optimization under 3 independent seeds shows the held-out gain is a property of the method, not of one lucky run (Table 6). Every seed produces a positive held-out gain whose bootstrap confidence interval excludes zero, with a mean of 0.107 and a range of 0.079 to 0.130. What is not stable is the validation-minus-held-out gap, which ranges from 0.034 to 0.160, a factor of about 5. A quantity that measured the memorized portion of the same selected prompt should not swing this much across seeds, so the spread is direct evidence that the gap reflects optimizer-selection and judge variance rather than a stable memorization signal.

4.4 Does the memorized answer key earn reward?

The topic-shared evaluation isolates the effect of the memorized answer key by scoring the fixed seed and best prompts on questions from the training families, with no further search (Table 7). On the 24 exact training questions, which is the best possible case for a memorized lookup, the best prompt beats the seed by only 0.090. That gain is statistically indistinguishable from the 0.079 held-out gain and nowhere near the 0.202 validation gain, and the answer gate stays flat at 0.92 for both prompts, so the memorized key does not even raise answer recall. If memorization drove the validation gain, this exact-leak number would approach 0.202, but it does not. On fresh same-topic questions the memorized families gain less (0.071) than the non-memorized families (0.154), so the answer-key blocks behave as prompt bloat that slightly depresses performance rather than as a useful lookup table. Several memorized entries do not even match the dataset’s gold answers, which reinforces that the key is not a reliable table of answers. A topic-shared split does inflate the apparent gain modestly (0.112 versus the 0.079 disjoint gain), but the inflation is the transfer of general citation discipline to same-topic questions, not memorized reward, and it remains far below the validation gain.

4.5 Which verifier property drives transfer?

This ablation varies the verifier while holding the data, policy, reflector, and budget fixed, and re-scores every optimized prompt on one canonical held-out verifier so the configurations are comparable (Table 8, Figure 6). It varies three properties of the reward signal that supervises the search, its resolution, whether scoring is repeated, and whether the feedback is decomposed per dimension, and asks which of them determines what transfers and where reward gaming appears. The canonical held-out baseline is 0.682 over 120 topic-disjoint tasks, and each configuration is 2 seeds, so we read the cell means as directional rather than precise (Section 6).

Decomposed feedback is what makes the gain transfer.

Replacing the per-dimension textual feedback with a single scalar reward drives held-out transfer negative, from +0.075 for the decomposed reference to −0.021 for the scalar variant. Per seed the scalar reward either found no accepted improvement at all or raised validation while regressing on held-out, so a reflector given only a scalar has nothing to diagnose and cannot discover transferable citation-discipline rules. The levers that carry the transferable gain in the main results, link_works, relevant_content, and fact_check, are exactly the per-dimension signals that the scalar variant discards.

Finer or repeated scoring buys stability, not transfer.

Along the reliability ordering of coarse, fine, and fine with 3x repetition, mean transfer is flat to declining at 0.079, 0.075, and 0.059, so more granular or more repeated judging did not improve generalization at a fixed budget, and repetition cost 3x the judge calls. Its one benefit was stability, since repeated scoring gave the tightest seed-to-seed spread at 0.018, against 0.037 for the reference and 0.079 for the coarse verifier.

Reward gaming tracks weak verifiers, and held-out testing exposes it.

The gaming gap, the difference between the validation gain and the held-out transfer, is largest for the low-information verifiers rather than the strongest one, at 0.071 for coarse and 0.061 for scalar on top of its negative transfer, against 0.043 for the fine decomposed reference (Figure 7). Stronger verification therefore did not induce more gaming in this setting, low-information verification did, and the held-out re-scoring is what reveals it, most starkly for the scalar cell where a positive validation delta hides a held-out regression.

4.6 A hand-written recipe baseline

The transferable rules the optimizer found are describable, which raises the question of whether a practitioner could write them by hand and skip the search, so we test that directly. We construct a recipe prompt equal to the best prompt with all memorized answer-key content removed, dropping the Known Benchmark Facts and Known Prior Research Hints blocks and the two answer-bearing citation examples, and keeping only the describable rules of answer-first structure, one claim per citation, verified page content, corpus-only URLs, and sequential numbering. We score this recipe against the seed and the best prompt on 100 held-out tasks under the same policy, reward, and corpus (Table 9).

The recipe alone recovers 0.047 of the 0.083 total held-out gain on these tasks, about 57%, so a practitioner who writes down the rules captures most of the improvement. The full optimized prompt adds a further 0.036, but this increment is not individually significant at 100 tasks (95% confidence interval [−0.021,+0.096]), consistent with the memorized answer key being inert (Section 4.4). The total seed-to-best gain of 0.083 is significant and matches the headline 0.079 on the full 500-task set. The improvement is therefore largely a transferable, describable recipe, and reflective search adds only a small residual that we cannot separate from noise at this sample size.

4.7 Cost and latency

Because prompt optimization changes only the prompt, the optimized agent runs at inference with no added machinery, and we confirm it carries no latency penalty. On 6 held-out queries per prompt at concurrency one, the optimized prompt averaged 138.8 seconds per query against 159.9 for the seed, and output length was unchanged at about 3200 tokens. The gain does cost input tokens, since mean total policy tokens rose about 32%, from 146.0k to 193.1k per query, almost entirely on the input side (an added 47k), because the optimized prompt retrieves and grounds more context per query. GEPA’s search is a one-time cost of 240 policy rollouts, on the order of 40M policy tokens, after which every query runs at the per-query cost above with no recurring optimizer overhead. These are small-sample point estimates over 6 queries per prompt, reported for scale rather than as a powered comparison.

4.8 Judge validation and self-preference

The reward is scored by language-model judges [Zheng et al., 2023], so we check that they agree with a human and are not inflated by a shared provider family [Panickssery et al., 2024, Pombal et al., 2026]. One author labeled 48 held-out (attribution, citation) pairs blind to the judge decision, and we re-scored the same pairs with an out-of-family judge, Anthropic Claude Opus 4.8 (Table 10). The in-family factual-support judge, which scores the binding reward lever, matched the human on 80.0% of pairs (Cohen’s κ=0.60; Cohen, 1960), and its disagreements are mildly lenient, accepting topically-close sources on 7 of its 9 mismatches. The relevance judge matched on 77.8% (κ=0.23), and all 10 of its disagreements are one-directional under-crediting, where it withholds relevance because the source does not also support the specific fact, so it never inflates the score. The out-of-family judge agrees with the human as well as or better than the in-family judge, on 91.1% of relevance and 80.0% of factual-support decisions, and where the two families differ it is the in-family judge that is the more lenient. Re-scoring the composite with the out-of-family judge on 12 held-out tasks, the seed-to-best gain persists and is if anything larger, 0.129 against 0.069 in-family on the same tasks, so the improvement is not an artifact of the judges’ provider family. These are spot-checks with 48 pairs and a single annotator, and they complement the larger judge-versus-human benchmark of Leung et al. [2026].

5 Discussion

The winning lineage rewrote the researcher prompt once and the orchestrator twice. Across these edits the reflector added topic-independent rules, each targeting a different citation lever. It moved the answer to the front of the report, split multi-claim sentences so that one claim carries one citation, required the agent to verify that the cited page’s main content supports the specific claim, and restricted citations to corpus URLs. These edits are what raise the 3 citation levers on held-out data.

The verifier ablation (Section 4.5) explains why the reflector could find these transferable rules at all. Each of them answers a specific diagnostic the decomposed reward surfaces, that a link failed to resolve, that a cited page was off-topic, or that a page did not support the claim, and the reflector rewrites the prompt to close the named failure. When the same reward is collapsed to a single scalar the reflector loses that diagnostic channel, and transfer turns negative, so the mechanism behind the gain is the per-dimension feedback rather than the optimizer’s search machinery on its own. This also reframes the reward signal as the design variable that matters most, since a practitioner who reports only a scalar would both learn less and be more exposed to the gaming that the held-out protocol exists to catch.

The reflector also made a second kind of edit. It wrote Known Benchmark Facts and Known Prior Research Hints sections that memorized specific training answers, for example a language count for one model, a product name for one company, and a policy date for one central bank. This is a textbook case of prompt-level overfitting [Fu et al., 2026], in which the optimizer copies specific training values into the prompt instead of learning a rule that generalizes, an attempted reward hack against the training signal [Skalse et al., 2022, Mahmoud et al., 2026]. This memorization is not confined to the candidates that Pareto selection rejected. It appears in the first accepted edit and grows to dominate the winning program, in which the memorized answer key is 52% of the orchestrator prompt and 39% of the researcher prompt by word count. Nearly half of the deployed program is therefore a table of verbatim training questions paired with their exact answers and source URLs (Table 11). The natural reading is that this key is what inflates the validation score, but the topic-shared ablation (Section 4.4) shows it is not. The memorized key adds only 0.090 even on the exact training questions it encodes, which is indistinguishable from the transferable held-out gain and far below the 0.202 validation gain, and it helps the memorized families less than the non-memorized ones. The memorized content is therefore attempted reward hacking that does not pay off, raising the prompt’s length far more than its reward, and the validation-minus-held-out gap is better explained by small-sample selection under a noisy judge, which we take up next.

Prompt bloat is a subtler symptom of the same pressure. Past candidate 4 the reflector kept adding emphatic and lengthy instruction, more capitalized CRITICAL directives and more memorized-fact dumps, growing the orchestrator prompt to 3261 words in candidate 6, yet both over-grown candidates regressed on validation (Table 3). The extra instruction inflated length without improving behaviour, and GEPA’s Pareto selection rejected it, a useful partial guard against reward hacking by bloat, but only a validation-time one. It is the topic-disjoint held-out test, not the Pareto objective, that ultimately separates the transferable recipe from the memorized and over-instructed content.

5.1 Why the topic-disjoint protocol is necessary

The protocol has three parts, and each catches a failure the others miss. The topic-disjoint split is what makes a held-out score mean transfer, because no article family is shared across train, validation, and test, so a memorized training answer cannot inflate the held-out number. Held-out evaluation then converts the apparent validation gain into a transfer estimate, and the gap between the two measures the part that does not transfer. A practitioner who selected and reported on validation alone would have described a gain of 0.202 when the transferable gain is 0.079, an overstatement of about 2.6 times.

The gap is not, however, a stable measurement of memorized reward, and two results establish this. Across the candidates the memorized answer-key fraction and the validation reward rise together (Pearson r=0.83), from 0% at the seed to 48% at the selected best (Table 11, Table 3), which invites the reading that the optimizer is buying validation reward with memorization. That reading mistakes correlation for causation, because memorization and the transferable citation-discipline edits were added along the same accepted lineage, so the two rise together. The topic-shared ablation (Section 4.4) nonetheless shows the memorized content earns no reward even on the questions it encodes. What remains true is that a validation-only workflow surfaces the most-memorized program as its best, so the practitioner not only mis-sizes the gain but also ships the most overfitted prompt, even though that overfitting is chasing judge noise rather than memorized reward.

The prompt autopsy supplies qualitative ground truth for the overfitting itself. Reading the winning prompt confirms that its Known Benchmark Facts sections pair specific training questions with their exact answers, for example a fixed language count for one model and a fixed policy date for one central bank. The overfitting is therefore literal memorization rather than benign paraphrase. The autopsy establishes that the memorized content is real and large, and the ablation establishes that it is non-functional. Together they separate a genuine prompt-hygiene problem from the size of the transfer gap.

The topic-shared counterfactual is no longer only a prediction. The ablation in Section 4.4 constructs same-family questions and scores the fixed prompts on them, and it does not collapse the gap toward the validation number. The apparent gain rises only to 0.112, well short of the 0.202 validation gain, and the memorized families gain less than the non-memorized ones. Because the gap is dominated by small-sample selection under a noisy judge, we expect it to be largest not where memorization is heaviest but where the validation set is smallest and the judge noisiest. We separately expect memorization pressure to be largest in narrow, stationary domains where the same entities recur (Section 6).

5.2 Error analysis

After optimization, factual support is still the lowest lever at 0.62 on the held-out subset, and 59 of 200 tasks score below 0.5 on it. Table 12 categorizes the weak citations by the judge’s stated reason. The dominant theme is that the cited source does not support the specific claim, consistent with factual support being the binding lever. The corpus-only-URL rule largely worked, since not-in-corpus URLs are nearly gone.

The regressions are mostly answer-gate flips, where the optimized prompt is more assertive because it puts the answer first, which usually helps but sometimes commits to a wrong specific answer where the seed hedged. On the paired 200-task subset the gate gained on 12 tasks and lost on 6. The gate losses are the largest single-task regressions, where a task can drop from near 0.9 to 0, but they are outnumbered by the gate gains, and the mean factual-support change on the subset is positive at 0.095.

6 Limitations and Future Work

The reward is a language-model judge, validated only at small scale here.

Every number in this paper is measured by the composite reward, which is itself scored by language-model judges. A spot-check of 48 held-out pairs (Section 4.8) shows the judges agree with a human, most strongly on the binding factual-support lever (κ=0.60), and the per-dimension judge models were validated more broadly against human labels in a companion benchmark [Leung et al., 2026]. Our own check is single-annotator and small, so a larger multi-annotator study on the optimized outputs would strengthen it, but the reported gain is a gain against judges of demonstrated human agreement rather than an unaudited proxy.

Self-preference and same-family bias.

The policy, the reflector, and the citation judges are all from the same provider family, which risks the reflector learning to please the judge’s scoring idiosyncrasies and same-family agreement inflating the measured reward [Panickssery et al., 2024, Pombal et al., 2026]. We check this directly in Section 4.8, where an out-of-family judge agrees with the human as well as or better than the in-family judge and the seed-to-best gain persists under it, so the improvement is not an artifact of the shared family. The check uses one alternative model on a small sample, so a larger multi-provider judge panel would strengthen it, though the current evidence points against a self-preference confound.

Constraining the optimizer against memorization.

Our protocol detects prompt-level overfitting but does not prevent it. The reflector is free to write anything into the prompt, including the memorized answer key of Section 5, and the topic-disjoint split only discounts that content at evaluation rather than stopping it at the source. Here the memorized key turned out not to improve the reward, so constraining it is a matter of prompt hygiene and auditability rather than recovering lost accuracy, but in a domain where such content did transfer the same edit would be an active hazard. A complementary direction is to constrain the update step itself [Fu et al., 2026]. SkillOpt [Yang et al., 2026c] enforces generalizability inside the optimizer prompt, instructing the editor that edits must be generalizable and must not hardcode task-specific values, filtering example-specific proposals before they are applied, and accepting only bounded edits that pass a held-out validation gate. A generalization-constrained, validation-gated update rule of this kind could suppress memorized-answer-key edits during optimization instead of leaving them to be caught after the fact, and pairing such a rule with our protocol is a natural next step.

Small-sample validation and the winner’s curse.

The optimizer selects its best candidate on 24 validation tasks scored by a stochastic judge, so the reported validation reward of the winner is subject to winner’s-curse inflation. Selecting the maximum over several noisy candidates favours the one that caught the most upward noise, a regressional-Goodhart effect [Manheim and Garrabrant, 2018, Gao et al., 2023]. This is the most likely source of the validation-minus-held-out gap, consistent with the gap’s instability across seeds (0.034 to 0.160, Section 4.3) and with the ablation showing the memorized content earns no reward (Section 4.4). A larger validation set, an explicit selection penalty, or repeated scoring to average out judge noise would shrink the gap, but we did not run these and leave them for future work.

Reward engineering as a complementary guard against hacking.

A complementary intervention point is the reward function itself. Verbatim answer injection is detectable at scoring time: a prompt-content novelty penalty, subtracting a term proportional to token overlap between the candidate prompt and training questions, would directly penalize this strategy in-run rather than catching it post-hoc via the held-out split. In addition, a judge could be prompted to identify leakage. More broadly, reward ensembles have shown promise in improving robustness against reward hacking [Coste et al., 2024], and the same principle applies to prompt optimization: scoring candidates against multiple independent reward heads makes it harder to inflate any single proxy. Designing more robust and tamper-resistant reward functions and detecting misalignment via behavioral or distributional anomaly detection [Yang et al., 2026b] are both active directions for mitigating reward hacking, and adapting these techniques to the prompt-optimization setting, where the policy becomes the prompt instead of model weights, is a natural extension of the present work [Wang et al., 2026a].

Three seeds, not many.

The held-out gain replicates across 3 independent seeds (Section 4.3), so its direction and significance are robust rather than a single lucky run. What varies is the magnitude of the gain (0.079 to 0.130) and which candidate the optimizer selects, and 3 seeds is too few to characterize that spread precisely. We therefore do not treat the validation-minus-held-out gap as a stable estimate of anything. It is a diagnostic that flags overfitting, not a calibrated measurement.

The verifier ablation is directional.

The verifier ablation (Section 4.5) runs each of the 4 configurations at only 2 seeds, and the seed-to-seed spreads are wide. We therefore read the cell means as directional rather than as tight estimates and do not lean on the exact ordering among the decomposed cells. The robust readings are the two that survive the spread. Scalar feedback collapses transfer, and there is no reliability-to-transfer trend. All cells are re-scored by a single canonical verifier from one provider family on a 120-task held-out subset, so the transfer numbers inherit that judge’s biases, and confirming the mechanism with an out-of-family verifier and more seeds is the natural follow-up.

Reward under-specification and corpus binding.

The reward weights, the tiered answer-gate definition, and the judge prompts are released with the artifact but should also be stated in the paper. The search tool queries a frozen corpus, so the link-accessibility and URL-discipline conclusions are relative to that corpus.

A single agent, task, and domain.

We study one multi-agent deep-research agent on one citation task drawn from a single news corpus, so the size of the leakage and the exact transferable recipe are specific to this setting. Whether reflective optimization leaks as heavily on other tasks, agents, and domains is open, and we expect the memorization pressure to be larger, not smaller, in narrow domains. A domain with a small and recurring entity space, a fixed document template, or a reward that keys on a few specific facts gives the optimizer more opportunity to bake answers into the prompt and more reward for doing so. Replicating the protocol on such a domain, with splits that are disjoint by entity or document source rather than by news topic, is the natural test of external validity.

Held-out monitoring for deployed prompt optimization.

In this setting the gap was dominated by selection noise, but the same instrumentation is what would catch the more dangerous case in which memorized content does transfer. Recomputing the held-out gain on a schedule against a refreshed, entity-disjoint set, and pairing it with a periodic prompt autopsy, gives two complementary signals for a deployed prompt-optimized system. The held-out score tracks whether the gain still generalizes, and the autopsy flags when the optimizer has started baking specific entities into the prompt. In a stationary domain a baked-in answer keeps scoring well until the entity it names changes, at which point the held-out score falls while the in-distribution score does not. Tracking these signals over time, rather than treating any single validation-minus-held-out gap as a calibrated leakage measurement, is a concrete direction for keeping reflective optimization auditable in deployment.

7 Conclusion

Reflective prompt optimization delivered a statistically significant, generalizing improvement to a deep-research agent’s citation reward. The held-out gain was 0.079 on 500 topic-disjoint tasks with no weight updates, it replicated across 3 seeds at a mean of 0.107, and it spread across the citation-quality levers rather than concentrating in one. Factual support, the hardest lever, carried the largest share. The methodological contribution is the protocol rather than the optimizer. A topic-disjoint held-out test, a controlled topic-shared ablation, multi-seed replication, and a verifier ablation together let us ask whether the large validation-minus-held-out gap is memorization, and the answer is that it is not. The optimizer does write a literal memorized answer key, nearly half of the best prompt by word count, but that content earns no reward even on the exact training questions it encodes. The validation over-statement is therefore small-sample selection under a noisy judge, not memorized reward. Varying the verifier shows what does drive the gain. Decomposed per-dimension feedback is the causal ingredient, since a scalar reward drives held-out transfer negative while finer or repeated scoring does not raise it. The recipe the search discovered is a portable prompting pattern for citation-faithful research agents, namely an answer-first structure, one claim per citation, verified page content, and corpus-only URLs. A hand-written version of this recipe recovers most of the held-out gain, which confirms the improvement is a transferable rule set rather than a property of the search itself. It complements rather than replaces reinforcement learning, and it generalizes only when the reward is decomposed and the evaluation is held-out and multi-seed.

References

• Agarwal et al. (2024) E. Agarwal, J. Singh, V. Dani, R. Magazine, T. Ganu, and A. Nambi PromptWizard: task-aware prompt optimization framework. Note: https://arxiv.org/abs/2405.18369 Cited by: §2.
• Agrawal et al. (2025) L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab GEPA: reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457. Note: ICLR 2026 oral. Code: https://github.com/gepa-ai/gepa External Links: Link Cited by: §1, §1, §2.
• Amodei et al. (2016) D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. External Links: Link Cited by: §1.
• Bai and Shi (2026) J. Bai and L. Shi MAS-PromptBench: when does prompt optimization improve multi-agent LLM systems?. arXiv preprint arXiv:2606.23664. External Links: Link Cited by: §2.
• Chen et al. (2025) M. Chen, L. Sun, T. Li, H. Sun, Y. Zhou, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, F. Yang, Z. Zhou, and W. Chen ReSearch: learning to reason with search for LLMs via reinforcement learning. Note: https://arxiv.org/abs/2503.19470 Cited by: §2.
• Cohen (1960) J. Cohen A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1), pp. 37–46. Cited by: §4.8.
• Coste et al. (2024) T. Coste, U. Anwar, R. Kirk, and D. Krueger Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations (ICLR), External Links: Link Cited by: §6.
• Decagon (2026) Decagon Optimizing GEPA for production: a test-driven approach to prompt engineering. Note: https://decagon.ai/blog/optimizing-gepa-for-production Cited by: §1, §3.4, §3.6, §4.2.
• Efron (1979) B. Efron Bootstrap methods: another look at the jackknife. The Annals of Statistics 7 (1), pp. 1–26. Cited by: §4.1.
• Fernando et al. (2024) C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rocktäschel Promptbreeder: self-referential self-improvement via prompt evolution. In Proceedings of the 41st International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.
• Fu et al. (2026) L. Fu, Y. Yu, Y. Wang, Y. Jin, H. Jin, B. A. Prakash, and H. Wang TextReg: mitigating prompt distributional overfitting via regularized text-space optimization. arXiv preprint arXiv:2605.21318. External Links: Link Cited by: §5, §6.
• Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning (ICML), External Links: Link Cited by: §6.
• GEPA Team (2026) GEPA Team Scaling GEPA with combee: parallel prompt learning for self-improving agents. Note: https://gepa-ai.github.io/gepa/blog/2026/04/09/gepa-at-scale-with-combee/ Cited by: §1.
• GEPA-AI (2025) GEPA-AI GEPA: optimize prompts, code, and more with AI-powered reflective text evolution. Note: https://github.com/gepa-ai/gepaSoftware repository Cited by: §2, §3.3.
• Guo et al. (2024) Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. Bian, and Y. Yang Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.
• Java et al. (2025) A. Java, S. Koundinyan, N. Natarajan, and A. Sharma FrugalRAG: less is more in RL finetuning for multi-hop question answering. Note: https://arxiv.org/abs/2507.07634 Cited by: §2.
• Jiang et al. (2020) Y. Jiang, S. Bordia, Z. Zhong, C. Dognin, M. Singh, and M. Bansal HoVer: a dataset for many-hop fact extraction and claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3441–3460. Cited by: §2.
• Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han Search-r1: training LLMs to reason and leverage search engines with reinforcement learning. Note: https://arxiv.org/abs/2503.09516 Cited by: §2.
• Khattab et al. (2022) O. Khattab, K. Santhanam, X. L. Li, D. Hall, P. Liang, C. Potts, and M. Zaharia Demonstrate-search-predict: composing retrieval and language models for knowledge-intensive NLP. arXiv preprint arXiv:2212.14024. External Links: Link Cited by: §2.
• Khattab et al. (2024) O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts DSPy: compiling declarative language model calls into self-improving pipelines. In The Twelfth International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1, §2.
• Leung et al. (2026) E. Leung, E. Lumer, C. Feld, A. Huber, V. K. Subbiah, and K. Paul Do you need a frontier model as a citation verifier? benchmarking rubric LLMs for deep-research source attribution. External Links: 2607.08700, Link Cited by: Appendix C, §2, §3.6, §4.8, §6.
• Lumer et al. (2025) E. Lumer, A. Cardenas, M. Melich, M. Mason, S. Dieter, V. K. Subbiah, P. H. Basavaraju, and R. Hernandez Comparison of text-based and image-based retrieval in multimodal retrieval augmented generation large language model systems. arXiv preprint arXiv:2511.16654. External Links: Link Cited by: §1, §2.
• Lumer et al. (2026) E. Lumer, S. Sen, K. Paul, and V. K. Subbiah Recursive agent harnesses. arXiv preprint arXiv:2606.13643. External Links: Link Cited by: §1, §2, §3.1.
• Mahmoud et al. (2026) A. Mahmoud, M. Rezaei, Z. Wang, A. Gunjal, B. Liu, and Y. He Reward hacking in rubric-based reinforcement learning. arXiv preprint arXiv:2605.12474. External Links: 2605.12474 Cited by: §1, §2, §5.
• Manheim and Garrabrant (2018) D. Manheim and S. Garrabrant Categorizing variants of Goodhart’s law. arXiv preprint arXiv:1803.04585. External Links: Link Cited by: §6.
• Onweller et al. (2026) H. Onweller, E. Lumer, A. Huber, P. Ramchandani, V. K. Subbiah, and C. Feld Cited but not verified: parsing and evaluating source attribution in LLM deep research agents. arXiv preprint arXiv:2605.06635. External Links: Link Cited by: §1, §2, §3.2.
• Opsahl-Ong et al. (2024) K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §1, §2.
• Panickssery et al. (2024) A. Panickssery, S. R. Bowman, and S. Feng LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §4.8, §6.
• Peng et al. (2025) H. Peng, Y. Qi, X. Wang, B. Xu, L. Hou, and J. Li VerIF: verification engineering for reinforcement learning in instruction following. Note: https://arxiv.org/abs/2506.09942 Cited by: §2.
• Pombal et al. (2026) J. Pombal, R. Rei, and A. F. T. Martins Self-preference bias in rubric-based evaluation of large language models. arXiv preprint arXiv:2604.06996. External Links: Link Cited by: §4.8, §6.
• Pyatkin et al. (2025) V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi Generalizing verifiable instruction following. Note: https://arxiv.org/abs/2507.02833 Cited by: §2.
• Rezaei et al. (2026) M. Rezaei, A. Mahmoud, Z. Wang, U. Tyagi, A. Gosai, R. Dumitru, A. Sabharwal, B. Liu, and Y. He Rubric-guided self-distillation: post-training without rubric verifiers. arXiv preprint arXiv:2606.12507. External Links: 2606.12507 Cited by: §2.
• Sen et al. (2026a) S. Sen, A. Kasturi, E. Lumer, A. Gulati, and V. K. Subbiah Is grep all you need? how agent harnesses reshape agentic search. arXiv preprint arXiv:2605.15184. External Links: Link Cited by: §1, §2.
• Sen et al. (2026b) S. Sen, E. Lumer, A. Gulati, and V. K. Subbiah Chronos: temporal-aware conversational agents with structured event retrieval for long-term memory. arXiv preprint arXiv:2603.16862. External Links: Link Cited by: §1, §2.
• Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: Link Cited by: §1, §2.
• Shi et al. (2025) Z. Shi, Y. Chen, H. Li, W. Sun, S. Ni, Y. Lyu, R. Fan, B. Jin, Y. Weng, M. Zhu, Q. Xie, X. Guo, Q. Yang, J. Wu, J. Zhao, X. Tang, X. Ma, C. Wang, J. Mao, Q. Ai, J. Huang, W. Wang, Y. Zhang, Y. Yang, Z. Tu, and Z. Ren Deep research: a systematic survey. arXiv preprint arXiv:2512.02038. External Links: Link Cited by: §1.
• Si et al. (2025) S. Si, H. Zhao, C. Gao, Y. Bai, Z. Wang, B. Gao, K. Luo, W. Li, Y. Huang, G. Chen, F. Qi, M. Zhang, B. Chang, and M. Sun Teaching large language models to maintain contextual faithfulness via synthetic tasks and reinforcement learning. Note: https://arxiv.org/abs/2505.16483 Cited by: §2.
• Skalse et al. (2022) J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §1, §5.
• Song et al. (2025) H. Song, J. Jiang, Y. Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J. Wen R1-searcher: incentivizing the search capability in LLMs via reinforcement learning. Note: https://arxiv.org/abs/2503.05592 Cited by: §2.
• Soylu et al. (2024) D. Soylu, C. Potts, and O. Khattab Fine-tuning and prompt optimization: two great steps that work better together. Note: https://arxiv.org/abs/2407.10930 Cited by: §2.
• Sun et al. (2025) Z. Sun, Q. Wang, H. Wang, X. Zhang, and J. Xu Detection and mitigation of hallucination in large reasoning models: a mechanistic perspective. Note: https://arxiv.org/abs/2505.12886 Cited by: §2.
• Suzgun et al. (2025) M. Suzgun, M. Yuksekgonul, F. Bianchi, D. Jurafsky, and J. Zou Dynamic cheatsheet: test-time learning with adaptive memory. Note: https://arxiv.org/abs/2504.07952 Cited by: §2.
• Tan et al. (2025) S. Tan, L. A. Agrawal, A. Singhvi, L. Lai, M. J. Ryan, D. Klein, O. Khattab, K. Sen, and M. Zaharia LangProBe: a language programs benchmark. Note: https://arxiv.org/abs/2502.20315 Cited by: §2.
• Tyagi et al. (2026) U. Tyagi, X. Guo, M. Rezaei, D. George, A. Mahmoud, J. Lee, B. Liu, and Y. He Not every rubric teaches equally: policy-aware rubric rewards for RLVR. arXiv preprint arXiv:2605.20164. External Links: 2605.20164 Cited by: §1, §2.
• Wan et al. (2024) X. Wan, R. Sun, H. Nakhost, and S. Arik Teach better or show smarter? on instructions and exemplars in automatic prompt optimization. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 37. Cited by: §2.
• Wang et al. (2026a) X. Wang, M. Tian, Y. Zeng, Z. Huang, J. Yuan, B. Chen, J. Xu, M. Zhou, W. Liu, M. Wu, Z. Guo, Q. Qian, Y. Wang, F. Zhang, R. Yin, S. Dou, C. Lv, T. Chen, K. Song, X. Tan, T. Gui, X. Zheng, and X. Huang Reward hacking in the era of large models: mechanisms, emergent misalignment, challenges. arXiv preprint arXiv:2604.13602. External Links: Link Cited by: §6.
• Wang et al. (2026b) X. Wang, Z. Hao, S. Hou, H. Peng, J. Li, and X. Wang Reproducing, analyzing, and detecting reward hacking in rubric-based reinforcement learning. arXiv preprint arXiv:2606.04923. External Links: Link Cited by: §1, §2.
• Wang et al. (2024) Z. Z. Wang, J. Mao, D. Fried, and G. Neubig Agent workflow memory. Note: https://arxiv.org/abs/2409.07429 Cited by: §2.
• Wei et al. (2023) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. Note: NeurIPS 2022https://arxiv.org/abs/2201.11903 Cited by: §2.
• Wilcoxon (1945) F. Wilcoxon Individual comparisons by ranking methods. Biometrics Bulletin 1 (6), pp. 80–83. Cited by: §4.1.
• Wu et al. (2025a) P. Wu, M. Zhang, X. Zhang, X. Du, and Z. Z. Chen Search wisely: mitigating sub-optimal agentic searches by reducing uncertainty. Note: https://arxiv.org/abs/2505.17281 Cited by: §2.
• Wu et al. (2025b) S. Wu, P. Sarthi, S. Zhao, A. Lee, H. Shandilya, A. M. Grobelnik, N. Choudhary, E. Huang, K. Subbian, L. Zhang, D. Yang, J. Zou, and J. Leskovec Optimas: optimizing compound AI systems with globally aligned local rewards. Note: https://arxiv.org/abs/2507.03041 Cited by: §2.
• Xu et al. (2025) W. Xu, A. Nie, R. Zheng, A. Modi, A. Swaminathan, and C. Cheng Formalizing learning from language feedback with provable guarantees. Note: https://arxiv.org/abs/2506.10341 Cited by: §2.
• Xu et al. (2026) Y. Xu, S. Lu, J. Cheng, M. Wang, Q. Xie, X. Wang, R. He, and J. Liang How to train your deep research agent? prompt, reward, and policy optimization in Search-R1. arXiv preprint arXiv:2602.19526. External Links: Link Cited by: §2.
• Yang et al. (2024) C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen Large language models as optimizers. arXiv preprint arXiv:2309.03409. External Links: Link Cited by: §2.
• Yang et al. (2026a) M. Yang, X. Guo, U. Tyagi, M. Zhang, R. Dumitru, S. Hou, Y. He, D. Y. Zhang, and Y. Liu Rubric dropout: a simple way to mitigate reward hacking in rubric-as-reward RL. arXiv preprint arXiv:2608.11669. External Links: Link Cited by: §2.
• Yang et al. (2026b) R. Yang, Y. Liu, S. Luo, H. Jia, C. Zhang, H. Zheng, Y. Yang, Y. Huang, J. Zhang, Y. Qi, K. Xu, C. Ran, and B. Chong Harness-agnostic detection and immunization of reward hacking in self-evolving language models. arXiv preprint arXiv:2609.04665. External Links: Link Cited by: §6.
• Yang et al. (2026c) Y. Yang, Z. Gong, W. Huang, Q. Yang, Z. Zhou, Z. Huang, Y. Li, X. Gao, Q. Dai, B. Liu, K. Qiu, Y. Yang, D. Chen, X. Yang, and C. Luo SkillOpt: executive strategy for self-evolving agent skills. arXiv preprint arXiv:2605.23904. External Links: 2605.23904 Cited by: §2, §6.
• Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.
• Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. Note: ICLR 2023https://arxiv.org/abs/2210.03629 Cited by: §2.
• Yuksekgonul et al. (2025) M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, P. Lu, Z. Huang, C. Guestrin, and J. Zou Optimizing generative AI by backpropagating language model feedback. Nature 639, pp. 609–616. Cited by: §2.
• Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, External Links: Link Cited by: §4.8.
• Zhou et al. (2023) Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.
• Ziems et al. (2025) N. Ziems, D. Soylu, L. A. Agrawal, I. Miller, L. Lai, C. Qian, K. Song, M. Jiang, D. Klein, M. Zaharia, K. D’Oosterlinck, C. Potts, and O. Khattab Multi-module GRPO: composing policy gradients and prompt optimization for language model programs. arXiv preprint arXiv:2508.04660. External Links: Link Cited by: §2.

Appendix A Reproducibility

The analysis runs as a pipeline over the released artifact. The optimization step runs the agent with the seed prompts on the train and validation splits and searches with GEPA under the configuration in Section 3.3. The analysis step re-scores every candidate on validation and re-scores the seed and best prompts on the 200-task held-out subset with per-citation reasons enabled. The reporting step derives the prompt lineage and diffs, the paired held-out statistics, the figures, the tables, and the error analysis. The commands below are illustrative and use the released artifact layout.

# 1. Optimize (the run analysed here)
optimize --dataset dataset.jsonl --index corpus.db --out run/ \
--components both --train-size 24 --val-size 24 --test-size 500 \
--max-metric-calls 240 --reflection-effort high
# 2. Re-analysis (trajectory, decomposition, error material)
analyze --run-dir run/ --dataset dataset.jsonl --index corpus.db \
--out data/ --test-decomp-size 200
# 3. Lineage and diffs, paired stats, figures, tables, error analysis
report --run-dir run/ --data-dir data/ --out .

Appendix B Artifacts

We release the seed and all candidate prompts, their pairwise lineage diffs, the machine-readable lineage in data/prompt_lineage.json, the per-lever analysis data, the figures, and the tables, with the prompt-evolution and error-analysis write-ups in analysis/ (paths are given in the README).

Appendix C Reward weights, answer gate, and judge prompts

This appendix specifies the gated composite reward of Equation (1) exactly as used in the run analysed here and in every re-evaluation.

Composite weights and aggregation.

The four weights are wlink=0.25, wrel=0.30, wfact=0.30, and wnum=0.15, which sum to 1.00. Each of link_works, relevant_content, and fact_check is the unweighted mean over every parsed (attribution, citation) pair in the report, and a report with no parseable citation pair scores 0 on all three. Two levers are deterministic rather than model-scored. link_works is 1 when the cited URL resolves to content in the frozen corpus, and citation_numbering rewards inline [n](url) citations numbered 1..N contiguously with a matching Sources list, penalizing gaps, duplicates, and mismatches. All judges see frozen-corpus content only, with no live web fetch at reward time.

Answer gate.

The gate is a stateless, cheapest-first cascade returning {0,1}. Tier 1 is a deterministic number-canonicalized substring match of the gold answer in the report (thousands separators stripped, unit forms such as “41 billion”, “41bn”, and “41B” unified, quotes and dashes normalized, whitespace collapsed, lowercased); a multi-part gold answer joined by ; or | passes only if all parts are present. Tier 2 is a deterministic coverage router on the fraction of the gold answer’s distinctive tokens present in the report, with a pass threshold of 1.0 and a fail threshold of 0.5. Coverage at or above 1.0 passes, coverage below 0.5 fails with no language-model call, and the near-miss band in between escalates to Tier 3. Tier 3 fires only in that near-miss band and uses OpenAI GPT-5-mini (gpt-5-mini-2025-08-07, max 256 tokens, report truncated to 6000 characters), reasoning before emitting a final VERDICT: yes or VERDICT: no line. Its verbatim prompt is:

You judge whether a research REPORT answers the QUESTION with the same fact as the GOLD ANSWER. Paraphrase, restatement, and rounding-free equivalents count as the same fact; a different or absent fact does not.
First reason in 1-3 short sentences about whether the report expresses the gold fact. Then, on a FINAL separate line, output exactly ’VERDICT: yes’ or ’VERDICT: no’.
QUESTION: {question}
GOLD ANSWER: {truth}
REPORT: {report[:6000]}

Citation judges.

The source-relevance and factual-support levers are binary (0/1) language-model judges that return a structured {score, reason} object and see the cited URL’s corpus content truncated to 5000 characters. Source relevance uses OpenAI GPT-5-mini (gpt-5-mini-2025-08-07) and factual support uses OpenAI GPT-5.4-mini (gpt-5.4-mini-2026-03-17), following the per-dimension picks of Leung et al. [2026]. The verbatim system prompts are:

Source relevance (OpenAI GPT-5-mini). You are an expert evaluator assessing content relevance. Your task is to determine if the content at a URL is relevant to a given attribution text. Consider whether the URL content discusses the same topic as the attribution, provides supporting information for its claims, and has a clear connection to it. This is a binary evaluation: score = 1 if the content IS relevant, score = 0 if it is NOT.
Factual support (OpenAI GPT-5.4-mini). You are an expert fact-checker evaluating factual accuracy. Your task is to determine if the factual claims in an attribution text are supported by the content at a URL. Consider whether the specific facts, numbers, dates, and claims in the attribution are present in the URL content, whether they match or are consistent, and whether there are any contradictions. This is a binary evaluation: score = 1 if the facts ARE supported, score = 0 if they are NOT (contradicted, missing, or unverifiable).

Each judge is called with the attribution text, the cited URL, and the truncated URL content, and is asked to evaluate relevance or factual support for that pair.

注意转载授权 · 阅读原文