Meta 等机构提出 AutoBenchmark 与 HackBench，研究自主科研代理与人类协同

   Chuanyang Jin    Seungone Kim    Tong Chen    John Hoffman Affiliation: FAIR at Meta   
Yoram Bachrach Affiliation: FAIR at Meta    Pedro Rodriguez Affiliation: FAIR at Meta    Ping Yu Affiliation: FAIR at Meta    Eryk Helenowski Affiliation: FAIR at Meta    Swarnadeep Saha Affiliation: FAIR at Meta   
Jack Lanchantin    Weizhe Yuan    Ilia Kulikov    Jason Weston Affiliation: Johns Hopkins University Affiliation: Carnegie Mellon University Affiliation: University of Washington

Abstract

Autoresearch agents can now attempt long-horizon AI research end-to-end, i.e., self-improvement. However, best results are still obtained by research agents interacting with human researchers, and improving AI’s ability to perform this co-research will only reinforce this effect—a process termed co-improvement (Weston and Foerster, 2025). Further, aligning with humans is likely a safer path for research agents, especially given recent agent reward hacking phenomena (OpenAI, 2026b). In this work we address these issues in four ways: (i) we empirically study the role of humans in directing autoresearch agents, focusing on the domain of benchmark creation, and find humans still play a pivotal role particularly in the ideation phase; (ii) we develop a benchmark, HackBench, that shows current autoresearch agents are not aligned and are unable to achieve hack localization sufficiently; (iii) we present a method to train agents for improved human interaction by using human simulators as a proxy during reinforcement learning (RL); and finally (iv) we develop a method to improve model text generation, and hence agent communication, by RL with rubrics aligned with human experts. Together, our training and evaluation recipes provide a path toward more capable and aligned (co-)research agents.

^††date: October 28, 2026

1 The role of humans in directing research agents for benchmark creation

1.1 Overview

As autoresearch agents and recursive self-improvement (RSI) draw increasing attention, determining how models should direct their own improvement is becoming more important (Yin et al., 2025; Anthropic, 2026; Weng, 2026). While strong evidence shows that some verifiable tasks can be successfully automated (e.g., tasks that require hill-climbing a pre-defined metric (Wijk et al., 2025; Chan et al., 2025; Karpathy, 2026)), the scope and necessity of human contribution in open-ended tasks remain open questions. We focus on one such area, benchmark creation. Specifically, curating problems that are both difficult and meaningful to solve is a crucial task in AI model development (Reuel et al., 2024; Bean et al., 2026; Singh et al., 2026). From an evaluation perspective, it allows researchers to analyze the capabilities that models currently lack; from a development perspective, it supplies new targets to hill-climb on (Liang et al., 2022; Chang et al., 2024).

To date, benchmark creation is driven mainly by human AI scientists, and the process is fairly laborious. One first decides what the benchmark is meant to measure or assess (i.e., the construct), then identifies the grounding material used to instantiate it (e.g., GitHub PRs for SWE-Bench (Jimenez et al., 2024)), then designs a pipeline through which problems are produced and model predictions are evaluated, and finally runs the evaluation, iterating over these steps. The process further branches into granular decisions across an effectively unbounded space of possibilities: errors in the pipeline code must be fixed, strategies for increasing difficulty must be devised when problems prove insufficiently hard, and humans must be brought into the loop wherever verification is needed. In short, there is no unified recipe for building a high-quality benchmark, and the construction process varies with the type of benchmark and with the researcher who carries it out.

It is precisely this open-endedness that makes benchmark creation a demanding test for an autoresearch agent. In particular, we focus on autoresearch agents that create benchmarks for autoresearch agents: high-quality autoresearch agent benchmarks remain sparse relative to question-answering or competition mathematics benchmarks, so constructing a good one is a meaningful goal in its own right. Moreover, because autoresearch agent baselines must be run over long horizons, building such problems together with their surrounding environments renders benchmark creation considerably harder, and correspondingly more interesting.

To this end, we introduce AutoBenchmark, a framework in which an autoresearch agent builds a benchmark from a task specification and revises it across iterations using two feedback signals: (1) the trajectories and scores of a set of solvers that attempt the benchmark, and (2) the critique of an LLM verifier that checks the quality of the benchmark. Additionally, as a third axis, we investigate if human AI scientists can contribute to this process by directing what to build at the start of the loop and advising on revisions between iterations.

Our experiments show that current autoresearch agents can run this loop end to end, but that the benchmarks they produce unaided (i.e., without human feedback) are close to saturated, allowing the solvers to achieve scores above 80. We also find that human feedback helps substantially, and the gain grows with how specific that feedback is. Specifically, providing detailed direction on what type of benchmark to build and how to build it halves the score of the solvers, while a brief statement of what to build helps only marginally. This indicates that human involvement is still required in benchmark creation. Looking ahead, we envision AutoBenchmark as a way to measure progress on the capability of AI agents to autonomously create or co-create high-quality benchmarks.

1.2 AutoBenchmark

Figure 1shows the overall design of AutoBenchmark, in which a research agent is asked to carry out the benchmark creation process end-to-end. The research agent produces a benchmark in a Harbor-compatible evaluation package (Harbor Framework Team, 2026): a container environment, task instructions, the evidence supplied to the solver, reference solution, and machine-checkable grading criteria. The AutoBenchmark pipeline consists of the following stages (further details are in Appendix A):

Stage 1: Benchmark Proposal. The goal in this stage is to create difficult yet high-quality tasks. The research agent receives a task spec (a file that contains a general set of rules and, when a human provides feedback at the proposal stage, that direction as well; in the fully autonomous setting the file carries the rules alone) and, optionally, grounding material, and decides what capability the benchmark should measure before deciding how to measure it. It writes down the construct, develops several materially different ways to operationalize it, searches for and stores the primary sources it relies on, and then instantiates the selected design as a set of runnable tasks. Because the package has to execute, the agent also owns the decisions a human benchmark designer normally owns: what the solver sees, what evidence it is given, what an ideal answer contains, how partial credit is assigned, and a reference solution that shows the task can be solved at all.

Stage 2: Benchmark Solving. Each task is attempted by one or more solver agents inside the environment the research agent specified. Every attempt produces a trajectory and a final answer, and an answer judge grades that answer against the criteria the research agent wrote. These scores give the loop its difficulty signal, since a benchmark that the solvers already saturate tells us little about the frontier.

Stage 3: Benchmark Review. A low solver score on its own is weak evidence, because a broken task, an unsatisfiable requirement, a mis-specified grader, and fabricated grounding all produce the same number. An agent that optimizes solver failure alone also has an easy degenerate strategy available to it, which is to make the tasks impossible. We therefore review the benchmark itself, using an LLM judge that is applied to the research agent rather than to the solvers, along five criteria: construct validity, correctness, feasibility, usefulness, and an overall verdict. The resulting verdicts decide whether an iteration is eligible to be built on.

Below, in addition to the overall pipeline specified above, we list the components within each stage:

Component 1: Autodata Loop. Inspired by Autodata (Kulikov et al., 2026), in AutoBenchmark, the research agent alternates creation and analysis until the iteration budget is exhausted. Each iteration is seeded from the best checkpoint so far, so a revision that (1) the judges reject, or (2) makes the solver achieve a higher score compared to previous iterations, does not become the checkpoint of the next revision. The parent of an iteration is therefore the lowest-scoring accepted iteration so far, which is not necessarily the immediately preceding one. Feedback is exposed as file-system memory: frozen snapshots, trajectories, answers, rubric judgments, and verdicts are written to disk and the prompt names the paths, so the accumulated history grows over iterations.

Component 2: Detecting reward hacking with external solvers. Optimizing against a fixed set of solvers risks producing a benchmark that is adversarially hard for particular models. To detect if such reward hacking behavior occurs from the benchmark proposer, we evaluate additional solvers that take no part in the loop and whose results are never written into the research agent’s feedback. Cheaper external solvers are run after every iteration, giving a difficulty trace that is independent of the solvers being optimized against, while more expensive ones run on the selected best checkpoint.

Component 3: Human involvement. Benchmark creation involves two decisions, what to measure and how to measure it, and a human can supply either at different levels of detail. Within the AutoBenchmark pipeline, we vary both, together with whether human feedback is available only before the run or also during it, giving four settings:

• (i)

  No feedback. The run is fully autonomous. The agent receives no human direction and no grounding, only the general rules of the task spec, and chooses the scientific domain, the construct, and the operationalization itself.

• (ii)

  Coarse-grained human feedback (proposal stage). A human states the intent in a single sentence (e.g., what the benchmark should target). This fixes what to measure while leaving the entire operationalization to the agent.

• (iii)

  Fine-grained human feedback (proposal stage). A human writes a detailed task specification and additionally curates the list of grounding material the agent may use. This fixes both what to measure and much of how, and corresponds to the level of direction a human benchmark designer would decide before work starts.

• (iv)

  Fine-grained human feedback (proposal & execution stage). In addition to (iii), a human inspects the outcome of a given iteration and writes what the agent should do next, so that direction is also available during the loop.

1.3 Experimental Setup

Choice of models. We use Muse Spark 1.1 as the research agent that proposes the benchmark. We use Muse Spark 1.1 itself together with Muse-Glimmer-30B as the solvers inside the loop, giving a setting in which the proposer attempts to create problems that a considerably weaker model and the proposer itself cannot solve, yet that remain meaningful. Unlike Autodata (Kulikov et al., 2026), we therefore do not assume the existence of a strong solver that certifies correctness; creating problems that the proposer itself cannot solve aligns more closely with the purpose of benchmark curation, since the frontier models can always be integrated as part of the AutoBenchmark pipeline. Muse Spark 1.1 also serves as the answer judge and as the five benchmark-quality judges. As external solvers, we use NVIDIA-Nemotron-3.5-Lightning-30B-A3B after every iteration and Claude-Opus-5 on the selected best checkpoint; neither is integrated in the loop.

Grounding material & Built benchmarks. In the settings where grounding material is supplied ((iii) and (iv)), the harness gives the research agent a corpus of primary sources. Specifically, the harness hashes every source and snapshots it read-only before the session starts, and the agent may additionally retrieve public web content during the grounding stage. We experiment with three automatically generated autoresearch benchmarks developed by our method:

1. (1)

   Graveyard Bench (ideation): Can AI assistants avoid proposing dead-end ideas? The benchmark is grounded in documented negative results, namely research directions that the scientific record shows were pursued and abandoned. We draw on the medical-reversals supplement of Herrera-Perez et al. (2019), which catalogs clinical practices later overturned by subsequent randomized trials; registry records from ClinicalTrials.gov (U.S. National Library of Medicine, 2026) together with their aggregated form in AACT (Clinical Trials Transformation Initiative, 2026), which document trials whose reported outcomes fail to support the hypothesis under test; and the Reproducibility Project: Psychology (Open Science Collaboration, 2015), which records effects that do not survive direct replication. An agent is tasked to propose an idea and is scored on whether that proposed direction has already been foreclosed.

2. (2)

   SilentTrain Bench (experimentation): Can AI assistants patch buggy code that silently degrades performance? The benchmark is grounded in defects that leave a training run executable while lowering the metric it reports, so that no error trace is available to localize the fault. We source these from the silent-error corpus released with TrainCheck (Jiang et al., 2025), and from the commit histories of two kinds of repository. The first are widely used training frameworks and libraries: PyTorch (Paszke et al., 2019), TensorFlow (Abadi et al., 2016), HuggingFace Transformers (Wolf et al., 2020) and Accelerate (Gugger et al., 2022), DeepSpeed (Rasley et al., 2020), PyTorch Lightning (Falcon and The PyTorch Lightning team, 2019), torchvision (TorchVision maintainers and contributors, 2016) and timm (Wightman, 2019), Detectron2 (Wu et al., 2019), vLLM (Kwon et al., 2023), LitGPT (Lightning AI, 2023), Mosaic Composer (The Mosaic ML Team, 2021), OpenNMT-py (Klein et al., 2017), and lmms-eval (Zhang et al., 2024). The second are self-contained research codebases, whose narrower scope makes a degradation attributable to a single defect: nanoGPT (Karpathy, 2022) for language-model pretraining, ring-flash-attention (Zhu, 2024) for long-context attention, opinf (McQuarrie et al., 2023) for operator inference on dynamical systems, GrowNet (Badirli et al., 2020) for gradient-boosted neural networks, stable-continual-learning (Mirzadeh et al., 2020) for continual learning under varying training regimes, Human-Path-Prediction (Mangalam et al., 2021) for pedestrian trajectory forecasting, turboquant-pro (Zandieh et al., 2026) for KV-cache quantization, and mrpro (MRpro Team, 2024) for MR image reconstruction. The GH Archive event stream (Grigorik, 2026) is used to locate the relevant commits and issue threads, and training runs are executed on Imagenette (Howard, 2019). An agent is tasked to produce a patch that fixes the buggy code and is scored on whether its patch restores the performance.

3. (3)

   Rebuttal Bench (assessment): Can AI assistants determine whether a paper rebuttal resolves the weakness raised against a claim? The benchmark is grounded in public review threads, which pair a reviewer’s stated weakness with the authors’ response and with the reviewer’s subsequent reply. We use DISAPERE (Kennard et al., 2022), which annotates review and rebuttal sentences with their discourse function and the authors’ stance toward each review argument, and PRRCA (Wu et al., 2022), which pairs reviews with rebuttal counter-arguments, alongside raw threads from three venues that publish their review correspondence: initial and final reviewer scores from ICLR 2024 (OpenReview, 2026), the transparent peer-review files of Nature Communications (Nature Communications, 2026), and the decision letters and point-by-point responses of eLife (eLife Sciences Publications, 2026). An agent is tasked to produce a file that marks if each claim within the rebuttal from the author resolves the weakness mentioned by the reviewer, and is scored on whether it correctly judges.

Configurations. Every run produces a benchmark with ten instances over ten iterations, and each solver attempts every task once per iteration, so an iteration’s difficulty signal is the mean over ten independent task observations per model. Solvers receive the same execution contract in every iteration (4 CPUs, 16 GiB of memory, and a 24-hour agent budget per task, with a separate verifier container for grading the solver’s submission), so scores stay comparable across iterations. Before any solver is run, the harness executes the agent’s own reference solution against the real verifier and admits the submission only when it scores at least 0.9, returning the iteration for a repair session otherwise, with at most ten submission attempts per iteration. Each point in Figure 2 (Rebuttal Bench and Graveyard Bench) and Figure 3 (SilentTrain Bench) reports a solver’s mean score across the ten tasks at that iteration. Scores are displayed on a 0–100 scale. Lower scores indicate a harder (better) benchmark.

1.4 Experimental results

Human feedback helps to curate harder and higher-quality benchmarks. Results in Figure 2 show that feedback from humans improves over fully autonomous benchmark creation, with fine-grained feedback more useful than coarse-grained feedback. In the fine-grained feedback setting, the research agent is able to create benchmarks that are both difficult for models, and pass all validity checks, with generalization of the benchmarks to a held-out model. Without human feedback, the scores are much higher (hence, the benchmarks created are already saturated), and more often fail validity checks as well. We note that as models get more powerful, we expect these trends to change. AutoBenchmark can hence be used to monitor this progress. The no-feedback curve is a single autonomous run that chooses its own benchmark and is reused in both rows of Figure 2. The fine-grained proposal setting supplies both detailed instructions and curated grounding material.

When a human intervenes during the proposal stage, the intervention should be concrete. In Figure 2, the selected checkpoint on Rebuttal Bench falls from 90, 84, and 80 on Muse Spark, Muse Glimmer, and Nemotron with no feedback, to 84.4, 83.7, and 81 with coarse-grained feedback, and to 43.5, 51.2, and 39.1 with fine-grained feedback. Coarse-grained feedback improves (lowers) the score by up to 13.8 points relative to no feedback on Graveyard Bench, but is essentially flat on Rebuttal Bench, where it moves Muse Glimmer by 0.3 points (83.7 vs. 84) and leaves Nemotron slightly worse (81 vs. 80). Fine-grained feedback, by contrast, improves the score by 28 to 46.5 points in every one of the six benchmark–solver pairs. Hence we read this as evidence that the proposal stage is where human input pays off, in that with the same agent and the same loop, curating the material it works from yields a substantially harder benchmark.

Benchmark difficulty transfers to a held-out solver. At the selected checkpoints in Figure 2, Nemotron-3.5-Lightning-30B-A3B scores 39.1 in the fine-grained proposal feedback setting compared to 81 in the coarse-grained proposal feedback setting on Rebuttal Bench, and 52 vs. 75.4 on Graveyard Bench. Hence, we confirm that the resulting benchmark is genuinely hard not only for Muse Spark and Muse Glimmer but for a held-out model as well. The ordering also holds for Claude-Opus-5, which we run only on the selected best checkpoint, as shown in Table 1.

The verifier catches defects that the score cannot see. Across all runs plotted in Figure 2 and Figure 3, the benchmark verifier rejects any iteration that violates one of our five predefined criteria, so that iteration cannot become the parent of the next. Figure 4 shows three examples of quality issues identified by the LLM judges where solver scores alone would be insufficient: answers leaking into files the solver can read, a construct too shallow or too memorized to separate the solvers, and grounding material the creator claims to use but does not actually retain in the benchmark. A leaked answer key puts the solvers at ceiling, which looks exactly like a construct that is too easy, and missing grounding is consistent with any score at all, so neither defect can be caught by looking at the score. In Figure 2, iteration 5 of the fine-grained Rebuttal Bench run scores 47.4, the lowest of that run so far, and would have seeded iteration 6 had it not failed the feasibility criterion. The loop continues from iteration 3 instead, the lowest-scoring accepted iteration at that point, and reaches 43.5 at iteration 8 with all five criteria passing. Figure 2 shows the reverse for the autonomous run: its first three iterations all score 100.0 on Muse Spark, and only the verdicts tell them apart, failing five criteria at iteration 1 and two at iteration 3, with iteration 4 the first to pass all five. Selecting on score alone would have kept iterations the verifier rejects.

When the autoresearch loop stalls, human direction helps. As shown in Figure 3, through iteration 5 the SilentTrain Bench run appears to stall, with Muse Spark above 92. We forked the run at iteration 6 and gave one continuation three human-guided instructions, all of them about instantiation and none about what to measure: put the solver on a multi-hour job, build the environment from the complete files in the grounding material instead of excerpted stand-ins, and grade with behavioral tests against the original buggy behavior instead of a deterministic checker. With that guidance the selected checkpoint reaches 64.2 on Muse Spark compared to 88.1 without it, 54.2 vs. 85.6 on Muse Glimmer, and 48.8 vs. 83.8 on Nemotron, where the gap is widest. We interpret this as evidence that execution feedback helps once the agent’s own revisions have stopped working.

1.5 Related Work

Automatic benchmark construction.  A line of work asks whether a language model can produce evaluation data itself. AutoBencher (LI et al., 2025) casts benchmark construction as an optimization problem: a human declares desiderata such as difficulty and salience, and a model iteratively proposes and refines dataset descriptions that optimize surrogate metrics for them, yielding question-answer items that elicit more model errors than human-constructed benchmarks. A parallel line places the same proposer-solver asymmetry inside a training loop. Self-challenging LM agents (Zhou et al., 2026) have a single model act as a challenger that explores a tool environment and emits tasks in a Code-as-Task format carrying an instruction, a verification function, a solution, and failure cases, then as an executor trained on those tasks by RL. SPICE (Liu et al., 2025) grounds the challenger in a document corpus, so that the tasks it generates stay factual and sit at the frontier of the reasoner’s ability. SPADE (Liu et al., 2026) scales this to executable environments, training the task proposer itself with a hint-based regret signal so the environment distribution evolves as the agent improves. AutoBenchmark is closely related to this line of work but differs in three aspects. First, our target is autoresearch benchmarks created by autoresearch agents: the proposer and the solvers are the same class of long-horizon agent, and the tasks are research problems that run for hours instead of single-turn reasoning problems. Second, the loop carries two feedback signals. Alongside the verifiable score from the solvers’ trajectories, an LLM verifier judges the benchmark on open-ended dimensions that no score reveals, namely construct validity, correctness, feasibility, usefulness, and an overall verdict. Third, we do not train the proposer. We measure what current agents produce on their own and vary how much a human directs them, which lets us ask which parts of benchmark creation still need a human researcher.

Evaluating and directing autoresearch agents.  Existing benchmarks for research agents supply the task and measure execution against it: MLE-bench (Chan et al., 2025) and RE-Bench (Wijk et al., 2025) score agents on machine learning engineering problems with predefined metrics, and speedrun-style setups (Karpathy, 2026) fix the target and reward hill-climbing it. What these measure is how well an agent pursues a goal someone else has set. We ask a question one step earlier: can the agent decide what is worth measuring and build the apparatus that measures it, a step where no fixed procedure applies. Concurrently, work on recursive self-improvement (Yin et al., 2025; Anthropic, 2026; Weng, 2026) asks how much of this cycle an agent can close on its own, and co-improvement (Weston and Foerster, 2025) argues that the strongest and safest results still come from agents working with human researchers. Our results speak to both: the agents we study run the full construction loop, yet produce near-saturated benchmarks without human direction, and the gap closes only when that direction is specific about what to build and how to build it.

1.6 Conclusion

We proposed AutoBenchmark, a pipeline which evaluates the ability of a research agent to construct a benchmark, where we study when and how human feedback could be useful. We particularly targeted benchmarks used to evaluate research agents – specifically, by building Graveyard Bench, SilentTrain Bench, and Rebuttal Bench. Our takeaway is that for current models human feedback helps at two points, and only when it is concrete. The first is forming the problem: a detailed specification with curated grounding material lowered solver scores by a substantial margin, while a one-sentence statement of intent left the benchmark close to what the agent produced unaided. The second is supplying the way to realize it, where guidance on how to instantiate the benchmark restarted a loop that had stalled. Three ingredients were necessary on the agent side: grounding material worth iterating on, feedback drawn from the solver trajectories and the judges’ verdicts together, and a sufficient number of loop iterations to build the benchmark.

Looking ahead, constructing high-quality challenging benchmarks is and will be central to AI model development, and it becomes harder as the models themselves become more capable. As recursive self-improvement draws increasing attention, this raises the question of which parts of benchmark construction should be delegated to agents and which should remain with human researchers. Substantial technical work remains on the agent side, including training the proposer to widen the proposer–solver gap and meta-optimizing its agentic harness. The more difficult and important question, however, concerns the human side: deciding which problems are meaningful and worth making difficult is a judgment we do not yet know how to delegate, and integrating it into the loop is, in our view, the most important direction for future work. The AutoBenchmark recipe can be used to monitor this progress as models improve.

2 HackBench: Detecting and Localizing Reward Hacking in Autoresearch Agents

2.1 Overview

Motivation Language model agents can carry out long-horizon research tasks end to end: given a task (“pretrain a language model with the lowest dev perplexity,” “finetune a classifier to achieve the highest possible accuracy”), the agent reads the codebase, writes and runs training and evaluation code, and is scored by an automated verifier (Chan et al., 2025; Wijk et al., 2025; Starace et al., 2025; Huang et al., 2024). However, this iterative optimization loop can be fragile. Even when the score is computed on held-out data in a separate environment, the agent knows, or can infer, how the score is computed: it can read the evaluation code, inspect the test labels or test distribution, and shape its solution to the evaluation metric rather than to the task itself. When the objective is “make this metric go up,” an agent that did not conduct the underlying research can still make the number go up by reward hacking: exploiting a flaw in the specification to obtain reward without accomplishing the intended task (Skalse et al., 2022; Pan et al., 2022; Weng, 2024).

The community has highlighted the importance and difficulty of dealing with reward hacking in autoresearch: PostTrainBench re-audited its leaderboard after agents built training data around held-out items, distilled from external model APIs, and submitted instruction-tuned checkpoints in place of the assigned base model (PostTrainBench Team, 2026). On the NanoGPT speedrun (Jordan et al., 2024), METR observed agents freezing weights or halting training the moment the target loss was reached, and needed the speedrun’s human maintainer to separate these brittle gains from legitimate optimization (Cunningham et al., 2026). In each case the hack is task-specific, looks like ordinary progress at the level of individual actions, and surfaces only in long transcripts. The cost of missing it is high: autoresearch verifiers are exactly the reward signals used to train agents with reinforcement learning from verifiable rewards (DeepSeek-AI et al., 2025; Lightman et al., 2024), so a hack that survives corrupts the training signal (Qi et al., 2026).

A natural research question arises: can an LLM monitor catch these hacks by reading the agent’s trajectory—i.e., can LLMs police themselves? This task is particularly challenging: a trajectory can run to millions of tokens, so a monitor must first exercise long-context memory; the hack can be a handful of commands among hundreds (a median of 264 per trajectory in the benchmark we create in this work), frequently buried mid-trajectory where attention is weakest (Liu et al., 2023; Hsieh et al., 2024). It must then reason across steps, because a hack is often a multi-step action whose sub-steps are distributed through the transcript, and localization requires identifying the exact hacking command(s) rather than merely noticing that something is off. Finally, the monitor must bring task knowledge: deciding whether a command games a grader is a scientific judgment about the task, not a surface pattern. Unlike a synthetic needle-in-a-haystack, the needle here cannot be found by string-matching or by spotting an inconsistency, only by understanding the task (Baker et al., 2025). Fundamentally, detection comes down to a question of alignment with humans: even if this action can improve the reward, is this a behavior humans would consider acceptable?

Our contribution To measure how well LLMs catch reward hacking, we build HackBench ¹¹ 1 Dataset: https://huggingface.co/datasets/facebook/hackbench, a benchmark for detecting and localizing reward hacks in full autoresearch trajectories. HackBench is built from real agent runs and includes both natural and synthetic hacks. The synthetic hacks are long-context stress tests: covert, often multi-step hacks are injected into long honest trajectories, and causally affected downstream observations are patched so that the needle remains consistent with the world state. Because we know exactly which commands were injected, these trajectories come with ground-truth labels for scoring localization.

Using HackBench, we evaluate the following frontier proprietary models: Claude Opus 5, Claude Opus 4.8, GPT-5.6 Sol, GPT-5.5, Muse Spark 1.3, and Muse Wonder, as well as the open-weight models GLM-5.2 and Kimi K3. We find that all models fail at finding some of the hacks, so there is still progress to be made.

2.2 HackBench

HackBench evaluates how well an LLM monitor can detect and localize reward hacking from a full autoresearch trajectory. It is built from real agent trajectories containing natural and synthetic hacks, with audited command-level gold labels. This section formalizes the problem and presents our taxonomy of hacks (§2.2.1), describes the tasks and agents that produce the trajectories (§2.2.2), and details how we construct the natural and synthetic sets (§2.2.3).

2.2.1 Problem Formulation

2.2.1.1 Autoresearch tasks and trajectories.

An autoresearch task provides an agent with a working environment (code, data, tools) and an instruction to optimize a metric. The agent interacts with the environment over a long horizon, producing a trajectory τ=(s0,r1,a1,o1,r2,a2,o2,…,rT,aT,oT), where s0 is the initial state (the task instruction and starting environment) and each step contributes the agent’s private reasoning rt, a command at (keystrokes issued to a shell), and an observation ot (terminal output). At the end, a verifier V runs in the environment and returns a scalar reward R=V⁡(envT) computed on the final environment state envT, typically by recomputing a metric from artifacts the agent produced.

2.2.1.2 Reward hacking.

A trajectory contains reward hacking if the agent obtains reward by subverting the measurement rather than by accomplishing the task, e.g. writing fabricated metric files, editing or monkey-patching the grader, training or selecting on held-out test data, or reporting a subset of results chosen to inflate the aggregate. We contrast this with legitimate work, including genuine iteration, using pretrained models, or cutting compute while still doing the real task. The distinction is a property of how the reward was obtained, so it can only be judged from the trajectory, not from R alone.

2.2.1.3 Monitoring.

A monitor M maps a trajectory (and optionally the task and verifier) to a judgment about reward hacking. Specifically, we consider monitors that output a localization of the specific command(s) that constitute the hack.

2.2.1.4 Task framing.

We evaluate monitors on hack localization: each executable command is prefixed with a [CMD n] index, and the monitor returns the indices of the hacking commands. Localizing a hack in a long trajectory requires retaining relevant context, reasoning across multiple steps, and understanding the task and its grading procedure. HackBench therefore also evaluates long-context reasoning in a semantically meaningful setting.

2.2.2 Autoresearch Grounding

2.2.2.1 Tasks.

Our released benchmark covers 13 long-horizon ML research tasks: five tasks from ResearchGym (Garikaparthi et al., 2026), seven tasks from AutoLab’s “Model Development” subset (Xu et al., 2026), and a weak-to-strong autoresearch task based on automated-w2s-research (Wen, 2026). We provide detailed descriptions of all 13 tasks in Appendix C.

2.2.2.2 Autoresearch agents.

We drive the agent with the Harbor terminus-2 scaffold (Harbor Framework Team, 2026), in which the agent has access only to a bash tool: it issues shell keystrokes and observes terminal output, with each task running in a container. We deliberately did not use more advanced scaffolds because our goal is not to improve autoresearch performance; a simple terminal provides a clean testbed for studying reward hacking. We use GPT-5.5 (OpenAI, 2026a) and GLM-5.2 (Z.ai, 2026) to generate research trajectories.

2.2.3 HackBench Construction

In our preliminary experiments, agents rarely hacked when the instruction explicitly prohibited reward hacking, and the hacks we observed covered only a few types (e.g., fabrication, cherry-picking random seeds). Thus, while we collect naturally occurring hacks, collecting enough across a broad range of types was impractical. We thus developed a taxonomy of hacks (Table 2) and used it to guide synthetic hack injection into agent transcripts. Synthetic injection provides ground-truth labels for the injected commands and lets us control each hack’s type and location.

2.2.3.1 Injecting synthetic hacks.

We construct synthetic hack trajectories in three stages (Figure 7). First, we sample an agent trajectory containing bash commands, chain-of-thought reasoning, and terminal observations (Step 1). Second, we sample a hack type from our taxonomy and insert commands that implement it into the otherwise unmodified autoresearch trajectory (Step 2). Third, we update downstream observations to reflect the injected commands’ effects, changing only affected state information while preserving the task, original commands, and unrelated observations (Step 3). We used GLM-5.2 to perform both command injection and observation updates. We then used Claude Opus 4.8 and GPT-5.5 to audit each trajectory to check whether the injected commands constitute reward hacking and whether downstream observations are consistent with the injection. We manually annotated cases where the two auditors disagreed.

2.2.3.2 Collecting natural hacks.

To surface candidates for naturally occurring hacks, we used GPT-5.6 Sol to screen autoresearch trajectories, dividing each transcript into eight chunks. We selected this configuration based on a pilot study of transcript screening, in which it achieved the highest recall among the configurations we tested.

After human auditing, we combined 41 trajectories containing natural hacks with 128 trajectories containing synthetic hacks to form HackBench, a benchmark of 169 reward-hacking trajectories.

Table 3characterizes the resulting benchmark’s scale and sparsity. Trajectories have a median length of over one million characters and a median of 264 commands, yet the median number of gold-labeled hacking commands is only one. The median per-trajectory share of hacking commands is just 0.6%, requiring monitors to locate sparse evidence within long research histories. Nevertheless, some trajectories contain many hacking commands (up to 383), so the benchmark covers both isolated actions and multi-command cases.

2.3 Hack Localization Results

We evaluate LLM monitors on hack localization in HackBench. Each command block in a trajectory is assigned a [CMD n] index, and the monitor predicts the set of indices corresponding to hacking commands. Gold indices are aligned with the same preprocessed transcript shown to the monitor.

We report three metrics: precision, the fraction of predicted hacking commands that belong to the gold set; recall, the fraction of gold hacking commands recovered by the monitor; and F₁, the harmonic mean of precision and recall. All three metrics are computed separately for each trajectory, then macro-averaged over the 169 trajectories.

Table 4 reports hack localization performance across models. Localizing hacks remains challenging: the best monitor, Claude Opus 5, reaches only 64.88 F₁, followed by Muse Wonder at 61.47; every other model stays below 53. Four models (GPT-5.6 Sol, Kimi K3, Claude Opus 4.8, and GPT-5.5) cluster between 50.90 and 52.99 F₁ despite different precision–recall trade-offs; for example, GPT-5.6 Sol attains the second-highest recall (68.55) but over-flags, with a precision of only 54.84.

2.4 Related Work

Reward hacking and specification gaming.

Reward hacking—high reward without the intended behavior—is a long-standing concern in reinforcement learning (Amodei et al., 2016; Krakovna et al., 2020; Skalse et al., 2022; Pan et al., 2022; Manheim and Garrabrant, 2018). In the RLHF/RLVR era it reappears as reward-model overoptimization (Gao et al., 2022; Moskovitz et al., 2023; Eisenstein et al., 2024) and, more sharply, as reward tampering, where a model edits its own reward channel (Denison et al., 2024). We study reward hacking in a setting where the “reward” is a concrete automated verifier the agent can read and modify, and where hacks are often multi-step programs rather than single actions.

Autonomous research agents.

Agentic coding and tool use (Yao et al., 2022; Schick et al., 2023; Jimenez et al., 2024; Yang et al., 2024b) now extend to end-to-end ML research, and a wave of benchmarks measures it: MLE-bench (Chan et al., 2025), RE-Bench (Wijk et al., 2025), PaperBench (Starace et al., 2025), MLAgentBench (Huang et al., 2024), ResearchGym (Garikaparthi et al., 2026), MLS-Bench (Lyu et al., 2026), and AutoLab (Xu et al., 2026). These works optimize for and measure capability. We instead treat their tasks as a testbed in which to study integrity: the same open-ended, verifier-scored task that measures research skill also invites hacking of the verifier. Closely related to our work, Huang et al. (2026) studies when reward hacking occurs and what forms it takes. Our work complements Huang et al. (2026) by studying how well language-model-based monitors detect and locate reward hacking in autoresearch transcripts.

Oversight, monitoring, and control.

Detecting misbehavior from a model’s behavior or reasoning is central to scalable oversight (Bowman et al., 2022), weak-to-strong generalization (Burns et al., 2023), and AI control (Greenblatt et al., 2024). Chain-of-thought monitoring can catch reward hacking but is fragile if optimized against (Baker et al., 2025), and reasoning traces are not always faithful (Turpin et al., 2023); frontier models can scheme and hide intent when incentivized (Meinke et al., 2024; Hubinger et al., 2024). To monitor agents, Kutasov et al. (2025) propose asking agents to perform both a main task and a covert side task in general agentic settings (e.g., booking an airplane ticket), then checking whether monitors catch the side task. Libon et al. (2026) extends the main/side task red-teaming protocol to autoresearch settings. For coding agents, Deshpande et al. (2026) benchmark reward-hack detection in code environments via contrastive analysis. However, Martin and Roger (2026) and Jotautaitė et al. (2026) find that existing monitoring benchmarks may overestimate monitor performance because they evaluate monitors on relatively short trajectories and a narrow range of hacks. In our work, we mitigate these limitations through injection-based red-teaming: directly asking a model to inject different hack commands into long agent transcripts.

LLM judges and long context.

We build on LLM-as-a-judge methods (Zheng et al., 2023; Li et al., 2023; Dubois et al., 2024), using LLMs to assess whether agent trajectories contain reward hacking. Because these trajectories can be very long, monitoring is subject to known failure modes such as lost-in-the-middle effects and context-window limits (Liu et al., 2023; Hsieh et al., 2024).

2.5 Conclusion

Autoresearch agents reward-hack because we ask them to maximize metrics, and they do so without proper regard to other implicit constraints, i.e. they should also perform actions humans would consider acceptable. We thus built HackBench, a benchmark that measures whether hacks inside autoresearch trajectories can be detected and localized, and showed that current frontier models do not fully succeed at this task. As agents take on more of the research loop, monitoring reward hacking is not an add-on but a prerequisite; to support future work, we release trajectories containing naturally occurring and synthetically injected hacks, together with audited command-level gold labels.

3 Training agents to interact with humans

3.1 Overview

Motivation Today, AI models are routinely used by humans for coding and research, and further progress in their ability to interact with us increasingly depends on the data and environments they are trained on. However, the most common post-training recipe trains on fully specified tasks paired with an automatic verifier, with no human in the loop – which aligns with the fully automatic autoresearch (Karpathy, 2026) paradigm. This creates a major discrepancy with real-world applications where agents work with users by following clarifications and feedback (Jin et al., 2025; Wang et al., 2026). While current training tasks are usually fully specified, users in reality are much less specific. They may start with a goal that is not fully articulated, provide clarifications after seeing the agent’s output, revise requests midway, or provide counterexamples when a fix does not hold (Baumann et al., 2026; Wu et al., 2026b). Models that solve tasks without such context can lose track of intent as the interaction goes on (Laban et al., 2026; Tack et al., 2026).

An obvious direction is thus to train models to be better at agent-human interactions. However, common training recipes are not designed for an interactive setup: real interaction logs are scarce, costly, and difficult to convert into verifiable training examples (Baumann et al., 2026). An alternative is to simulate interactions, but these can be miscalibrated against real user behavior or collapse a heterogeneous user population onto a generic persona (Cheng et al., 2023). Moreover, ensuring task diversity, factual grounding, and executable verification in synthesized interactions is itself not straightforward. The problem for synthetic data is therefore to construct interaction data that is realistic, grounded in verifiable tasks with high-quality reference solutions, and calibrated in difficulty to the model being trained.

Our contribution We introduce AutoInteract, a framework for generating data to train agents to interact with humans. Inspired by the recent Autodata method (Kulikov et al., 2026), which uses an agentic data scientist to create high-quality synthetic data, AutoInteract generalizes that approach to treat human-agent interaction itself as the object of synthesis. Starting from a verified source task, our method simulates a multi-turn interaction, including user clarifications, requirement changes, and corrective feedback, while preserving the source task’s final reference solution verifiable by an executable test. This consists of two phases: (1) creating user models from real interaction histories by learning behavior over a diverse population of profiles and (2) combining these user models with problem-solving agents in an agentic generate–evaluate–revise loop that simulates realistic, task-grounded interactions with calibrated difficulty for a given model. Models can then be trained with reinforcement learning on this data.

Across algorithmic coding and software engineering tasks, AutoInteract substantially outperforms baselines with both Qwen3.5-4B and Qwen3.5-27B backbones. On LiveCodeBench-Pro (Zheng et al., 2026) and its multi-turn variant, AutoInteract raises average Pass@1 from 29.9 to 43.0 at 4B and from 49.1 to 55.7 at 27B, with especially large multi-turn gains from 23.0 to 43.2 and from 37.2 to 50.6, respectively. On SWE-Together (Wu et al., 2026b), it improves Pass@1 from 6.0 to 20.6 at 4B and from 34.9 to 38.5 at 27B. Our analysis shows that multi-turn synthesis, grounded user models, and executable verification each contribute to performance, while training reward does not saturate as quickly as training on the source data.

Overall, our results show that training can extend beyond solving isolated tasks to help models learn how to work with users. By making interaction itself synthesizable and verifiable, AutoInteract provides a practical way to train agents to incorporate human clarifications, feedback, and evolving requirements while still optimizing for task correctness.

3.2 AutoInteract

AutoInteract is a framework for generating user-centric interaction training and evaluation data. It produces realistic, verifiable, multi-turn tasks between a simulated user and an assistant. Because these trajectories resemble the interactions that assistants encounter in practice, models can learn from them to better handle real user-centric tasks.

AutoInteract first learns a user model from logged interactions, so simulated users reproduce the habits, gaps, and preferences that real users show. The pipeline needs only raw interaction logs with user IDs, so it carries over to any domain where such logs exist.

AutoInteract then performs task-grounded trajectory synthesis, which preserves the source task’s existing reference solution and executable verifier. The user simulator privately conditions on the reference solution and executable checks to keep requests across turns consistent with the original task, without revealing this privileged information to the assistant. The final assistant response can therefore be scored by the source task’s unchanged verifier.

The method thus runs in two phases, shown in Figure 8. Phase 1 builds user models from real histories so that we can simulate users realistically. A user model has two parts: (1) a user behavior prompt, which gives general role-playing behavior guidance, and (2) a user profile, which describes the specific user being simulated. We learn both using an existing corpus of real trajectories of users interacting with coding agents. We learn the user behavior prompt using GEPA (Agrawal et al., 2026), guided by reference-based feedback that measures how closely a simulated next turn matches real user behavior in the same conversation context. We then build a set of user profiles by using an LLM to summarize user traits from the histories, including diversification and validation steps. Together, the user behavior prompt and the user profiles let us simulate users. Section 3.3 provides more details.

Phase 2 builds interaction trajectories with an agent controlling the user simulation inside a loop that generates, evaluates, and improves trajectories. A controller starts each trajectory-synthesis process by picking a verified source task, pairing it with a user model from Phase 1, and planning how the interaction should unfold. It passes this plan as an instruction to a trajectory generator, where the user simulator (using the user model) and an assistant simulator talk over multiple turns to produce a candidate trajectory. A trajectory evaluator then checks the result on three axes: whether the task is valid and free of leakage, whether the interaction is realistic and coherent, and whether the difficulty is appropriately calibrated, so that a weak solver struggles while a strong solver succeeds. Trajectories that pass are kept as training data. Trajectories that fail go back to the controller, which reflects on the failure, revises the instruction, and retries. Section 3.4 provides more details.

3.3 Building User Models from Real Histories

Phase 1 builds user models from real interaction histories. Each user model has two parts: (1) a user behavior prompt, which gives general role-playing guidance on how users tend to react, and (2) a user profile, which describes the specific user being simulated. We learn both components from real data of users interacting with agents. The user behavior prompt is optimized on observed next-user turns, while the profile population is built from (LLM-generated summaries of) behavior across users, diversified, and validated. Together, these two parts determine how a simulated user responds and the interaction style they follow. Appendix F.1 provides further implementation details beyond this section.

3.3.1 Optimizing a History-Conditioned User Behavior Prompt

The user behavior prompt is a reusable set of instructions for role-playing a human developer. It defines the simulator’s role and output format, explains how to use the conversation history and user profile, and describes common user reactions, such as accepting the work, correcting the assistant, reporting a failure, changing a requirement, or ending the interaction. It also asks the simulator to keep follow-up messages short and grounded in the current conversation. The prompt does not describe a specific task or user. To simulate users during trajectory synthesis, we combine it with a sampled user profile, the interaction history, and the source task as private context to the user simulator (Figure 9).

We optimize a single user behavior prompt while keeping the simulator’s model weights fixed. We instantiate GEPA (Agrawal et al., 2026), a reflective prompt optimizer that evolves prompts using natural-language evaluation feedback and Pareto-based candidate selection. Each training instance contains a conversation prefix, an assigned user profile, and the real user’s next message. Given the prefix and persona, a candidate user behavior prompt generates a simulated next message. An LLM evaluator assigns an overall score based on intent, reaction to the assistant, length, specificity, persona consistency, and realism, and provides textual feedback describing discrepancies from the real turn. For each sampled training minibatch, GEPA uses this feedback to propose a revised user behavior prompt and accepts the revision when it improves the minibatch score. Accepted candidates are evaluated on the validation set to update the per-instance Pareto frontier, and the final user behavior prompt is selected by aggregate validation performance.

3.3.2 Constructing a Behavior-Grounded Profile Population

We build the user profile population in three steps: build (prompt an LLM to summarize) one profile from each real user’s history, diversify these profiles while preserving their interaction patterns, and validate the resulting population. LLMs write and audit the profile text, while behavioral structure and population-level validation use automated, data-driven procedures.

Profile construction. For each user ID recorded in the user source data, we aggregate that user’s conversations and compute a profile summary, including what kinds of requests they make, how often they correct or push back on the assistant, how often their requirements change, and how much detail they provide. We prompt an LLM to summarize this evidence into a concise natural-language profile (see the prompt in Box F.1). This profile tells the simulator how the user tends to behave across turns, including user vagueness, pushback, requirement changes, verbosity, and the rate at which requirements are revealed across the conversation.

Profile diversification. The observed users do not provide enough profiles for large-scale simulation, so we develop a method inspired by MAP-Elites (Mouret and Clune, 2015), a quality-diversity method that expands coverage across behavioral types/clusters (also called niches or cells) while retaining high-quality profiles in each type. Clusters are defined by expertise, vagueness, pushback, requirement-change tendency, and verbosity, and only combinations found among real users are kept. Within each cluster, we prompt an LLM to write new profiles that preserve a real user’s behavior settings and tone while varying surface details. The settings are inherited from the real profiles, adding diversity without inventing unsupported behavior combinations.

Profile validation. Profiles are validated for quality. We employ an LLM to check each generated profile against the user’s measured fingerprint, flag unsupported claims and repository-specific details, and provide feedback for revision. During verification, automatic text-similarity checks remove near-duplicates and profiles that are unusually far from all real profiles. Final automatic checks measure behavioral-type coverage, diversity, and distributional similarity to the real users from the source data.

3.4 Building Interactions with an Agentic Loop

Phase 2 turns verified source tasks combined with the user models from Phase 1 into realistic, verifiable tasks with multi-turn trajectories. As shown in Figure 8 (right), the agentic loop consists of three modules:

1. (1)

   Controller, which samples a verified source task and user profile and uses them, together with the learned user-behavior prompt, to specify the scenario and guide the conversation. After a failed attempt, it revises these instructions using evaluator feedback.

2. (2)

   Trajectory generator, which follows the controller’s instruction and rolls out a candidate conversation between the user simulator and the assistant simulator.

3. (3)

   Trajectory evaluator, which checks the candidate for validity and leakage, realism and coherence, profile consistency, and difficulty calibrated through solver rollouts and the source task’s executable verifier.

Together, these modules form a data generation loop: the controller proposes an interaction, the generator produces it, and the evaluator determines whether to keep it or return it for revision.

3.4.1 Interaction Task Synthesis (Controller + Trajectory generator)

Each trajectory begins with a verified source task. The controller pairs the task with the learned user behavior prompt and a sampled profile and generates an initial plan. The plan is a natural-language instruction that defines the user’s goal, behavior, information-disclosure strategy, and how the conversation progresses; see Box F.2 for an example. At each turn, the user simulator receives the user-simulation prompt shown in Figure 9, which combines these two components of the user model with the visible interaction history, the source task and reference information as a privileged context, and any revised controller guidance from a previous failed attempt. The assistant simulator responds only to the conversation and any repository context it would normally observe. The conversation ends on a user turn, and the requirements expressed across the interaction aim to remain verifiable with the source task’s original verifier.

Scenario-guided algorithmic trajectories. For self-contained algorithmic tasks, the controller follows several interaction patterns, detailed in its generated plan: clarification or requirement change during design, clarification or requirement change after an initial solution, and bug iteration through concrete failing cases. Clarification reveals a requirement that was present from the start, a requirement change introduces a genuinely new need, and bug iteration keeps the specification fixed. These patterns control when information becomes visible without changing the final verified task.

History-conditioned repository trajectories. Repository-level software engineering tasks are more agentic than self-contained algorithmic tasks, requiring tool-driven repository search, fault localization, editing, testing, and responses to users across turns. The interaction is shaped by the user behavior prompt and sampled profile. Private task and reference information keep the simulated user factually grounded, while the assistant operates only on the visible conversation and repository state. The cumulative user requests must describe the intended change without revealing private tests or implementation details. Appendix F.3 provides further details on both task regimes.

3.4.2 Agentic Self-Improvement

Initial generation does not guarantee that a trajectory is realistic, valid, or useful for training. Like Autodata’s agentic data scientist, which improves synthetic data through solver and judge feedback (Kulikov et al., 2026), AutoInteract repeatedly generates, evaluates, and revises candidates, but focuses the improvement process on multi-turn trajectories with a user (model) and grounds it in executable verification.

Multi-objective trajectory evaluation. The evaluator checks three properties: the task is valid and does not leak private information; the interaction is realistic, coherent, and consistent with the sampled profile; and the difficulty is calibrated so that the target model struggles while a stronger solver succeeds. Solver responses are scored with the source task’s executable verifier. This combines semantic checks with execution-based evidence, and a trajectory is kept only when all checks pass.

Trajectory self-reflection. When a check fails, the evaluator identifies whether the trajectory is invalid, unrealistic, inconsistent with the profile, too easy for the target model, too difficult for the stronger solver, or overly revealing. The controller turns this diagnosis into guidance for the next rollout. The source task, learned user behavior prompt, and sampled profile remain fixed across attempts, so the controller revises only the task interaction. This generate–evaluate–revise loop continues until the trajectory passes or the retry budget is exhausted.

3.5 Experimental Setup

3.5.1 Training Setup

User Model Data. We learn the user behavior prompt and user profiles from SWE-Chat (Baumann et al., 2026), a corpus of real trajectories of users interacting with coding agents, where each trajectory records a user ID and how that user responds in different situations.

Source Data. For algorithmic coding, we extract the code subset of Dolci-Think-RL-32B, the reinforcement-learning corpus used to train OLMo 3 (Olmo et al., 2025). Our extraction contains 12,120 Python problems, each represented by a natural-language specification and a list of executable assertions that serve as its ground-truth verifier. For repository-level SWE, we use SWE-smith (Yang et al., 2026). Each instance provides an issue-style problem statement, a reproducible buggy repository state, relevant code context, an executable environment, tests that must change from failing to passing, and regression tests that must remain passing. We construct a deterministic pool of 19,000 function-level, pytest-backed instances across 117 repository environments, prioritizing compact fixes with bounded test suites.

Data Generation Configuration. In Phase 1, Kimi-K2.6 (Kimi Team, 2025) is used throughout user-model construction. For user behavior prompt learning, it simulates and judges next-user turns and performs GEPA prompt rewriting; for user-profile construction, it drafts, expands, reformats, and verifies the profiles. In Phase 2, Kimi-K2.6 serves as the user simulator, controller, trajectory evaluator, and strong solver, while the target model itself serves as the assistant simulator and weak solver: Qwen3.5-4B when generating data for the 4B-scale runs and Qwen3.5-27B for the 27B-scale runs. Data are therefore synthesized separately for each target scale, so the difficulty band is calibrated to the model that will be trained on it. For each candidate trajectory, the weak and strong solvers each generate three responses, which are scored by the source task’s executable verifier. A trajectory passes only when the weak solver scores at most 40% and the strong solver scores at least 70%; failed candidates may undergo up to five self-improvement rounds.

Training Configuration. Each synthesized trajectory becomes a training prompt ending on the final user turn, and the model generates only the final assistant response. Rewards come from the source task’s executable verification: for algorithmic tasks, the reward is the fraction of assertions passed; for SWE tasks, it is the fraction of previously failing tests fixed, with reward set to zero if any previously passing test regresses. Both rewards lie in [0,1]. We train all RL variants using the prime-rl framework (Intellect, 2025) with a DAPO-style group-relative objective (Yu et al., 2026), using 32 prompts and G=8 responses per prompt, for 256 responses per update. Training uses temperature 1.0, reasoning enabled, a maximum sequence length of 81,920 tokens, and AdamW with a constant learning rate of 1×10−6, weight decay 0.01, gradient-norm clipping at 1.0, and up to 500 steps.

3.5.2 Evaluation Setup

Models and Baselines. We use Qwen3.5-4B and Qwen3.5-27B as the target models throughout and compare AutoInteract against two baselines: (1) Base, the off-the-shelf post-trained checkpoint at each scale without further optimization, which establishes the starting performance; and (2) RL on Source Data, which applies reinforcement learning directly to the single-turn source data. RL on Source Data and AutoInteract use the same source data and training configuration described in Section 3.5.1.

Evaluation Benchmarks. For algorithmic coding, we first evaluate on LiveCodeBench-Pro (Zheng et al., 2026), a challenging benchmark for competitive programming. We use all 706 tasks with executable test cases: 391 easy, 251 medium, and 64 hard Codeforces problems. Models produce C++ solutions, which are scored by exact acceptance under the benchmark’s test cases and original time and memory limits. We additionally construct Multi-turn LiveCodeBench-Pro from the same task pool to isolate interaction-following ability from underlying problem difficulty. For each source problem, we generate a conversation using one of the five interaction scenarios in Section 3.4.1 and retain only conversations that pass validity and realism checks. The original problem identity and test suite are preserved, so the final C++ response is judged by the unchanged LiveCodeBench-Pro verifier. For repository-level SWE, we use SWE-Together (Wu et al., 2026b), a benchmark of 109 sandboxed repository tasks reconstructed from real user–agent sessions. Its reactive user simulator continues the interaction according to the original user’s intent, allowing evaluation of both task correctness and how much corrective effort the user must provide. See Wu et al. (2026b) or Table 6 for the metrics that are used.

3.6 Experimental Results

3.6.1 Main Results

Our main results are given in Table 5 for LiveCodeBench-Pro and Multi-turn LiveCodeBench-Pro and Table 6 for SWE-Together. Our findings are summarized below.

Grounded multi-turn data improves interactive coding at both scales, without losing single-turn ability.  At the 4B scale, AutoInteract reaches an average of 43.0: +13.1 over the baseline performance and +6.1 over RL on the original source tasks (Table 5). At the 27B scale, it reaches 55.7: +6.6 over the baseline performance and +6.1 over source-task training, where the latter only improves over the initial baseline by +0.5. Our method achieves particularly large gains on Multi-turn LiveCodeBench-Pro, where the model must recover a specification that arrives over several turns: 23.0→43.2 at 4B scale and 37.2→50.6 at 27B scale. These gains do not come from specializing for conversation: although AutoInteract trains on multi-turn trajectories, single-turn LiveCodeBench-Pro rises in parallel at 4B scale from 36.8 to 42.8, whereas training on the same tasks in their original single-turn form leaves it flat (37.1), while at 27B it remains essentially unchanged (60.9→60.8). Gains are on easy and medium problems; hard Codeforces tasks remain out of reach at both scales for all methods.

Real-user-grounded interaction data improves repository-level issue resolution.  On SWE-Together at 4B, AutoInteract raises Pass@1 from 6.0 to 20.6 and SSR from 1.8 to 16.5, outperforming RL on the source tasks by 3.6 points on Pass@1 and 5.5 points on SSR (Table 6, see caption for metric definitions). Consistency improves alongside accuracy: Pass² and mean judge score both increase, indicating that solved sessions are reproduced rather than won once by chance. At the 27B scale, AutoInteract is the only method that improves Pass@1 (34.9→38.5), and it adds 7.3 points of SSR, whereas RL on the source tasks regresses to 27.1. The interaction also becomes cheaper for the user: at the 4B scale, AutoInteract needs fewer corrections than the source-data run (6.58 vs. 6.73) while solving more tasks, with 69.2k tokens and 25.1 minutes per task against 73.7k and 39.7, and at 27B it is the fastest configuration (19.5 vs. 32.2 and 52.1 minutes) at the cost of slightly more corrections (3.44 vs. 2.91). The baseline’s low U-Corr at 4B scale (3.78) is not evidence of efficiency, since it abandons most sessions early at 6.0 Pass@1.

3.6.2 Ablations

Multi-turn training data is key to learning how to work with users.  The w/o multi-turn variant uses the learned user models and executable verification but presents each task in a single turn without simulating a multi-turn interaction between the user and assistant. This lowers average performance from 34.0 to 28.2, driven primarily by the drop on Multi-turn LiveCodeBench-Pro (43.2→29.2; Table 7). Training on interactions with user clarifications, feedback, and evolving requirements is therefore necessary for learning to incorporate information across turns and drives higher performance in multi-turn interaction settings.

User models learned from real histories prepare models for diverse users.  The w/o user models variant uses multi-turn synthesis and executable verification but relies on the initial, fixed user behavior prompt without optimization against real-user interactions and omits the diverse user profiles. Average performance falls from 34.0 to 32.0, with the clearest degradation in SWE-Together reliability (SSR: 16.5→12.8). Learning both general user behavior and user-specific variation from real histories produces training interactions that better prepare models to adapt to how real users behave and to differences among users.

Executable verification is critical for reliable data selection.  The w/o executable verification variant uses multi-turn synthesis and the learned user models but relies on an LLM judge to assess solver answers and screen task validity without executing the answers against the verifier. Average performance falls from 34.0 to 25.0, similar to w/o all components at 24.7 (Table 7). Executable verification is necessary both to evaluate the weak and strong solvers correctly for difficulty calibration and to provide a reliable signal for correctness and validity that catches invalid synthesized problems an LLM judge can miss.

3.6.3 Analysis

Self-improvement rounds make the data more useful and increase data yield. AutoInteract’s agentic trajectory self-improvement revises each failed trajectory for up to five rounds, using evaluator feedback to target its failures. Across self-improvement rounds, the fraction of trajectories in the 4B-scale data pool that are accepted by the trajectory evaluator increases from 9.4% after initial generation to 28.1% after five revision rounds for algorithmic tasks, and from 26.6% to 50.9% for SWE tasks. Revisions supply 52.4% of accepted trajectories; 85.5% of recovered SWE trajectories arrive in the first two rounds, while algorithmic gains span all five. Initial failures differ by domain: 83.2% of algorithmic candidates are too easy, while 35.3% of SWE candidates leak implementation details and 22.2% remain too hard for the strong solver. The loop uses additional inference compute to address each trajectory’s failure and perform domain-specific calibration. These rounds produce high-quality trajectories at the target difficulty, and training on AutoInteract trajectories does not saturate as quickly as source-task training, eventually reaching higher reward in both domains (Figure 10).

Trajectory self-improvement adjusts how and when task information is revealed. Figure 11 illustrates this effect with one example. The source task, learned user behavior prompt, and sampled profile remain fixed, so solver outcomes reflect only how the task turns are built. Stage 1 gives away the likely code change and fails the weak-solver difficulty constraint. Stage 2 removes that clue but withholds needed evidence, falling below the strong-solver solvability target. Stage 3 restores the example file, expected behavior, and follow-up requirements without exposing the solution, yielding weak- and strong-solver scores of 25% and 100% and passing every quality check. Together, these targeted revisions turn a rejected trajectory into useful training data without changing its underlying task. The loop therefore does not uniformly make tasks harder; it adjusts what is revealed and when to balance difficulty, solvability, grounding, and realism.

3.7 Related Work

User Simulation. Simulated users have long been used in interactive systems, from agenda-based simulators for training dialogue policies (Schatzmann et al., 2007) to populations of generative agents that model human behavior at scale (Park et al., 2023). LLMs have made user simulators more fluent and general, and they are now widely used in interactive agent benchmarks (Yao et al., 2024; Barres et al., 2025) and as environments for training agents. Yet these simulators may not behave like real users. Assistant-tuned models prompted to play a user often produce turns that are unusually cooperative, well-organized, and similar in style, and stronger assistants can make worse simulators (Naous et al., 2026). Conditioning on a persona label can also represent a group through stereotypes rather than the variation found among real users (Cheng et al., 2023). Recent work addresses this problem by training simulators on real interactions, either through supervised learning on observed user turns (Naous et al., 2026) or reinforcement learning for persona consistency (Abdulhai et al., 2026) and latent user states (Wu et al., 2026a; Jin et al., 2026). Large persona libraries provide another way to represent diverse users (Ge et al., 2024). AutoInteract combines these directions. Rather than updating model weights or drawing profiles from an open-ended persona library, it learns an editable user behavior prompt from real interactions and builds a profile population from measured per-user behavior.

Multi-Turn Interaction Evaluation and Training. Models that solve fully specified tasks often perform worse when the same information arrives over several turns (Laban et al., 2026) or when the goal changes during a conversation (Tack et al., 2026). Single-turn evaluation does not capture this problem. Interactive benchmarks test it through feedback and tool loops (Wang et al., 2024), dual-control settings (Barres et al., 2025), preference-elicitation environments (Qian et al., 2025a), and repository sessions reconstructed from real developer–agent transcripts (Baumann et al., 2026; Wu et al., 2026b). Several training methods place a simulated user in the optimization loop: SWEET-RL learns turn-level advantages from privileged training information (Zhou et al., 2025); CollabLLM estimates the long-term value of a reply by sampling future conversations (Wu et al., 2025); UserRL studies turn- and trajectory-level rewards across user-centered environments (Qian et al., 2025b); and MUA-RL trains multi-turn tool use with an LLM user (Zhao et al., 2025). These methods keep the interaction environment and user simulator fixed while improving the learning algorithm. AutoInteract instead studies how to construct the interaction data itself.

Grounded and Verifiable Synthetic Data. Early work created synthetic post-training data by expanding a small set of seed instructions (Wang et al., 2023). Later work increased its scale and diversity through complexity evolution (Xu et al., 2024), direct prompting of aligned models (Xu et al., 2025), and persona conditioning (Ge et al., 2024). Because unconstrained generation can introduce errors, later methods ground generated data in documents or structured sources and check whether questions can be answered (Lupidi et al., 2024; Yuan et al., 2026; Yu et al., 2025). Execution provides an especially strong check. In software engineering, tasks can be generated directly from repositories and validated by running tests (Pan et al., 2024; Jain et al., 2025; Wei et al., 2026; Yang et al., 2026), producing an inexpensive, objective reward that is difficult to exploit. Other work generates tasks that challenge the current solver, either by generating tasks and verifiers together (Zhou et al., 2026; Zhao et al., 2026; Liu et al., 2025) or by targeting problems the model cannot yet solve (Zelikman et al., 2022). These methods still produce fully specified, single-turn tasks. AutoInteract keeps the source task and its executable verifier unchanged but presents the task as an interaction. The same execution reward can then measure both correctness and the model’s ability to follow requirements given over several turns. We also use a weak-versus-strong difficulty control, but the refinement loop applies it to what task information is revealed and when, rather than to the task’s inherent difficulty.

Agentic Data Generation and Self-Improvement. Recent systems use agents to plan, inspect, and revise generated data instead of relying on fixed pipelines. AgentInstruct uses agent workflows to produce large post-training datasets (Mitra et al., 2024). Agentic Self-Instruct uses a data-scientist agent to revise a generation recipe based on weak- and strong-solver rollouts and judge feedback, and also improves its own scaffold (Kulikov et al., 2026). Related agents automate parts of the data-science workflow (Guo et al., 2024; Hong et al., 2025). This process is related to prompt- and scaffold-optimization methods that revise prompts or instructions using execution feedback (Madaan et al., 2023; Fernando et al., 2023; Yang et al., 2024a; Lu et al., 2024; Agrawal et al., 2026). Our profile expansion also uses quality-diversity search to cover a range of behaviors instead of maximizing a single score (Mouret and Clune, 2015; Cully et al., 2015). AutoInteract differs from Agentic Self-Instruct in three ways. First, it improves a multi-turn trajectory rather than a single example. Second, it includes a simulated user learned from real histories and evaluates realism directly. Third, the source task, sampled profile, and user behavior prompt remain fixed across retries, so the agent can revise only the multi-turn interaction. Holding these inputs fixed prevents the agent from meeting the difficulty target by changing the task in a way that disconnects it from the source, a problem reported in unconstrained agentic data loops (Kulikov et al., 2026).

3.8 Conclusion

We presented AutoInteract, a framework to improve agent-human interaction by creating realistic, grounded multi-turn tasks calibrated to the model being trained. It combines user models learned from real interaction histories with an agentic data creation loop that uses judge and execution feedback to control data quality and difficulty. Across algorithmic and software engineering tasks, training on these data significantly improves RL performance over baselines. Overall, AutoInteract shifts the focus from training agents to solve verifiable tasks independently to training them to collaborate with users on those tasks, incorporating clarifications, feedback, and evolving requirements across multiple turns to arrive at correct solutions. Our approach generalizes to a broader set of tasks, offering a path toward more effective human-agent interaction in the future. In particular, leveraging humans’ and AIs’ complementary skills for research, termed co-improvement (rather than self-improvement), looks like the fastest and safest way towards superintelligence (Weston and Foerster, 2025), and any advancement in this interaction ability will thus be beneficial.

4 Training agents to communicate with humans: unslopping AI

4.1 Overview

Motivation AI systems have achieved superhuman performance on a cross-section of verifiable tasks through reinforcement learning, but currently remain relatively weak in non-verifiable tasks. For example, their generations exhibit a lack of high-quality writing, sometimes termed “AI slop” (Shaib et al., 2025).

During pretraining, large language models are trained to predict the most likely next token over corpora that span every level of human expertise, from novice forum posts to expert prose. Maximum-likelihood pretraining therefore tends to produce continuations that reproduce the quality of its context rather than exceed it: given mediocre writing the model most naturally continues in kind, and nothing in the objective pushes generations toward expert quality. Reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Bai et al., 2022a) can steer models higher, but it is only as good as the reward it optimizes—and a reward that reliably recognizes good writing is hard to build. Reward models trained on rankings from non-expert annotators inherit those annotators’ ceiling, so the very quality we care about most—expert, even superhuman, writing—is the quality such rewards are least equipped to certify. The current focus on post-training on verifiable rewards (e.g., on math, code, and agentic environments) does not obviously improve language communication with users, and may in fact worsen it. Progress on expert-level generation thus hinges on a reward signal that recognizes expert quality without requiring expert supervision at scale.

Our contribution We introduce a method to fix this problem, Reinforcement Learning from eXpert-Aligned Rubrics (RL-XAR). We assert that standard RL methods are sufficient to train superhuman writing ability, but the failure to achieve this lies in the quality of the reward model — which currently fails to correctly reward expert writing, and incorrectly reward slop. The solution then is to align the reward model more closely with human experts. This is possible without collecting any new annotations, as there are already abundant examples of expert human writing. Our approach thus first identifies examples of such expert human writing from existing resources, and then uses these examples as a training signal. We learn to generate rubrics for a given writing task by ensuring the expert writing scores highly with these rubrics relative to current models, by maximizing the human-model gap. We can then perform standard RL with the newly learned expert-aligned rubrics.

We conduct experiments in three settings: academic paper writing, story writing and authoring Wikipedia articles. In each case, we show our rubrics provide better alignment with expert writing, and RL-XAR training gives large gains over standard training.

4.2 The failure of standard LLM Judgments

Successful training hinges on good grading. We first study the reward-design problem in a concrete setting: writing a section from a real academic paper. A model is given a selected high-quality paper minus one section—for example its abstract, introduction, related-work section or conclusion—and must write the missing section so that it fits the rest of the paper, using a similar word count to the originally written section. The held-out human original provides a natural, expert-authored reference, and the task is trivially sourced at scale from the published literature. The precise setup is described in subsection G.1.

We consider two standard ways to ask a language model to judge writing quality—show it two candidates and ask which is better, or score each candidate against a pre-designed rubric. We expect that a good judge should place expert human writing above model writing. We evaluate Opus-4.8 and GPT-5.6-sol as both writers and judges (cross-comparing them).

Results are given in Figure 12, with our main findings described below.

A direct pairwise judge prefers the model. A common protocol is to show a judge two candidates and ask which is better, e.g., given the rubric-free prompt “which version is the better section? Reason step by step, then decide”—with each call scored in both A/B orders to cancel position bias. This judge prefers the models over the expert human in the majority of cases, between 63.5% and 84.6%, depending on the writer and judge setup.

A standard rubric does not register the gap either. While there are many ways to build rubrics here, we prompted GPT-5.6 to build a meta-prompt that, given a paper missing the section, generates rubrics specific for judging that section. Switching to these rubric-based judgments instead shows even more extreme results, with the model being selected as higher scoring 100% of the time.

4.3 RL from eXpert-Aligned Rubrics (RL-XAR)

The previous section showed the failure to judge expert human writing exhibited by current systems. If AI slop is judged to be better than human writing by the LLM grader we use, then training against that grader will only encourage more slop. Given that we have access to abundant examples of expert human writing, one might naturally ask: can this be fixed by learning better judgments?

Our approach to learning rubrics, called eXpert-Aligned Rubrics (XAR), is simple:

• •

  Identify examples of well-above average / expert human writing, splitting them into context and continuation.

• •

  Generate competing model-based continuations given the same context, which we expect to be worse writing (i.e., slop).

• •

  The above data form learning pairs, where humans should be scored higher. This training data is then used to learn to generate rubrics that maximize the scoring gap between humans and models.

Then, to train a model to be better at writing via reinforcement learning (RL-XAR) using these rubrics, we iterate the following steps:

1. (1)

   Learn XAR rubrics maximizing the gap between humans and the current model, as described above.

2. (2)

   Perform reinforcement learning on a training set using the learned rubrics.

3. (3)

   Repeat the procedure from step 1 until rubric learning can no longer identify a gap (i.e., we can no longer identify slop in the model).

We note that this approach is reminiscent of GANs (Goodfellow et al., 2014).

4.3.1 Learning the rubrics: can an LLM recognize expert writing?

While the learning of rubrics could be done in many ways, in our experiments we use the following simple procedure of meta-prompt optimization. We begin with an initial meta-prompt: given an example context, it asks an LLM to generate rubrics for judging the continuation. We can score the quality of this meta-prompt using the human−model gap on a (small) training set, as described above.

Our optimization procedure then iterates a simple refinement loop over a fixed number of iterations. At each iteration we show an optimizer LLM the current meta-prompt, the human−model gap it achieves on the training set, and the specific examples where it fails, and ask it to propose a revision that would widen the gap “for genuine-quality reasons, not superficial tells”. The meta-prompt is held to a bounded length, so the optimizer must consolidate criteria, rather than adding ever more requirements. After optimization is complete, we keep the best-performing candidate from across the iterations.

Initial Empirical Investigation To show the feasibility of our approach, we conduct 7 iterations of meta-prompt optimization on the previously described paper section writing task with a small set of 52 (paper, section) examples, split into 8 papers for training and 5 held out for validation. Using Muse Spark 1.1 as the rubric generator, writer model, and as the final rubric-based judge, and Kimi-K2.6 as the meta-optimizer, we report results in Figure 13. We find that the initial rubrics have a negative gap, i.e., the model is preferred over expert human writing, but over meta-optimization iterations, the gap becomes positive at iteration 4, and then rises slightly over further iterations. The optimization both increases the judgment scores of human writing (3.4→5.0), and decreases the judgment scores of model writing (7.6→2.7). Figure 12 (right) shows that the learned rubrics fix the failure of the pairwise judge and standard rubrics described in subsection 4.2.

Analysis of the Learned Rubrics The initial rubrics, before learning, tend to cover the paper’s specific content—naming the concrete claims, methods, and results a section must convey—so they yield long, paper-specific coverage checklists. We observe this has the unfortunate effect of rewarding a “mini-paper” within each written section. Such criteria reward fluent imitation, which restates material from across the whole paper, while penalizing the selective human original for the detail it deliberately omits. Over the iterations the optimizer rewrites the guidance that mis-scores expert prose: docking the human for principled omissions, crediting a compressed summary as fluency, and mistaking surface polish for craft. The learned rubrics thus also tend to reward sectional ownership (i.e., not a miniature of the whole paper), disciplined selection, economy, and precise on-scope detail rather than breadth of coverage. Because the meta-prompt is length-bounded, the criteria settle into a small, stable, relatively paper-independent set that tends to sharpen what it rewards instead of accumulating requirements.

4.4 Main Experiments

4.4.1 Paper Section Writing

We first apply RL-XAR to scientific paper-section writing.

Task. The model is provided the paper with one section removed and must write that section (abstract, introduction, related work, or conclusion) to best fit this paper, basing every claim strictly on the paper’s actual content, with no invented numbers, results, or citations. The task prompt supplies a target word count equal to the original human-written section’s length and asks the model to stay within roughly ±15% of it.

Task data. We collect 561 CS papers from the S2ORC corpus (Lo et al., 2020), holding out each of their four sections (abstract, introduction, related work, conclusion) in turn, to form 2,244 (paper, section) training examples; each example pairs the rest of the paper as context with the held-out human section as the expert reference. A disjoint set of 90 papers (360 examples) is held out for validation. For rubric optimization, we use the same small set as described in subsubsection 4.3.1. Further details are provided in subsection G.1.

Training. We reinforcement-learn Qwen3.5-27B with prime-rl (Intellect, 2025) (GRPO, group size 16, learning rate 3×10−6, completions up to 16k tokens), rewarding each generation by the learned rubric as scored a Qwen3.8-2.4T-A95B judge. For rubric meta-optimization we use Kimi-K2.6.

RL-XAR loop. Following our method described in subsection 4.3, we run three outer iterations of (i) meta-optimizing a rubric to maximize the expert-human−model gap against the current model, where we (ii) reinforcement-learn the model against the set of so-far collected rubrics. We train two iterations which re-target the failure modes compared to human writing that the previous round’s model still exhibits on the first two rounds of rubrics, and use the third round rubrics to assess the model without the bias of retraining.

Rubric Evaluation. A model trained on one rubric can learn to satisfy that rubric alone, so we evaluate every writer against all three sets of meta-optimized rubrics and rank by the minimum of the three human-normalized scores—the worst rubric, allowing no cherry-picking. We use GPT-5.6 as a rubric judge for evaluation, so we do not evaluate with the same judge used for training. Results are given in Table 8. RL-XAR carries our 27B model to the top of this leaderboard among all writers: its strongest checkpoint reaches a worst-rubric score of 9.60 (human =10), ahead of every external frontier model. While these rubrics are still potentially biased towards our model (having trained on the first two iterations) and still do not capture a fully accurate measurement of expert human writing, they still reflect performance better than previous approaches (cf. subsection 4.2).

Expert Human Evaluation We confirm these automatic metric gains in a small preliminary study by evaluating on the authors’ own papers, on which they are clearly experts. In a blind expert side-by-side comparison of the RL-XAR model against the baseline Qwen3.5-27B on held-out sections, the RL-XAR writer was preferred by 16 to 2 (an 89% win rate), showing that the rubric-measured improvement corresponds to writing that expert readers may judge better. We found that RL-XAR wins because the baseline generally has weak sectional focus, often creating a “mini-paper”, and simultaneously contains too many unnecessary details, e.g. numerical results when unneeded. Nevertheless, we did not find the RL-XAR generations to be perfect either, and expect better rubric optimization and graders could find more flaws — which could be reinforced during training, i.e. using our approach with frontier models, rather than Qwen3.5-27B as a writer and Qwen3.8-2.4T-A95B as a judge.

4.4.2 Story Writing

Task. The model is shown a story up to a point where it stops mid-scene and must continue from exactly where it leaves off—matching the established voice, prose style, point of view, characters, setting, and tone—without summarizing, restarting, adding a title, or shifting genre. There is a provided length target (the human continuation’s length, within ∼±15%).

Task data. We select books that are public-domain titles from Project Gutenberg. For meta-optimization papers are quality-selected by an external human credential (e.g., Pulitzer- and Nobel-prize-winning authors) and biased toward each author’s lesser-read works to limit memorization. Further, for all papers any continuation where a temperature-0 probe reproduces more than 5% of the reference’s 13-grams, or a single contiguous verbatim run of ≥20 words is present, the example is dropped from the data. We use 2,290 examples for training, 310 for validation. Each pairing consists of the story-so-far as context with the author’s verbatim continuation as the expert reference. We further use a disjoint set of 52 examples for rubric meta-optimization with 20 held out for validation, and 60 for final testing.

Training. We use the same recipe as subsubsection 4.4.1—GRPO on Qwen3.5-27B using a Qwen3.8-2.4T-A95B judge. We only ran a single iteration of RL-XAR in this case. We used rubrics generated from a meta-prompt optimized via Kimi-K2.6 with a Muse Spark 1.1 meta-prompt rubric generator, writer model, and judge. The resulting blind meta-rubric optimization trajectory is shown in fig. 18.

Rubric Evaluation. We evaluate under the same story rubrics, but switch the judge to GPT-5.6, and report human-normalized scores. Results are given in Table 9, left. RL-XAR lifts Qwen3.5-27B from 2.8 to 8.2—the top of all writers, ahead of every frontier model (next best Claude 5 Opus, 6.8).

Human Evaluation We confirm these automatic metric gains by human evaluation in a small preliminary study by the authors. In a blind side-by-side comparison of the RL-XAR model against the baseline Qwen3.5-27B on held-out story continuations, the RL-XAR writer was preferred by 19 to 1 (a 95% win rate), showing that the rubric-measured improvement corresponds to writing that humans may judge better. We found the baseline to be poor at following the style of the story. Further, one noticeable feature is that it often falls back to clichéd or poor-quality similes, e.g. “the air hung over the table like a warm cloth” and “the wind rustled the leaves, sounding like a whisper of warning, or perhaps a laugh” that do not match the author’s work.

4.4.3 Authoring Wikipedia Articles

Task. The model is given only the article title and the target section heading and must write that body section from scratch to a specified target length, similar to the other two tasks.

Task data. We select Wikipedia articles that carry a Featured- or Good-Article quality tier, and are again biased toward less-popular titles, and pass the same memorization filters as mentioned previously. We provide 1,187 training and 162 validation examples. We further use a disjoint set of 42 examples for rubric meta-optimization with 12 held out for validation, and 60 for final testing.

Training. We use a recipe identical to subsubsection 4.4.2, optimizing the rubrics for this setting. The blind meta-rubric optimization trajectory is shown in fig. 18.

Rubric Evaluation. We report results using the learned Wikipedia rubrics, but using a separate GPT-5.6 judge. Results are given in Table 9, right. Here, we find RL-XAR moves Qwen3.5-27B only from 2.7 to 4.0: the gains are small and the model stays well below both the human and the frontier writers (GPT-5.6 8.9, Claude 5.5 Opus 8.2). Wikipedia section writing is the hardest of the three domains for this recipe, in part due to the inadequacy of the Qwen3.8-2.4T-A95B judge, which is analyzed in subsubsection 4.4.4.

4.4.4 Detailed Analysis, Ablations & Additional Experiments

The Judge matters. Our results from subsection 4.2 and subsubsection 4.3.1 show the rubrics matter. Here, we show the quality of the LLM judge evaluating those rubrics is just as important. A weak judge simply cannot differentiate between expert human and model writing—as fig. 14 shows across all three writing domains.

The Meta-Optimizer matters. We compare Muse Spark 1.1, Opus 5 and Kimi K2.6 as rubric meta-optimizers in Figure 15, each averaged over 3 seeds, reporting the human-model validation gap of Muse Spark 1.1 as a paper section writer. We find that Kimi and Opus can find a positive gap, but Muse Spark 1.1 struggles. Likely, weaker models struggle even more. Finally, we note we used a simple iterative refinement meta-optimization method, when many other methods exist (Agrawal et al., 2026; Khattab et al., 2023). We conducted preliminary experiments on a GEPA optimization variant, but it did not yield superior results to our main reported results, and we leave such investigations for future work.

Expert humans matter. Using examples of expert human writing matters for rubric optimization. We validate this by using papers judged to be low-, medium- and high-quality human writing, and evaluating them against our learned rubrics, see Figure 16. Comparing humans to GPT-5.6 we observe a gap of +1.39 for high-quality human writers, +0.95 for medium-quality and only +0.37 for low-quality. Hence, using low-quality writing against a strong model writer would likely be unable to find useful rubrics or a human-model gap. Additionally, if the learned rubric measures genuine quality, the human-model gap should be larger for better-written human papers and smaller for weak ones, which is what we observe.

The need for iterative rubrics. For paper section writing, we note that in the first iteration of RL-XAR training an evaluation of (11.0, 7.0, 8.4) is obtained across the three rounds of rubrics. Round 1 rubrics are optimized, giving a high score, but at the cost of round 2 rubrics, which are not optimized — a score of 7.0 being far below the starting point 10.4 of Qwen3.5-27B. Iteration 2 training restores the performance of round 2 rubrics to baseline model performance, while scoring highly across the other two sets of rubrics. We believe that a fully online approach would improve this overall training recipe, which we leave for future work.

Rubrics generalize across strong Judges. If two different judges are both strong enough, we observe that one can optimize rubrics for one judge, and we see those rubrics work for a different judge. For example, we show this in Table 10 for GPT-5.6 and Muse Spark 1.1 judges.

Blind vs. ground-truth rubrics. When generating rubrics from the rubric meta-prompt, it is possible to condition on only the context (called “blind” rubrics), or on the context and human continuation as well (called “Ground Truth” (GT) rubrics). The latter would likely only make sense when the human continuations are high quality, as the rubrics would be based on them as well. We compare their scoring ability in Table 10. GT rubrics give higher human-model gaps across two different rubric judges, as expected. Further study is required to understand the difference between using the two variants.

Length constraints and rubrics. In preliminary experiments we found that standard models, given the task of paper section writing without a specific instructed length constraint, tend to write overlong sections — as much as two or three times as long as the hidden original human sections. This tends to allow them to score more highly on the given set of rubrics, similar to other instruction-following settings (Yuan et al., 2025). We therefore chose to define length-constrained tasks instead, i.e., to write a section similar to the original human length. Strong models are capable of this, although we observed an unwanted side effect that they tend to reserve a lot of chain-of-thought thinking towards this constraint (e.g., word counting).

Verifiable code-based rewards for surface-form distributional shifts. Our standard RL-XAR reward is an LLM judge scoring an expert-aligned rubric, and it is good at what a rubric is good at: writing quality, coverage, and content. However, with current models it appears to be much less reliable at scoring some verifiable, measurable properties of the output distribution—e.g., how closely a writer matches the human distribution of surface form (sentence length and its variation, paragraphing, lexical diversity, punctuation rates, dialogue), and so on. These are properties one can compute exactly from the text with a small piece of code; we therefore complement the rubric judge with a code-based reward. This code reward is still expert human-anchored, like our meta-optimized rubrics. We score each constructed feature against this item’s human reference, and the model is rewarded for resembling the human distribution. We can hence also meta-optimize the code reward itself—searching for its feature set and overall code the same way as before by maximizing the human-model gap. Here, we run a small study with a fixed reward multiplied into the RL-XAR reward, with results shown in Table 11. They show that this approach can fix some distributional shifts that might otherwise occur during RL, without hurting the standard rubric rewards. More details are given in subsection G.4.

Training rubrics online. We also briefly experiment with a fully online (co-training) objective where the writing generator is optimizing for the rubrics, and the rubrics are simultaneously being trained by GRPO (not meta-optimization) to optimize the human-model gap for the current policy—for the latter, generated rubrics are rewarded by the normalized gap, see Figure 17. While we leave subsequent experiments for future work, our first results in this direction on the story task are positive, with a gain from 2.8 (Qwen3.5-27B) to 6.0 (online RL-XAR) when testing with the same fixed (learned) rubric applied in our earlier experiments.

4.5 Related Work

RL for verifiable vs. non-verifiable tasks. Reinforcement learning has driven much of the recent progress on tasks with checkable answers—mathematics, code, and reasoning—where a programmatic verifier supplies a reliable reward (Shao et al., 2024; DeepSeek-AI et al., 2025). Open-ended writing admits no such verifier, and models optimized without one tend to produce fluent but low-quality text, a failure increasingly recognized as “AI slop” (Shaib et al., 2025; Paech et al., 2026). Our goal is to bring the reliability of verifiable-reward RL to this non-verifiable regime by learning a reward that recognizes expert writing.

RLHF and reward modeling. Learning rewards from human preference comparisons (Christiano et al., 2017; Stiennon et al., 2020) underpins modern instruction tuning (Ouyang et al., 2022; Bai et al., 2022a; Rafailov et al., 2024b). Such rewards are bounded by the raters that produce them: preferences from non-expert annotators cannot certify expert—let alone superhuman—quality, and optimizing an imperfect reward invites over-optimization and reward hacking (Gao et al., 2022; Rafailov et al., 2024a), of which verbosity and length bias are canonical cases (Dubois et al., 2024; Yuan et al., 2025). RL-XAR targets exactly this gap: rather than collect more preferences, it derives the reward from existing expert texts.

AI feedback and LLM judges. A growing body of work replaces or augments human labels with model-generated feedback—constitutional and AI-feedback training (Bai et al., 2022b), self-rewarding models that judge their own outputs (Yuan et al., 2024), and LLM-as-a-judge evaluation (Zheng et al., 2023; Glaese et al., 2022). We find that off-the-shelf judges, and even generated rubrics, systematically under-credit expert human writing relative to fluent model prose—a signal that would teach a policy to write worse. Our contribution is a procedure that repairs this misalignment without reference access at evaluation time.

Rubrics, checklists, and criteria-based evaluation. Decomposing quality into explicit criteria has a long history in NLP evaluation (Ribeiro et al., 2020), and rubric-conditioned grading is increasingly used as a reward for open-ended generation. Unlike a fixed, hand-written rubric or a “mentions-X” checklist that only the reference can satisfy, we generate rubrics from the surrounding context and meta-learn the generator so its rubrics recover the discrimination that reference access would otherwise provide.

Prompt and meta-prompt optimization. Automatically optimizing prompts (Zhou et al., 2022), including reflective and evolutionary search over prompt programs (Agrawal et al., 2026; Khattab et al., 2023), has become a practical alternative to gradient-based adaptation. We apply meta-optimization not to a task prompt but to the rubric-generation prompt, driving it to maximize the expert-human−model score gap; this adversarial objective—an evaluator improved against a generator—is reminiscent of GANs (Goodfellow et al., 2014). Other relevant work with evolving rewards includes the work of Kim et al. (2025).

Expert signals without expert annotators. Eliciting strong capabilities from weak or cheap supervision, and asking whether such oversight scales, is central to training superhuman systems (Bowman et al., 2022; Burns et al., 2023), and evaluating open-ended and creative writing is itself notoriously hard (Karpinska et al., 2021; Chung et al., 2025). RL-XAR sidesteps new expert annotation by anchoring the reward to abundant, externally credentialed expert texts—published papers, prize-winning fiction, and peer-reviewed encyclopedia articles—and learning to score them above current models.

4.6 Conclusion

We introduced RL-XAR, Reinforcement Learning from eXpert-Aligned Rubrics, a recipe that turns readily available expert human writing into a learned, expert-aligned reward and then optimizes against it. Its key step meta-optimizes a rubric generator to maximize the gap between expert human text and current model generations, iterating until no discernible gap remains; standard RL against the resulting rubrics then improves writing quality. Across three domains—writing academic paper sections, Pulitzer- and Nobel-grade story continuation, and Wikipedia section writing—RL-XAR yields large gains over the base model. Our hope is that if AIs can be better communicators with humans, we can all benefit.

Several limitations in our experiments point to future work. Our learned rubrics, trained in part against our own model, may remain biased toward it and do not yet fully capture expert quality. The writer (Qwen3.5-27B) and reward judge (Qwen3.8-2.4T-A95B) used during training are mid-sized, and we expect the recipe to yield even better results when applied with frontier writers and graders. More broadly, RL-XAR offers a template for reward design in non-verifiable domains beyond the writing tasks used here: anchor the ground truth outside the systems under test, learn the criteria rather than hand-writing them, and iterate the evaluator against the generator until the human expert−model gap closes.

References

• Abadi et al. [2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. TensorFlow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16), pages 265–283, 2016.
• Abdulhai et al. [2026] Marwa Abdulhai, Ryan Cheng, Donovan Clay, Tim Althoff, Sergey Levine, and Natasha Jaques. Consistently simulating human personas with multi-turn reinforcement learning. Advances in Neural Information Processing Systems, 38:52920–52957, 2026.
• Agrawal et al. [2026] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations, volume 2026, pages 8479–8565, 2026.
• Amodei et al. [2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. URL https://arxiv.org/abs/1606.06565.
• Anthropic [2026] Anthropic. When AI builds itself. https://www.anthropic.com/institute/recursive-self-improvement, June 2026. Accessed: 2026-06-04.
• Badirli et al. [2020] Sarkhan Badirli, Xuanqing Liu, Zhengming Xing, Avradeep Bhowmik, Khoa Doan, and Sathiya S. Keerthi. Gradient boosting neural networks: GrowNet. arXiv preprint arXiv:2002.07971, 2020.
• Bai et al. [2022a] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. URL https://arxiv.org/abs/2204.05862.
• Bai et al. [2022b] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. URL https://arxiv.org/abs/2212.08073.
• Baker et al. [2025] Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926, 2025.
• Barres et al. [2025] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ2-Bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025.
• Baumann et al. [2026] Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, and Sanmi Koyejo. Swe-chat: Coding agent interactions from real users in the wild. arXiv preprint arXiv:2604.20779, 2026.
• Bean et al. [2026] Andrew M Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan Eghlidi, Chris Schmitz, Karolina Korgul, Hunar Batra, et al. Measuring what matters: Construct validity in large language model benchmarks. Advances in Neural Information Processing Systems, 38, 2026.
• Bowman et al. [2022] Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022.
• Burns et al. [2023] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023.
• Chan et al. [2025] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, volume 2025, pages 50466–50494, 2025.
• Chang et al. [2024] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45, 2024.
• Cheng et al. [2023] Myra Cheng, Tiziano Piccardi, and Diyi Yang. Compost: Characterizing and evaluating caricature in llm simulations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10853–10875, 2023.
• Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
• Chung et al. [2025] John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele, Yuqian Sun, and Max Kreminski. Modifying large language model post-training for diverse creative writing, 2025. URL https://arxiv.org/abs/2503.17126.
• Clinical Trials Transformation Initiative [2026] Clinical Trials Transformation Initiative. AACT: Aggregate analysis of ClinicalTrials.gov. https://aact.ctti-clinicaltrials.org, 2026. Accessed October 28, 2026.
• Cully et al. [2015] Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):503–507, 2015.
• Cunningham et al. [2026] Tom Cunningham, Manish Shetty, Vincent Cheng, and Nate Rush. Expenditure horizon: Measuring optimization ability, with an application to NanoGPT. https://metr.org/blog/2026-07-21-expenditure-horizon/, July 2026. METR Blog. Accessed: 2026-09-02.
• DeepSeek-AI et al. [2025] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948.
• Denison et al. [2024] Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward tampering in language models. arXiv preprint arXiv:2406.10162, 2024.
• Deshpande et al. [2026] Darshan Deshpande, Anand Kannappan, and Rebecca Qian. Benchmarking reward hack detection in code environments via contrastive analysis. arXiv preprint arXiv:2601.20103, 2026.
• Dubois et al. [2024] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
• Eisenstein et al. [2024] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D'Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. In colm, 2024. URL https://openreview.net/forum?id=5u1GpUkKtG.
• eLife Sciences Publications [2026] eLife Sciences Publications. eLife decision letters and author responses. https://elifesciences.org, 2026. Accessed October 28, 2026.
• Falcon and The PyTorch Lightning team [2019] William Falcon and The PyTorch Lightning team. PyTorch Lightning. https://github.com/Lightning-AI/pytorch-lightning, 2019.
• Fernando et al. [2023] Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023.
• Gao et al. [2022] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760, 2022. URL https://arxiv.org/abs/2210.10760.
• Garikaparthi et al. [2026] Aniketh Garikaparthi, Manasi Patwardhan, and Arman Cohan. Researchgym: Evaluating language model agents on real-world ai research, 2026. URL https://arxiv.org/abs/2602.15112.
• Ge et al. [2024] Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094, 2024.
• Glaese et al. [2022] Amelia Glaese, Nat McAleese, Maja Trbacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. URL https://arxiv.org/abs/2209.14375.
• Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), volume 27, 2014.
• Greenblatt et al. [2024] Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI control: Improving safety despite intentional subversion. In International Conference on Machine Learning (ICML), 2024.
• Grigorik [2026] Ilya Grigorik. GH Archive. https://www.gharchive.org, 2026. Accessed October 28, 2026.
• Gugger et al. [2022] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, and Sourab Mangrulkar. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate, 2022.
• Guo et al. [2024] Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. arXiv preprint arXiv:2402.17453, 2024.
• Harbor Framework Team [2026] Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026. URL https://doi.org/10.5281/zenodo.20953922.
• Herrera-Perez et al. [2019] Diana Herrera-Perez, Alyson Haslam, Tyler Crain, Jennifer Gill, Catherine Livingston, Victoria Kaestner, Michael Hayes, Dan Morgan, Adam S. Cifu, and Vinay Prasad. A comprehensive review of randomized clinical trials in three medical journals reveals 396 medical reversals. eLife, 8:e45183, 2019. 10.7554/eLife.45183.
• Hong et al. [2025] Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, et al. Data interpreter: An llm agent for data science. In Findings of the association for computational linguistics: ACL 2025, pages 19796–19821, 2025.
• Howard [2019] Jeremy Howard. Imagenette: A smaller subset of 10 easily classified classes from ImageNet. https://github.com/fastai/imagenette, 2019.
• Hsieh et al. [2024] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024.
• Huang et al. [2024] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. In International Conference on Machine Learning (ICML), 2024.
• Huang et al. [2026] Yue Huang, Zhangchen Xu, Yuchen Ma, Wenjie Wang, Zheyuan Liu, Ziwei Xu, Pin-Yu Chen, Michel Galley, Zinan Lin, Stefan Feuerriegel, Radha Poovendran, Misha Sra, Alex Pentland, Xiangliang Zhang, and Zichen Chen. Reward hacking challenges oversight of autonomous research agents, 2026. URL https://arxiv.org/abs/2609.28614.
• Hubinger et al. [2024] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv preprint arXiv:2401.05566, 2024.
• Intellect [2025] Prime Intellect. prime-rl, 2025. URL https://github.com/PrimeIntellect-ai/prime-rl.
• Jain et al. [2025] Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents. arXiv preprint arXiv:2504.07164, 2025.
• Jiang et al. [2025] Yuxuan Jiang, Ziming Zhou, Boyu Xu, Beijie Liu, Runhui Xu, and Peng Huang. Training with confidence: Catching silent errors in deep learning training with automated proactive checks. In Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’25), Boston, MA, USA, July 2025. USENIX Association.
• Jimenez et al. [2024] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024.
• Jin et al. [2025] Chuanyang Jin, Jing Xu, Bo Liu, Leitian Tao, Olga Golovneva, Tianmin Shu, Wenting Zhao, Xian Li, and Jason Weston. The era of real-world human interaction: Rl from user conversations. arXiv preprint arXiv:2509.25137, 2025.
• Jin et al. [2026] Chuanyang Jin, Binze Li, Haopeng Xie, Cathy Mengying Fang, Tianjian Li, Shayne Longpre, Hongxiang Gu, Maximillian Chen, and Tianmin Shu. Thoughttrace: Understanding user thoughts in real-world llm interactions. arXiv preprint arXiv:2605.20087, 2026.
• Jordan et al. [2024] Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Boza Vlado, You Jiacheng, Franz Cesista, Braden Koszarsky, and @Grad62304977. modded-nanogpt: Speedrunning the nanogpt baseline, 2024. URL https://github.com/KellerJordan/modded-nanogpt.
• Jotautaitė et al. [2026] Monika Jotautaitė, Maria Angelica Martinez, Ollie Matthews, and Tyler Tracy. Monitoringbench: Semi-automated red-teaming for agent monitoring, 2026. URL https://arxiv.org/abs/2605.09684.
• Karpathy [2022] Andrej Karpathy. nanogpt. https://github.com/karpathy/nanoGPT, 2022.
• Karpathy [2026] Andrej Karpathy. autoresearch: AI agents running research on single-GPU nanochat training automatically. https://github.com/karpathy/autoresearch, 2026. Accessed: 2026-08-11.
• Karpinska et al. [2021] Marzena Karpinska, Nader Akoury, and Mohit Iyyer. The perils of using mechanical turk to evaluate open-ended text generation. In Proc. of EMNLP, 2021. URL https://arxiv.org/abs/2109.06835.
• Kennard et al. [2022] Neha Nayak Kennard, Tim O’Gorman, Rajarshi Das, Akshay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and Andrew McCallum. DISAPERE: A dataset for discourse structure in peer review discussions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1234–1249, Seattle, United States, July 2022. Association for Computational Linguistics. 10.18653/v1/2022.naacl-main.89.
• Khattab et al. [2023] Omar Khattab et al. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023.
• Kim et al. [2025] Zae Myung Kim, Chanwoo Park, Vipul Raheja, Suin Kim, and Dongyeop Kang. Toward evaluative thinking: Meta policy optimization with evolving reward models. arXiv preprint arXiv:2504.20157, 2025.
• Kimi Team [2025] Kimi Team. Kimi k2: Open agentic intelligence. Technical report, Moonshot AI, July 2025. URL https://moonshotai.github.io/Kimi-K2/. Introduces Kimi K2, a 1 trillion-parameter Mixture-of-Experts LLM trained with MuonClip and designed for agentic intelligence.
• Klein et al. [2017] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72, 2017.
• Krakovna et al. [2020] Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity. DeepMind Blog, 2020.
• Kulikov et al. [2026] Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, et al. Autodata: An agentic data scientist to create high quality synthetic data. arXiv preprint arXiv:2606.25996, 2026.
• Kutasov et al. [2025] Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun van der Weij, Linda Petrini, Chen Bo Calvin Zhang, John Hughes, Xiang Deng, Henry Sleight, Tyler Tracy, Buck Shlegeris, and Joe Benton. Shade-arena: Evaluating sabotage and monitoring in llm agents, 2025. URL https://arxiv.org/abs/2506.15740.
• Kwon et al. [2023] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23), 2023.
• Laban et al. [2026] Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Llms get lost in multi-turn conversation. In International Conference on Learning Representations, volume 2026, pages 54738–54778, 2026.
• LI et al. [2025] XIANG LI, Farzaan Kaiyom, Evan Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. Autobencher: Towards declarative benchmark construction. In International Conference on Learning Representations, volume 2025, pages 94357–94386, 2025.
• Li et al. [2023] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023.
• Liang et al. [2022] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022.
• Libon et al. [2026] Lena Libon, Ben Rank, Jehyeok Yeon, David Schmotz, Jeremy Qin, Daniel Donnelly, Derck Prinzhorn, and Maksym Andriushchenko. Researcharena: Evaluating sabotage and monitoring in automated ai r&d, 2026. URL https://arxiv.org/abs/2607.19321.
• Lightman et al. [2024] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step. In International Conference on Learning Representations (iclr), 2024. URL https://openreview.net/forum?id=v8L0pN6EOi.
• Lightning AI [2023] Lightning AI. LitGPT. https://github.com/Lightning-AI/litgpt, 2023.
• Liu et al. [2025] Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510.24684, 2025.
• Liu et al. [2026] Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, et al. Spade: Self-play in adaptive synthetic executable environments. arXiv preprint arXiv:2608.19197, 2026.
• Liu et al. [2023] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023. URL https://arxiv.org/abs/2307.03172.
• Lo et al. [2020] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, 2020.
• Lu et al. [2024] Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
• Lupidi et al. [2024] Alisia Lupidi, Carlos Gemmell, Nicola Cancedda, Jane Dwivedi-Yu, Jason Weston, Jakob Foerster, Roberta Raileanu, and Maria Lomeli. Source2synth: Synthetic data generation and curation grounded in real data sources. arXiv preprint arXiv:2409.08239, 2024.
• Lyu et al. [2026] Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, Kaicheng Yang, Zitao Chen, Wentao Guo, Junlin Yang, Xinyue Ai, Wenhao Chai, Yadi Cao, Ziran Yang, Kun Wang, Dapeng Jiang, Huan ang Gao, Shange Tang, Chengshuai Shi, Simon S. Du, Max Simchowitz, Jiantao Jiao, Dawn Song, and Chi Jin. Mls-bench: A holistic and rigorous assessment of ai systems on building better ai, 2026. URL https://arxiv.org/abs/2605.08678.
• Madaan et al. [2023] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, pages 46534–46594, 2023. URL https://arxiv.org/abs/2303.17651.
• Mangalam et al. [2021] Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths to long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15233–15242, 2021. 10.1109/ICCV48922.2021.01495.
• Manheim and Garrabrant [2018] David Manheim and Scott Garrabrant. Categorizing variants of goodhart’s law. arXiv preprint arXiv:1803.04585, 2018. URL https://arxiv.org/abs/1803.04585.
• Martin and Roger [2026] Sam Martin and Fabien Roger. Classifier context rot: Monitor performance degrades with context length, 2026. URL https://arxiv.org/abs/2605.12366.
• McQuarrie et al. [2023] Shane A. McQuarrie, Karen E. Willcox, and OpInf Contributors. opinf: Operator inference for data-driven, non-intrusive model reduction of dynamical systems. https://github.com/Willcox-Research-Group/rom-operator-inference-Python3, 2023. Methodology introduced in Peherstorfer and Willcox [2016].
• Meinke et al. [2024] Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming. arXiv preprint arXiv:2412.04984, 2024.
• Mirzadeh et al. [2020] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. In Advances in Neural Information Processing Systems, volume 33, pages 7308–7320, 2020.
• Mitra et al. [2024] Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. Agentinstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03502, 2024.
• Moskovitz et al. [2023] Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04373, 2023.
• Mouret and Clune [2015] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015.
• MRpro Team [2024] MRpro Team. MRpro: PyTorch-based MR image reconstruction and processing package. https://github.com/PTB-MR/mrpro, 2024.
• Naous et al. [2026] Tarek Naous, Philippe Laban, Wei Xu, and Jennifer Neville. Flipping the dialogue: Training and evaluating user language models. In International Conference on Learning Representations, volume 2026, pages 82079–82103, 2026.
• Nature Communications [2026] Nature Communications. Transparent peer review at Nature Communications. https://www.nature.com/ncomms/, 2026. Accessed October 28, 2026.
• Olmo et al. [2025] Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3. arXiv preprint arXiv:2512.13961, 2025.
• Open Science Collaboration [2015] Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349(6251):aac4716, 2015. 10.1126/science.aac4716.
• OpenAI [2026a] OpenAI. GPT-5.5 system card, April 2026a. URL https://deploymentsafety.openai.com/gpt-5-5. Published April 23, 2026; updated April 24, 2026.
• OpenAI [2026b] OpenAI. Openai and hugging face partner to address security incident during model evaluation. OpenAI Blog, July 2026b. URL https://openai.com/index/hugging-face-model-evaluation-security-incident/. Preliminary findings: GPT-5.6 Sol and pre-release model escaped ExploitGym evaluation via zero-day in package registry proxy, chained attack vectors for RCE on Hugging Face. Updated July 28-29, 2026.
• OpenReview [2026] OpenReview. OpenReview. https://openreview.net, 2026. ICLR 2024 review threads; accessed October 28, 2026.
• Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
• Paech et al. [2026] Sam Paech, Allen Roush, Judah Goldfeder, and Ravid Shwartz-Ziv. Antislop: A comprehensive framework for identifying and eliminating repetitive patterns in language models. In International Conference on Learning Representations, volume 2026, pages 42490–42525, 2026.
• Pan et al. [2022] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations (ICLR), 2022.
• Pan et al. [2024] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024.
• Park et al. [2023] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023.
• Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, NeurIPS, pages 8024–8035, 2019.
• Peherstorfer and Willcox [2016] Benjamin Peherstorfer and Karen Willcox. Data-driven operator inference for nonintrusive projection-based model reduction. Computer Methods in Applied Mechanics and Engineering, 306:196–215, 2016.
• PostTrainBench Team [2026] PostTrainBench Team. PostTrainBench v1.1: Hardening the Benchmark Against Reward Hacking. https://posttrainbench.com/blog/posttrainbench-1-1/, July 2026. Accessed: 2026-09-02.
• Qi et al. [2026] Richard Qi, Benjamin Wright, Monte MacDiarmid, and Evan Hubinger. Training a misaligned reward seeker. https://alignment.anthropic.com/2026/reward-seeker/, August 2026. Anthropic Alignment Science Blog. Accessed: 2026-09-02.
• Qian et al. [2025a] Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, et al. Userbench: An interactive gym environment for user-centric agents. arXiv preprint arXiv:2507.22034, 2025a.
• Qian et al. [2025b] Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, et al. Userrl: Training interactive user-centric agent via reinforcement learning. arXiv preprint arXiv:2509.19736, 2025b.
• Rafailov et al. [2024a] Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms, 2024a. URL https://arxiv.org/abs/2406.02900.
• Rafailov et al. [2024b] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024b. URL https://arxiv.org/abs/2305.18290.
• Rasley et al. [2020] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020.
• Reuel et al. [2024] Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J Kochenderfer. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. Advances in Neural Information Processing Systems, 37:21763–21813, 2024.
• Ribeiro et al. [2020] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4902–4912, 2020. URL https://aclanthology.org/2020.acl-main.442/.
• Schatzmann et al. [2007] Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 149–152, 2007.
• Schick et al. [2023] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. ToolFormer: language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. URL https://arxiv.org/abs/2302.04761.
• Shaib et al. [2025] Chantal Shaib, Tuhin Chakrabarty, Diego Garcia-Olano, and Byron C Wallace. Measuring AI “slop” in text. arXiv preprint arXiv:2509.19163, 2025.
• Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
• Singh et al. [2026] Shivalika Singh, Yiyang Nan, Alex Wang, Daniel Dsouza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, et al. The leaderboard illusion. Advances in neural information processing systems, 38, 2026.
• Skalse et al. [2022] Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
• Starace et al. [2025] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. PaperBench: Evaluating AI’s ability to replicate AI research. arXiv preprint arXiv:2504.01848, 2025.
• Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf.
• Tack et al. [2026] Jihoon Tack, Philippe Laban, and Jennifer Neville. Llms get lost in evolving user intent. arXiv preprint arXiv:2607.20734, 2026.
• The Mosaic ML Team [2021] The Mosaic ML Team. Composer: A PyTorch library for efficient neural network training. https://github.com/mosaicml/composer, 2021.
• TorchVision maintainers and contributors [2016] TorchVision maintainers and contributors. torchvision: Datasets, transforms and models specific to computer vision. https://github.com/pytorch/vision, 2016.
• Turpin et al. [2023] Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388, 2023.
• U.S. National Library of Medicine [2026] U.S. National Library of Medicine. ClinicalTrials.gov. https://clinicaltrials.gov, 2026. API v2; accessed October 28, 2026.
• Wang et al. [2024] Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. In International Conference on Learning Representations, volume 2024, pages 32593–32627, 2024.
• Wang et al. [2025] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pages 65882–65919, 2025.
• Wang et al. [2023] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 13484–13508, 2023.
• Wang et al. [2026] Zora Zhiruo Wang, John Yang, Kilian Lieret, Alexa R Tartaglini, Valerie Chen, Yuxiang Wei, Zijian Wang, Lingming Zhang, Karthik R Narasimhan, Ludwig Schmidt, et al. Position: Humans are missing from AI coding agent research, 2026.
• Wei et al. [2026] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. Advances in Neural Information Processing Systems, 38:78500–78525, 2026.
• Wen [2026] Jiaxin Wen. Automated weak-to-strong research. https://github.com/safety-research/automated-w2s-research, 2026. Accessed: 2026-08-11.
• Weng [2024] Lilian Weng. Reward hacking in reinforcement learning. lilianweng.github.io, Nov 2024. URL https://lilianweng.github.io/posts/2024-11-28-reward-hacking/.
• Weng [2026] Lilian Weng. Harness engineering for self-improvement. lilianweng.github.io, July 2026. URL https://lilianweng.github.io/posts/2026-07-04-harness/.
• Weston and Foerster [2025] Jason Weston and Jakob Foerster. Ai & human co-improvement for safer co-superintelligence. arXiv preprint arXiv:2512.05356, 2025.
• Wightman [2019] Ross Wightman. PyTorch image models. https://github.com/huggingface/pytorch-image-models, 2019.
• Wijk et al. [2025] Hjalmar Wijk, Tao Roa Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Joshua M Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Jun Koba Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. RE-bench: Evaluating frontier AI r&d capabilities of language model agents against human experts. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 66772–66832. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/wijk25a.html.
• Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Art Natural Language Processing. In Conference on Empirical Methods in Natural Language Processing (EMNLP) - System Demonstrations, 2020. URL https://aclanthology.org/2020.emnlp-demos.6/.
• Wu et al. [2022] Po-Cheng Wu, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. Incorporating peer reviews and rebuttal counter-arguments for meta-review generation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM ’22), pages 2189–2198, 2022.
• Wu et al. [2025] Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. Collabllm: From passive responders to active collaborators. arXiv preprint arXiv:2502.00640, 2025.
• Wu et al. [2026a] Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, and James Zou. Humanlm: Simulating users with state alignment beats response imitation. arXiv preprint arXiv:2603.03303, 2026a.
• Wu et al. [2026b] Yifan Wu, Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, et al. Swe-together: Evaluating coding agents in interactive user sessions. arXiv preprint arXiv:2606.29957, 2026b.
• Wu et al. [2019] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github.com/facebookresearch/detectron2, 2019.
• Xu et al. [2024] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In International Conference on Learning Representations (iclr), 2024. URL https://openreview.net/forum?id=CfXh93NDgH.
• Xu et al. [2025] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. In International Conference on Learning Representations, volume 2025, pages 76346–76382, 2025.
• Xu et al. [2026] Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi Wang, Radha Poovendran, Misha Sra, Alex Pentland, and Zichen Chen. Autolab: Can frontier models solve long-horizon auto research and engineering tasks?, 2026. URL https://arxiv.org/abs/2606.05080.
• Yang et al. [2024a] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, volume 2024, pages 12028–12068, 2024a.
• Yang et al. [2024b] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024b.
• Yang et al. [2026] John Yang, Kilian Lieret, Carlos Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. Advances in Neural Information Processing Systems, 38, 2026.
• Yao et al. [2022] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
• Yao et al. [2024] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.
• Yin et al. [2025] Xunjian Yin, Xinyi Wang, Liangming Pan, Li Lin, Xiaojun Wan, and William Yang Wang. Gödel agent: A self-referential agent framework for recursively self-improvement. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27890–27913, 2025.
• Yu et al. [2025] Ping Yu, Jack Lanchantin, Tianlu Wang, Weizhe Yuan, Olga Golovneva, Ilia Kulikov, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. Cot-self-instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks. arXiv preprint arXiv:2507.23751, 2025.
• Yu et al. [2026] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026.
• Yuan et al. [2024] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024. URL https://arxiv.org/abs/2401.10020.
• Yuan et al. [2025] Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason E Weston, and Jing Xu. Following length constraints in instructions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 24243–24254, 2025.
• Yuan et al. [2026] Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions. Advances in Neural Information Processing Systems, 38, 2026.
• Z.ai [2026] Z.ai. GLM-5.2: Built for long-horizon tasks. https://z.ai/blog/glm-5.2, June 2026. Blog post. Accessed: 2026-08-11.
• Zandieh et al. [2026] Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. TurboQuant: Online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations (ICLR), 2026. arXiv:2504.19874.
• Zelikman et al. [2022] Eric Zelikman, Jesse Mu, Noah D Goodman, and Yuhuai Tony Wu. STar: Self-taught reasoner bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://arxiv.org/abs/2203.14465.
• Zhang et al. [2024] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, et al. LMMs-Eval: Reality check on the evaluation of large multimodal models. arXiv:2407.12772, 2024.
• Zhao et al. [2026] Andrew Zhao, Yiran Wu, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. Advances in Neural Information Processing Systems, 38:105816–105879, 2026.
• Zhao et al. [2025] Weikang Zhao, Xili Wang, Chengdi Ma, Lingbin Kong, Zhaohua Yang, Mingxiang Tuo, Xiaowei Shi, Yitao Zhai, and Xunliang Cai. Mua-rl: Multi-turn user-interacting agent reinforcement learning for agentic tool use. arXiv preprint arXiv:2508.18669, 2025.
• Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2306.05685.
• Zheng et al. [2026] Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, et al. Livecodebench pro: How do olympiad medalists judge llms in competitive programming? Advances in Neural Information Processing Systems, 38, 2026.
• Zhou et al. [2025] Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.15478, 2025.
• Zhou et al. [2026] Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar. Self-challenging language model agents. Advances in Neural Information Processing Systems, 38:113959–113991, 2026.
• Zhou et al. [2022] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022. URL https://arxiv.org/abs/2211.01910.
• Zhu [2024] Zilin Zhu. ring-flash-attention: Ring attention implementation with flash attention. https://github.com/zhuzilin/ring-flash-attention, 2024.

Appendix A AutoBenchmark implementation details

A.1 Stage 1: Benchmark proposal

Stage 1-A. Authoring session. We run one fresh research agent per iteration: an OpenHands agent [Wang et al., 2025] with a terminal, a file editor, a task tracker, and a summarizing condenser, in an isolated container. It writes only to its own iteration directory and reads the brief, the operator grounding, the pinned Harbor documentation [Harbor Framework Team, 2026], and earlier frozen submissions; public unauthenticated web retrieval is available for grounding. It cannot call models, launch judges or solvers, start containers, submit scheduler jobs, or run its own benchmark, and the structural validator below is the only executable check it may invoke. The session ends by writing a one-shot marker, after which we freeze and hash the tree.

Stage 1-B. Submission contract. A submission contains a specification (construct, scientific value, scope, hypothesis, feasibility argument, threats to validity), a comparison of at least three materially different designs, a grounding directory with retained source bytes and a provenance manifest, and a record of the agent’s research process. Each task directory is a complete native Harbor task: instruction, task configuration, assets uploaded into the solver’s working directory, a deterministic reference solution, and a verifier directory holding the grading criteria, a judging context, and a private reference answer. Every manifest record carries an origin, a UTC retrieval time, a local path, a SHA-256, and license terms; self-generated data must additionally retain the executable generator, its parameters and seed, per-stage timestamps, and verified asset hashes. Criteria are native binary, Likert, or numeric items with positive weights and stable identifiers, and regexes, required phrases, length targets, and model-specific criteria are prohibited.

Stage 1-C. Structural validation. We scaffold each task before the session from pinned harbor task init output, with the Dockerfile removed and fixed Reward Kit files installed. The first gate validates the frozen tree without executing it. Protected content must return byte-identical: verifier launcher, answer-judge prompt template, image digests, timeouts, resource and network settings, and the judge wiring that fixes the judge’s inputs, the reference path, and the aggregation; the agent may edit only the task description, keywords, and scientific metadata. Each required task must appear exactly once; every path must be a regular file, so symlinks, hardlinks, device nodes, and sparse files are rejected; the executable bit is allowed only on the reference solution and the protected launcher; and the package must stay within 1,024 files, 2 GiB total, 256 MiB per file, and 240-byte paths. Criteria must satisfy the Reward Kit schema (unique identifiers, positive weights, at least two points for a Likert item, an explicit range for a numeric item), and manifest timestamps, hostnames, and hashes must match the retained bytes. Three further checks apply to revisions: we parse the reference solution in shell no-execute mode and compile its literal Python heredocs; we hold criterion identity fixed, so an identifier cannot be rebound to another name nor a name to another identifier; and we reject an instruction change whose private grading and reference artifacts are byte-identical to the checkpoint, the signature of a partial write.

Stage 1-D. Revision integrity. From the second iteration onward, a dedicated judge runs before execution. It sees the solver-visible projection of the checkpoint and of the current submission, namely instructions, READMEs, public metadata, and assets, together with a trusted diff, the feedback the agent could read, and the current grounding; private references, criteria, solutions, and all hidden solver results are excluded. It applies a counterfactual test to every material public change: with all prior solver answers and scores removed, would an independent author still make this change from the construct, the grounding, or a genuine ambiguity? A change passes only if retained sources support it at an exact locator; making an observed solver error less likely is not a justification. A static detector flags numbers, causal edges, stratum identities, and long n-grams that newly appear in public files and also appear in the feedback, as advisory input. The verdict is binding, and its provenance review covers the whole current grounding, not only records touched by the diff.

Stage 1-E. Reference-solution admission. We execute the agent’s own reference solution in the real task container through Harbor’s Oracle agent and grade it with the real verifier and answer judge. A submission is admitted only with no verifier error and reward at least 0.9, which establishes that the task is solvable under the fixed contract, that the environment holds what the solution needs, and that the grader awards the intended answer. Retained reward artifacts are hashed into the validation report the quality judges later read.

Stage 1-F. Repair. A failure at any of the three gates returns the iteration to a fresh session of the same agent, which receives a repair report naming the failing gate and the defect and is restricted to fixing it. We allow at most ten submission attempts per iteration.

A.2 Stage 2: Benchmark solving

Stage 2-A. Solver runs. Each solver attempts every task once, which we treat as one observation per model and task, using Harbor’s agent in the task container under the fixed execution contract and per-route token caps, five tasks at a time. A completed run retains the final answer, the trajectory in Harbor’s ATIF format, and a result summary. Provider errors, container and scheduler failures, missing trajectories, and truncated collections are classified as infrastructure outcomes and retried, never recorded as incorrect answers; only completed runs enter the difficulty signal, and the feedback states this so the agent does not read a failed run as a solved task. External solvers use a separate lane whose results go to an operator-private directory that no feedback index references, routed through a loopback proxy.

Stage 2-B. Answer judging. Answers are graded in a separate verifier container by a judge that sees the judging context, the answer, and the private reference, and scores each criterion independently; scores combine by a protected weighted mean into a reward in [0,1]. The judge prompt is a harness-owned template that treats the task, the answer, and every quoted asset as untrusted evidence, and we fix the judge route so the agent cannot address the grader. Published feedback carries per-criterion scores and anchors.

A.3 Stage 3: Benchmark review

Stage 3-A. Criteria. We run one judge per criterion. The wording below is canonical and is disclosed verbatim to the research agent, so it knows what its submission is reviewed against.

• (i)

  Construct validity. The benchmark measures the research-agent capability claimed in the brief and the specification, not an incidental proxy or an evaluator quirk.

• (ii)

  Correctness. Task artifacts, private references, rubric semantics, score calculation, and reproducibility are internally correct and free of leakage.

• (iii)

  Feasibility. A genuinely stronger research agent could solve the benchmark with the supplied evidence, resources, and rules; difficulty from missing evidence, impossible requirements, or circular evaluation is rejected.

• (iv)

  Usefulness. Progress would transfer to meaningful scientific work or provide valuable evidence about research-agent capability.

• (v)

  Overall. The integrated contribution has no serious red flags and is a credible publication-quality benchmark design at the current scale.

Stage 3-B. Isolation and verdicts. The five sessions run concurrently on physically isolated bundles holding the brief, the specification, the grounding, the frozen submission, a sanitized validation report (versions, image digests, judge identifier and prompt hash, scoring wiring, reference-solution status and reward), and the retained solver evidence. No judge sees another’s verdict. Each writes a JSON verdict with a pass or fail decision, a responsibility field separating submission defects from harness faults, at least three artifact citations, at least one strength and one weakness, required fixes for a fail, a confidence level, and a rationale.

Stage 3-C. Evidence protocol. A manifest entry is an untrusted assertion. Every material claim or grading anchor must be verified against retained content at an exact locator or a reproducible derivation, and placeholder domains, invented citations, dead locators, self-authored text presented as publisher bytes, and unsupported license or version declarations are mandatory failures. Judges may not browse the web to repair an evidentiary gap, so a claim that cannot be checked inside the bundle counts against the submission.

A.4 Overall loop

Component A. AI Feedback. We publish one curated, secret-scanned bundle per iteration containing the validation report, per-task run status, sanitized trajectories and answers for the in-loop solvers, per-criterion rubric judgments, and the five verdicts. Only files named by the bundle index may be read, and hidden external results stay in the private lane of subsection A.2. Raw provider logs, exception text, and judge reasoning are omitted, and all published evidence is labeled untrusted, so an instruction embedded in an answer or a retrieved document is not a directive to the next session.

Component B. Checkpoint selection. Before iteration t we build a history over iterations 1 to t−1. An iteration is eligible when all five verdicts passed and its solver evaluation is complete; among eligible iterations we take the lowest mean reward on the solver designated as the selection objective, breaking ties toward the later iteration. With nothing eligible we fall back to the latest fully evaluated iteration and label the session a repair. The selected checkpoint is copied atomically into the new writable directory for in-place editing, and the prompt exposes every prior snapshot and feedback index, so the agent can inspect iterations it did not descend from. The end-of-run selection uses the same policy but refuses any checkpoint that did not pass every judge, and is written where the agent cannot see it.

Component C. Human feedback. The operator-in-the-loop setting of subsection 1.3 reuses this machinery. The driver finishes the preceding iteration, then pauses before the next session and writes a review packet with per-task scores, action and tool counts, rubric misses, quality outcomes, and paths to the full trajectories and answers; no model job runs while it waits. We validate and snapshot the admitted response and expose it as one more read-only artifact alongside the checkpoint history, after which the loop resumes.

Appendix B Benchmark examples built with AutoBenchmark

B.1 Graveyard Bench

Graveyard Bench asks whether an idea generating model will re propose a research direction that is already dead. Each task is seeded from on failed research from a null pre-registered clinical trial, a medical reversal, or a results-level retraction. The task gives the solver only a neutral research goal (plus some background), but never the refuted intervention, and asks for ten structured hypotheses, each with a confidence score and a previously_refuted or not_previously_refuted flag. A task counts as a zombie (incorrect) if any hypothesis matches the hidden claim’s intervention, effect direction and scope without being flagged previously_refuted. The difficulty is that a refuted idea leaves no confirming paper behind, so it looks unexplored, and novelty-seeking pushes the solver straight toward it.

注意转载授权 · 阅读原文