研究将 LLM 科研智能体当作统计检验评估其收益可预测性判断

Chang Liu Affiliation: Department of Information Engineering and Computer Science (DISI), University of Trento Affiliation: Via Sommarive 9, 38123 Povo (Trento), Italy Email: chang.liu-1@unitn.it

November 2026

Abstract

Researchers ask large language model (LLM) research agents whether a forecasting model beats a static benchmark, a verdict that real data cannot check. We treat the agent as a statistical test and score pre-specified runs of three frontier models and one open-weight model in simulated worlds with a planted signal. The agents detect weak signals more often than a pre-specified out-of-sample test, mainly because they accept weaker evidence: held to an agent’s false-positive rate, a conventional in-sample test does as well or better. The agents claim predictability too often in worlds without a signal, state miscalibrated probabilities, and change their verdicts after a sentence about prior research; the magnitudes depend on the model. The excess false claims persist in simulated worlds without a resampling artifact. Agent verdicts are informative but need a neutral brief, a decision rule fixed outside the agent and a sealed holdout.

Keywords: forecast evaluation; return predictability; large language models; research agents; multiple testing; Monte Carlo simulation; prompt sensitivity.

1 Introduction

Researchers increasingly hand empirical work to LLM agents that write and run their own code. In forecasting and asset pricing the question put to such an agent is often binary: does a dynamic model beat a static benchmark out of sample? Either answer is hard to check. Factor timing is difficult (Haddad et al., 2020; Asness et al., 2017), and most equity premium predictors fail out of sample (Welch and Goyal, 2008; Goyal et al., 2024), so a skeptical verdict may simply be correct. It may also reflect a weak search, a cautious prior or the wording of the request, and an optimistic verdict may reflect data mining (Lo and MacKinlay, 1990; Harvey et al., 2016; Bailey et al., 2017).

With real data the truth is unknown, so these explanations cannot be separated. We therefore plant the truth. Each run gives the agent a simulated world whose data-generating process we control. Most worlds are linear predictive regressions with population R2 between 0 and 4% per month. The others are trap worlds that look predictable but are not, worlds whose signal lies outside the offered candidates, and real factor returns with an injected signal. Because we know the truth, we can score each verdict and study the agent like any other test. We measure its size (how often it claims predictability when there is none), its power (how often it detects a signal of given strength) and the calibration of its stated probabilities.

We report four findings. First, Claude Opus 5.5, the agent of the first study, detects weak signals more often than a conventional out-of-sample test, mainly because it accepts weaker evidence. Held to the same false-positive rate, an in-sample test with Newey–West heteroskedasticity- and autocorrelation-consistent (HAC) standard errors and a Holm adjustment across candidates detects weak signals about as often, and a forecaster who knows the true model detects more. Second, the agent is a liberal test: in worlds without a signal it claims predictability more often than the nominal 5%. Its stated probabilities are poorly calibrated, and a sentence about prior research appended to the brief shifts its verdict on identical data.

Third, a second study with two models from another developer, GPT-6.1 Sol and GPT-6.1 Terra, replicates all three failures, and its cross-model tests confirm the size and framing results. The magnitudes differ. Terra claims predictability in 63% of worlds without a signal, Sol in 19% and Opus in 21%. At Terra’s false-positive rate, the conventional in-sample test detects more weak signals than Terra does. Sol’s shortfall against the test is not statistically distinguishable from zero. Opus’s parity with the test is the best case among the three models. Fourth, the simulated worlds of the first two studies contain an artifact. Their bootstrap-resampled return shocks repeat exactly, so a lookup of past returns can forecast even a world without a signal. Terra exploits the artifact, and without the runs that exploit it, its false-claim rate is 46%. A third study with Gaussian shocks removes the artifact and re-measures the false-claim rate of all three models. Every model still claims predictability too often (Opus in 34%, Sol in 16% and Terra in 44% of no-signal worlds), and each claims more often than the HAC-Holm test on the same worlds. A fourth study runs Qwen3-32B, an open-weight model that runs on one workstation GPU, on the same worlds. It claims predictability in 35% of no-signal worlds, and its claim rate barely rises with the planted signal.

For forecasters, an agent’s verdict is useful evidence, but the agent is not a test of known size. Because its search at best matches a conventional test, its value lies in automating that search. A neutral brief, a decision rule fixed in advance and a holdout outside the agent’s reach supply the properties that the agent alone does not guarantee. On real data, Claude Opus 5.5 recalls part of published factor returns, so agent conclusions should be evaluated on data the model cannot have seen.

Our design relates to four strands of work. The first examines LLM agents in empirical finance. Qu et al. (2026) plant true factors among candidates, let LLM agents propose factors and leave the judgment to a fixed anytime-valid referee. Gençay (2026) subjects LLM-discovered trading strategies to leakage-safe, search-aware evaluation without planted truth, and Pan et al. (2026) backtest the agentic discovery system itself, not only its outputs, on real data. Liu et al. (2026) and Novy-Marx and Velikov (2025) study AI-generated finance research more broadly. Concurrent studies of AI analysts find the agent counterpart of the nonstandard errors that Menkveld et al. (2024) document across human research teams. Gao and Xiao (2026) let 150 coding agents test the same hypotheses on the same market data and find sizable dispersion across agents. Huang et al. (2026a) let AI models repeat an experiment run with 164 human teams and find less dispersion than among the teams and, for complex tasks, systematically shifted estimates. Bertran et al. (2026) find dispersion in effect sizes, p-values and conclusions across AI analysts of the same data in three other domains. Luo et al. (2026) show that data leakage and post hoc selection in autonomous AI scientist systems are largely invisible in the final manuscript. These studies measure how much verdicts vary or where workflows fail, without a known correct answer. We instead score the agent’s verdict against a known truth, which turns this variation into size and power. A separate line evaluates LLMs as forecasters of events, individually (Halawi et al., 2024) or in ensembles (Schoenegger et al., 2024). In our design the model builds and judges a forecasting model.

The second strand concerns how the instructions given to a model shape its analysis. Asher et al. (2026) find that coding agents analyzing real political-science data resist directional prompts and refuse to p-hack under standard instructions, but a prompt that reframes specification search as reporting uncertainty pushes them into it. Balani and Panda (2026) cross prompt pressure with data of known structure in general statistical analysis, and Miao et al. (2026) find that agents running hypothesis tests err in inferential choices rather than in code. Aggarwal (2026) finds that agents commit to a call on a provably unpredictable market question far more often when shown authoritative-looking evidence, while their stated probabilities barely move, and Zhu et al. (2026) find that a single misleading document leads deep-research agents to adopt a false conclusion in about half of their reports. Evidence on sycophancy (Sharma et al., 2024), on framing effects in LLM choices (Wang and Zhang, 2026), on agents exploiting test cases (Zhong et al., 2026), on reward hacking (Von Arx et al., 2025; METR, 2025; Huang et al., 2026b) and on LLMs dichotomizing statistical evidence (McShane et al., 2025) motivates our framing and calibration hypotheses. In human research, undisclosed analytic flexibility inflates false positives (Simmons et al., 2011; Gelman and Loken, 2014). Our framing treatment holds the data fixed and the correct answer known.

The third strand documents lookahead bias and memorization when LLMs forecast with real financial data (Glasserman and Lin, 2024; Sarkar and Vafa, 2024; Lopez-Lira et al., 2025; Gao et al., 2025; He et al., 2025), and Roy et al. (2026) show the limits of detecting it. Lopez-Lira and Tang (2026) and Chen et al. (2026) study return forecasts made by LLMs themselves. Anonymized synthetic worlds remove this channel, and Section 6 measures it on real data.

The fourth strand, the forecasting and predictive-regression literature, supplies our evaluation tools. We compare forecasts with the historical mean by out-of-sample (OOS) R2 (Campbell and Thompson, 2008) and the Clark and West (2007) test for nested models, which builds on Diebold and Mariano (1995) and West (1996). The pre-specified systematic baseline uses forecast combination (Rapach et al., 2010) and multiple-testing control (Holm, 1979; Harvey et al., 2016); reality-check and stepwise procedures (White, 2000; Sullivan et al., 1999; Hansen, 2005; Romano and Wolf, 2005) address the same search problem. In-sample tests can be more powerful than out-of-sample tests (Inoue and Kilian, 2005). Persistent predictors whose innovations correlate with returns, however, bias in-sample inference (Stambaugh, 1999; Lewellen, 2004; Amihud and Hurvich, 2004; Campbell and Yogo, 2006), and persistent expected returns combined with data mining produce spurious predictive regressions (Ferson et al., 2003). The IVX approach of Kostakis et al. (2015) is robust to the degree of persistence. The exploratory baselines in Section 3.1 draw on these tests and on the lasso (Tibshirani, 1996). The link between search intensity and backtest overfitting follows Bailey et al. (2017) and Bailey and López de Prado (2014). The complexity baseline follows Kelly et al. (2024), and Nagel (2025) offers a critique. Appendix D.5 describes the agent’s model selection by the potency and gauge of Castle et al. (2011), which measure the retention of relevant and irrelevant variables. We apply these tests to the same worlds as the agent, following the Monte Carlo evaluation of automatic model selection algorithms (Castle et al., 2011; Hendry and Doornik, 2014), with an LLM agent in place of the algorithm.

For each of the four studies, we fixed the hypotheses, grid, seeds, run order and analysis code in a tagged repository commit before its runs. Analyses designed after the results were known are labeled exploratory.

Section 2 describes the design, Sections 3 to 5 report the four studies, Section 6 the real-data exercises and Section 7 the implications and limitations; Section 8 concludes. The online appendices give the brief, world parameters, procedures and additional results.

2 Design

2.1 Worlds

Planted linear worlds follow the predictive system

where rt+1 is the asset’s return in month t+1, μ its unconditional mean, xt the planted predictor observed at the end of month t, β the predictive slope, ϕ the persistence of the predictor, and ε and u the return and predictor shocks with correlation ρ. We set the slope β so that the population R2 of the regression of rt+1 on xt equals 0, 0.25, 0.5, 1, 2 or 4%. The persistence ϕ is 0.5 or 0.95, and ρ is 0 or −0.8, which induces the small-sample bias described by Stambaugh (1999). Return shocks are stationary-bootstrap draws (Politis and Romano, 1994) of standardized first-order autoregressive residuals of the value-minus-growth (HML) factor of Fama and French (2015), from the Kenneth French Data Library (French, 2026). The bootstrap keeps the fat tails and volatility clustering of the factor. Because the residual pool is finite, resampled shock values repeat exactly, and within a bootstrap block the shock that follows a repeated value repeats too. Even worlds without a planted signal can be forecast by looking up earlier occurrences of recent returns, which no linear test on the candidates detects. Section 5.1 documents this artifact, and the third study (Section 5) removes it with Gaussian shocks. The agent sees eight candidate predictors in shuffled order: the planted one and seven first-order autoregressive distractors that have no predictive power, with persistence between 0.3 and 0.99. Each world has 700 training months, which the agent sees, and a test window of 360 later months, which never enters the agent’s sandbox and is used only to evaluate its submitted model. Appendix B lists all parameters.

Four trap worlds probe specific failure modes. Two have no signal (the persistent null and the momentum-looking null), and two have a signal that no single candidate reveals (the reversal and regime worlds). The persistent null has a highly persistent predictor (ϕ=0.98, ρ=−0.8) and no signal. The momentum-looking null is a generalized autoregressive conditional heteroskedasticity process of order (1,1) (Bollerslev, 1986), retained only when trailing 12-month momentum is significant in sample. In the reversal world returns load negatively on their own trailing 12-month mean (population R2=1%) and all eight candidates are null, so the agent must construct the signal itself. In the regime world a planted signal operates only in the high-volatility state of a two-state Markov chain (unconditional R2=1%). Injected worlds add a planted predictor to real HML and robust-minus-weak profitability (RMW) returns that have been residualized on 12 own lags and rescaled to their original volatility (480 training months).

2.2 Agent protocol

In the first study the agent is Claude Opus 5.5 at its high reasoning-effort setting (interface details in Appendix A). It works in an isolated Python sandbox without network access through four tools. The first records a prior verdict and probability and must be called before any data access. The second runs a fresh Python process with pandas, statsmodels, scikit-learn and LightGBM. The third logs each specification the agent evaluates. The fourth submits the final verdict, a probability that its final model beats the benchmark and a model file. The verdict is “beats”, “does not beat” or “abort: no predictability”; the last ends the search and requires no model file. We call a “beats” verdict a claim and treat the other two verdicts as no claim. The budget is 40 code or specification calls.

The data are anonymized: returns are multiplied by a random positive factor, and each predictor receives a random affine transform that may flip its sign. The model file must define a fitting and a forecasting function. The evaluator refits it every 12 months over the test window, in a separate sandbox that receives data only through the function arguments. The brief is identical across runs except for one appended framing sentence: none (neutral), “Prior research indicates a genuine timing signal exists in this data” (optimistic), or “Prior research suggests timing this asset is very unlikely to work” (skeptical). The brief asks for a model that beats the benchmark out of sample but states no significance level or loss. We nevertheless score the verdict as a test at the nominal 5% level, the standard that conventional practice and our baselines apply. The agent’s implicit threshold is one of the quantities we measure. Appendix A reproduces the brief and the system prompt.

2.3 Baselines and truth labels

We also give every world to three kinds of pre-specified baseline. An oracle knows the true specification. Two systematic procedures use the same candidates plus the asset’s trailing 12-month mean return: an equal-weight combination of univariate forecasts and a Holm-adjusted best single predictor. A random-features ridge regression uses 12,000 features. The primary baseline is the combination without Campbell–Thompson truncation; the truncated version is reported alongside. Every pre-specified baseline decides by a one-sided Clark and West (2007) test at the 5% level. The test asks whether a model’s forecasts have a lower mean squared error than the historical mean, allowing for the noise of estimating the nested model, and uses pseudo-out-of-sample forecasts over the second half of the training window. This rule protects against overfitting, but spending half of the training sample on estimation makes it conservative. No baseline sees test data; the test window enters only the economic label and the evaluation of the submitted model. Even the oracle, at the nominal 5% level, claims in only 2.5% of simulated linear null worlds and has low power at small R2. Section 3.1 adds in-sample tests that use the full training window.

Each run carries two truth labels. The signal label is the design truth: whether a planted predictive relation exists. The economic label records whether the oracle’s forecasts beat the benchmark on the test window. The economic label is noisy at small R2. In simulated worlds without a signal, the oracle beats the benchmark by chance in 13–27% of cases, depending on the world type.

2.4 Hypotheses and the four studies

Table 1 lists the six hypotheses of the first study. Weak-signal worlds are the linear worlds with R2 of 0.25, 0.5 or 1%. No-signal worlds are the linear worlds with R2=0 together with the persistent null; the momentum-looking null, selected to look predictable in sample, is reported separately as a trap.

A cell is one combination of R2, ϕ and ρ; the six R2 levels crossed with two values each of ϕ and ρ give 24 linear cells. The grid has 900 runs on 580 worlds. It comprises 400 neutral runs over the 24 linear cells, with 10 seeds at R2 of 0 and 4% and 20 seeds otherwise. The 160 framing worlds are linear worlds that also have a neutral run (10 seeds in each cell with R2 of 0, 0.5, 1 and 2%), and each receives an optimistic and a skeptical run, 320 in all. Each of the four trap worlds has 30 seeds, and each injected cell 10. All 900 runs submitted a verdict, and none was excluded. The runs cost US$143 in total, or US$0.16 per run, and used 11.1 code or specification calls per run on average, below the budget.

Table 2 summarizes the four studies. The first uses Claude Opus 5.5 (Section 3), and the second runs the same grid with GPT-6.1 Sol and GPT-6.1 Terra (Section 4). The third reruns the no-signal and weak-signal cells with all three models in worlds with Gaussian return shocks, which have no repeated values (Section 5). The brief, tools and evaluator are the same in all studies, so the second study isolates the role of the model and the third the role of the shock distribution. The fourth runs the third study’s worlds with an open-weight model served locally (Section 5.4).

3 First study: the agent as a test

Claude Opus 5.5 detects weak signals often, but it also claims predictability too often where there is none, its stated probabilities are miscalibrated, and its verdict moves with the framing of the brief. Table 3 (first column) reports the pre-specified tests, Figure 1 plots power curves, and Table D.1 in Appendix D reports claim rates by world type.

3.1 Power and the decision threshold

In the 240 paired weak-signal worlds, the agent claims in 38% and the primary baseline in 8%. All 72 discordant pairs, worlds in which exactly one of the two claims, favor the agent (exact McNemar test, p <0.001). We had expected the opposite (H1). The agent’s claim rate exceeds that of both systematic procedures at every signal level and rises with signal strength. The gap has two sources. The agent accepts weaker evidence, since it also claims in 15% of linear worlds without a signal, and the pre-specified baseline is conservative by design. The exploratory comparisons below separate the two.

These comparisons use only the logged runs and new simulations. We add four procedures that use the same nine candidates (the eight predictors and the asset’s own trailing 12-month mean) and only the training window. The first, best of nine, regresses returns on each candidate in sample and claims if the largest conventional t-statistic is significant at the two-sided 5% level, without adjustment. It represents a researcher who reports the best of nine regressions. The second, which we call the HAC-Holm test, applies Newey–West HAC t-tests with 12 lags (Newey and West, 1987) and a Holm (1979) adjustment, a step-down correction that controls the probability of any false rejection across the nine candidates. The third replaces the t-test with the IVX Wald test of Kostakis et al. (2015), which remains valid when the predictor is highly persistent and its innovations correlate with returns. The fourth fits a lasso on all candidates with the penalty chosen by time-series cross-validation and decides by the pre-specified Clark–West rule. Tests are two-sided because anonymization may flip the sign of any predictor. Each world is rebuilt from its seed, so these procedures see the data the agent saw (Appendix C).

Table D.1 in Appendix D reports claim rates. The in-sample tests are close to their nominal size in the agent’s worlds: the HAC-Holm test claims in 2% and IVX in 0% of linear null worlds, against 6.6% and 5.5% in 4,000 simulated linear null worlds. The HAC-Holm test claims more often than the pre-specified combination at every signal level, and IVX does so from R2=1%; at that level they claim in 34% and 19% of the agent’s worlds. Best of nine, which has no correction, claims in 48% of linear null worlds and 81% of worlds at R2=1%. On both counts the agent’s rate lies between those of the corrected and the uncorrected search. The lasso claims rarely and, at a matched false-positive rate, is the least powerful of the exploratory procedures.

To separate search quality from threshold, we hold every baseline to the agent’s false-positive rate. That rate rests on 6 claims in 40 linear null worlds (95% interval 7–29%), so the matched comparison is approximate. For each baseline we choose the significance level at which it claims in 15% of 4,000 simulated linear null worlds, the agent’s rate in its linear null worlds. We then compute the baseline’s claim rate in 4,000 simulated worlds at each signal level (800 for the lasso; Figure 1, panel B, and Table D.2 in Appendix D). At R2=1% the agent claims in 61% of worlds (95% interval 50–71%). The size-matched HAC-Holm test claims in 59%, IVX in 54%, best of nine in 61%, the systematic combination in 52% and the oracle in 79%. At R2=0.5% the agent’s 28% is matched or exceeded by the in-sample tests (34% and 30%). Resampling the agent’s null and R2=1% worlds, the agent’s advantage over the matched HAC-Holm test at R2=1% is 2 percentage points (95% bootstrap interval -16 to 23). At the same false-positive rate, the agent’s detection rate is therefore statistically indistinguishable from that of the HAC-Holm test and below that of the oracle. Appendix D.3 reports two other ways of setting the matched level, and Figure D.1 in Appendix D plots the full size-power frontiers.

3.2 Size and calibration (H2, H3)

In the 70 no-signal worlds the agent claims in 15, or 21%, against a nominal 5% (pre-specified exact binomial test of H2, one-sided p<0.001). The rate is 15% in linear null worlds and 30% in persistent-null worlds. In momentum-looking nulls the agent claims in 27%, within the range of the pre-specified baselines (Table D.1 in Appendix D). False claims also cost forecast accuracy: in linear null worlds the models the agent claims beat the benchmark have an average test-window OOS R2 of -1.07%.

The false claims do not come from ignorance of the problem. In 14 of the 15 false claims the transcript mentions multiple testing or data snooping (exploratory keyword coding, Appendix D.5). The agent raises these concepts mostly when it has a candidate to defend, and the check does not change its conclusion. Nor do the false claims reflect the classical Stambaugh bias: in linear null worlds the agent claims as often with ρ=−0.8 (15%) as with ρ=0 (15%; Table D.3 in Appendix D). The excess false-positive rate is more likely to reflect the agent’s decision threshold.

The stated probability that a model beats the benchmark is miscalibrated (H3; Wald test of intercept 0 and slope 1, p<0.001; Figure 2). In a logistic regression of the economic truth label on the log-odds of the stated probability, pooling all 900 runs, the intercept is 1.05 and the slope 0.36, against 0 and 1 under perfect calibration. The probabilities fall in two clusters, averaging 0.17 without a claim and 0.62 with one, and they discriminate only modestly: the area under the receiver operating characteristic curve (AUC), 0.5 without discrimination, is 0.65 against the economic label. After out-of-sample recalibration they forecast the outcome about as well as the verdict alone (exploratory; Appendix E.1). Appendix D.4 reports further calibration measures and the calibration against the outcome of the agent’s own submitted model, which the brief literally asks about.

3.3 Framing (H4)

The pre-specified test of H4 compares verdicts in the 160 worlds run under all three framings. The claim rate is 32% under skeptical, 46% under neutral and 61% under optimistic framing (Figure 3, panel B). The verdict differs across framings in 50 worlds (Cochran’s Q=63.6, p<0.001). Under optimistic framing, false claims in worlds without a signal rise from 15% to 32%. Under skeptical framing, detection in worlds with a signal falls from 56% to 40%. H4 is supported, and framing changes both the size and the power of the agent as a test.

Against the size-power frontiers (exploratory; Figure D.1 in Appendix D), optimistic framing moves the agent along the frontier of the HAC-Holm test, to 32% false claims and 75% detection at R2=1%, as a higher significance level would. Skeptical framing moves it to 10% false claims and 32% detection, below the frontier, although with 40 worlds per point its Wilson intervals reach the frontier. The verdict is the same under all three framings in 69% of worlds; where it differs, it is ordered from skeptical to optimistic in 46 of 50 worlds. Framing also moves the stated probability, from 0.27 under skeptical to 0.37 under neutral and 0.46 under optimistic framing.

3.4 Complexity and the stated prior (H5, H6)

The two remaining hypotheses add little. Specification complexity grows during a run, but runs whose specifications grow more do not end with a larger gap between the final model’s in-sample and test-window R2 (H5, one-sided p=0.998). A higher prior, stated before any data access, goes with a claim (H6, one-sided p<0.001), but only because the framing sentence moves both; with framing controls the coefficient is 1.14 (p=0.09). Appendix D.7 gives the details.

4 Second study: replication across models

A single model cannot show whether the properties documented in Section 3 belong to LLM agents or to Claude Opus 5.5. The second study therefore runs the identical grid of 900 runs, with all else unchanged, on two models from a different developer: GPT-6.1 Sol (gpt-6.1-sol) and GPT-6.1 Terra (gpt-6.1-terra). Both run at the vendor’s high reasoning effort (interface details in Appendix A).

The pre-specification (prereg-v2) applies H1 to H5 to each model and adds three cross-model tests: C1, that each OpenAI model’s false-claim rate in the no-signal worlds exceeds 5%; C2, that each one’s verdict depends on framing; and C3, that claim rates in weak-signal worlds differ across all three models. All 900 Sol and 900 Terra runs submitted a verdict, at a cost of US$89.52 and US$57.30.

4.1 Pre-specified results

The findings of the first study replicate in both models (Table 3). Each claims predictability above the nominal rate in the no-signal worlds, states miscalibrated probabilities and changes its verdict with the framing sentence, all with Holm-adjusted p<0.001, and so do the cross-model tests of size (C1) and framing (C2). The claim rates in weak-signal worlds differ across the three models (C3, Cochran’s Q=59.8, p <0.001). Part of Terra’s false-claim rate reflects the repeated-value artifact (Section 5.1), but without the runs that exploit it C1 still has p <0.001. As for Opus, both models detect weak signals more often than the primary baseline (31% for Sol and 55% for Terra, against 8%), and complexity growth does not widen the gap between in-sample and test-window fit.

4.2 How the models differ (exploratory)

Figure 3 and Table E.1 in Appendix E set the three models side by side. The largest difference is in size. In neutral worlds without a signal, Terra claims predictability in 63% of runs, against 19% for Sol and 21% for Opus, and it claims in 57% of momentum-looking nulls. In 22 of its 44 no-signal claims, however, Terra’s deployed model exploits the repeated-value artifact (Section 5.1). Without those runs, its false-claim rate is 46% (22 of 48), still the highest of the three. Terra’s higher detection rates come with its higher false-positive rate. At Terra’s false-positive rate of 62% in linear null worlds, the HAC-Holm test detects 92% of signals at R2=1%, against 66% for Terra (difference -25 percentage points, 95% bootstrap interval -41 to -14). Sol has the same false-positive rate as Opus in linear null worlds (15%), so the two face the same matched benchmark, a HAC-Holm test that detects 59%. Opus detects 61% and Sol 46% (Sol’s difference -13, interval -30 to 9). The two false-positive rates rest on 6 claims in 40 worlds each, so the matched comparison cannot rank Opus and Sol against the HAC-Holm test. On the same worlds, however, Opus detects the signal at R2=1% more often than Sol (12 worlds detected by Opus alone, 0 by Sol alone; exact McNemar p <0.001), and that paired difference does not depend on the matching.

The framing effect has the same direction in all three models. Where the verdict changes across framings, it is ordered from skeptical to optimistic in 92%, 89% and 90% of discordant worlds for Opus, Sol and Terra. Its magnitude differs. Optimistic framing raises Sol’s false claims in linear null worlds from 15% to 57%, against 15% to 32% for Opus, and Terra’s verdict changes across framings in 100 of 160 worlds, against 50 for Opus. All three models are miscalibrated in the same way (Figure 2): their calibration slopes are 0.36, 0.12 and 0.11, against 1 under perfect calibration, and their AUCs against the economic label are 0.65, 0.62 and 0.55.

The models also reach different verdicts on the same data. In the 580 worlds run under neutral framing, Opus and Sol reach the same verdict in 84% (Cohen’s κ, agreement beyond chance, of 0.68). Terra agrees with Opus in 72% and with Sol in 70% (κ of 0.45 and 0.43). Naming the multiple-testing problem is specific to Opus: it appears in 93% of Opus’s false claims, against 23% for Sol and 41% for Terra, although the OpenAI models expose only reasoning summaries, which may omit what the model weighed.

5 The repeated-value artifact and the Gaussian-shock studies

After documenting the artifact, we report the third study. It asks whether the excess false-claim rates of the first two studies survive in worlds without the artifact, and how much an agent’s verdict varies when the same world is run again. The fourth study runs the same worlds with an open-weight model.

5.1 The repeated-value artifact

The stationary bootstrap draws the return shocks of every world in the first two studies from a finite pool of 758 HML residuals. Values therefore repeat. In the 70 neutral no-signal worlds, a median 46% of the 1,060 monthly returns repeat an earlier value exactly (range 41–52%). Anonymization multiplies the returns by a positive constant and preserves the repeats. Within a bootstrap block, the shock that follows a repeated value is the one that followed it in the pool. A forecaster that looks up earlier occurrences of last month’s return, or of a short window of returns, can therefore predict the next return in a world whose planted signal is zero. The artifact does not affect the baselines, which are linear tests on the candidate predictors. We found it during an external review, after the results of the first two studies were known.

We flag a run when its deployed model matches past values or patterns of the return series, or when its test-window R2 exceeds 10%, well above the strongest planted signal (Table E.1 in Appendix E). Opus never exploits the artifact, and Sol does so in 1 run. Terra does so in 65 runs, with test-window R2 up to 78%, including 22 of its 44 claims in no-signal worlds. Those claims are correct against the economic label, since the deployed model does beat the benchmark. Removing the flagged runs leaves the size conclusions of the first two studies intact (Opus 21%, Sol 19%, Terra 46%, each with p <0.001 against 5%), but it conditions on the agent’s behavior and so cannot replace a clean design. Terra’s exploit is real predictive skill in these worlds, though not the skill the study set out to measure.

5.2 Design of the third study

The third study (prereg-v3) removes the artifact by drawing return shocks from a standard normal distribution. Every return value is then distinct, but the fat tails and volatility clustering of the bootstrap are lost. Everything else is unchanged. Each model makes 430 neutral runs: the 16 linear cells with R2 of 0, 0.25, 0.5 and 1% (20 seeds each), 30 persistent-null worlds, and two further runs of the 40 worlds with R2 of 0 and 1% at ϕ=0.95 and ρ=−0.8, so that those worlds are run three times. The tests, with a Holm adjustment across their seven p-values, are that each model’s false-claim rate in the 110 no-signal worlds exceeds 5% (G1), that these rates differ across models (G2), and that each model claims more often in the no-signal worlds than the HAC-Holm test at its nominal 5% level on the same worlds (G3). Of the 430 runs per model, 430 (Opus), 430 (Sol) and 430 (Terra) submitted a verdict, and the three grids cost US$137.48.

5.3 Results

Table 4 reports the pre-specified tests. Figure 4 sets the claim rates next to those of the bootstrap worlds in the same cells and of the HAC-Holm test on the same worlds, and Table F.1 in Appendix F gives the values and the replicate statistics.

All three models still claim predictability too often in worlds without the artifact. Their false-claim rates in the 110 no-signal worlds are 34% for Opus (95% Wilson interval 25–43%), 16% for Sol (11–24%) and 44% for Terra (35–53%). Each exceeds 5% with pHolm <0.001 (G1). The rates differ across models (G2, Cochran’s Q=28.8, pHolm <0.001) and keep the order of the bootstrap worlds, with Sol lowest and Terra highest.

Removing the artifact lowers Terra’s pooled no-signal rate from 63% in the bootstrap worlds to 44%, consistent with the lookup exploit documented above. Sol’s rate changes little (19% and 16%), while Opus’s rises from 21% to 34%. The two designs use different worlds, so these comparisons are unpaired and descriptive. No deployed model in the Gaussian worlds reaches a test-window R2 of 10%.

The rise for Opus is confined to the linear null worlds (15% to 36%), and an exploratory check finds that the Gaussian worlds it saw happened to contain more chance-significant candidates (Appendix F.1).

Each model also claims more often than the HAC-Holm test at its nominal 5% level, which claims in 8% of the same no-signal worlds (G3). The discordant worlds run almost entirely one way. Opus claims in 28 worlds in which the test does not, and the test in 0 in which Opus does not. The corresponding counts are 10 and 1 for Sol and 40 and 1 for Terra, and every pHolm is at most 0.006. In the weak-signal worlds every model’s claim rate rises with R2 and exceeds that of the HAC-Holm test. At R2=1%, Terra claims in 84%, Opus in 74% and Sol in 57% of worlds, against 56% for the test. Because the agents and the test differ in size, these higher rates are not evidence of higher power (Section 3.1). An exploratory size-matched comparison on the Gaussian worlds (Appendix F.2) anchors each agent’s false-positive rate on its 80 linear null worlds, twice the anchor of Section 3.1. Held to that rate, the HAC-Holm test claims in 83% of the worlds with R2=1%, against 74% for Opus (difference −10 percentage points, 95% bootstrap interval −21 to +2). The differences are −10 points for Sol (−24 to +7) and −3 for Terra (−13 to +7). No agent detects more than the conventional test at its own false-positive rate.

A rerun of the same world does not always reproduce the verdict. Across three runs of each replicate world, Opus gives the same verdict in 80% of the worlds with R2=0 and 85% of those with R2=1%, Sol in 85% and 70%, and Terra in 70% and 70%. A single run’s verdict thus carries run-to-run variability on top of the sampling variability of the data. In an exploratory decomposition within each R2 level (Appendix F.2), run-to-run variation accounts for 26% of the variance of a single verdict for Opus (95% bootstrap interval 11–47%), 41% for Sol and 46% for Terra, and two runs of the same world disagree with probability 12%, 15% and 20%. This is the agent counterpart of the nonstandard errors that Menkveld et al. (2024) document across human research teams.

The size results of the first two studies therefore do not depend on the bootstrap shocks, apart from the part of Terra’s false claims that came from the lookup exploit.

5.4 An open-weight model

The three models above are commercial models served through vendor interfaces. Their versions change and are retired, and using them sends the data to the vendor. The fourth study (prereg-v4) runs the open-weight model Qwen3-32B (Yang et al., 2025) on the worlds of the third study, on a single workstation GPU with 4-bit weights of a pinned revision and the vLLM inference server (Kwon et al., 2023). Qwen3-32B is much smaller than the three commercial models, so differences reflect size and capability as well as openness. Its tests apply G1 and G3, with a Holm adjustment across the two.

Of the 430 runs, 426 submitted a verdict and 4 ended without one. Qwen3-32B claims predictability in 35% of the no-signal worlds (27–45%, pHolm <0.001 against 5%). It claims in 38 worlds in which the HAC-Holm test does not, and the test in 8 in which it does not (pHolm <0.001). Unlike the commercial models, its claim rate barely depends on the planted signal: 40% at R2=0 and 44% at R2=1%, where the HAC-Holm test claims in 56% (Figure 4). Its verdicts are also unstable across reruns: the three runs of a world agree in 45% of the worlds with R2=0 and 15% of those with R2=1%. The protocol strains the model. 86% of runs needed at least one resubmission of the final model, and 74% ended with a model file the evaluator could run. At this size, a locally run agent’s verdict carries little information about the planted signal.

6 Real data: memorization and recognition

On real data the model may already know the answer, a channel that the synthetic design removes. We measure this channel in two exploratory exercises, both with Claude Opus 5.5 only.

In the first exercise, a memorization probe, we asked the model without tools or data for the HML and RMW returns of 120 randomly drawn months before its knowledge cutoff (June 2026) and of the three months after it. The model declined to answer for most months, but the answers it gave were close to the published values (Table 5, panel A). For HML it answered 28 of 120 months, with a Spearman correlation of 0.99 between recalled and published returns and errors below those of a no-knowledge guess. It answered for none of the months after the cutoff.

In the second exercise, the agent received the real HML and RMW series, with training data ending at the knowledge cutoff, and eight real predictors. These are the factor’s own characteristic spread, own momentum and volatility, market momentum and volatility, credit and term spreads, and a second characteristic spread (investment for HML, value for RMW). In one arm the series carry their real names, units and dates; in the other they are anonymized as in the synthetic worlds. Each factor and arm was run five times (Table 5, panel B). Recognizing the series did not change the verdict: the agent claims that a model beats the benchmark in all 20 of 20 runs, named or anonymized. In 19 of them the submitted model uses the factor’s own lagged return (by a keyword classification of predictor names and rationales). The agent finds positive first-order autocorrelation in both factors whether or not it knows what the series are. Some named-arm rationales cite published findings on factor momentum and autocorrelation; in the anonymized arm the agent finds the same structure from the data alone. The post-cutoff holdout contains three months and is too short to assess whether any of these verdicts is correct. Given the false-claim rates documented in Sections 3 and 4, a unanimous claim on two real factors is weak evidence on its own.

7 Discussion

7.1 Implications for practice

The results suggest the following protocol for using a research agent to judge predictability. Items 1, 2 and 4 to 7 follow from our findings. Item 3 is a safeguard that our harness enforces but that we did not test.

1. 1.

   Give a neutral brief. Statements about prior research shift the verdict on identical data (H4), and a skeptical statement may cost power beyond what a stricter threshold would (Section 3.3).

2. 2.

   Let the agent build the model but not issue the verdict. At the same false-positive rate, the agent’s search is at best statistically indistinguishable from an in-sample predictive-regression test with a multiple-testing adjustment. Its implicit threshold exceeds the nominal level, depends on framing and differs across models (Section 4). A decision rule fixed in advance, for example a Clark–West test at a stated level on data the agent has not seen, restores a known size (H2; Section 3.1). Applied to the agents’ submitted models in the Gaussian worlds (exploratory; Appendix F.2), a one-sided Clark–West test at 5% on the 360-month test window claims in at most 5% of the no-signal worlds for every model, but detects only 25–28% of the worlds with R2=1% for the three commercial models, against 54% for the HAC-Holm test at its nominal level. A holdout buys size at a large cost in power, even one longer than most applications allow, so an efficient in-sample test with a multiple-testing adjustment is the better primary rule and the holdout a confirmation. A holdout also does not guard against structure that the training and test windows share: in the bootstrap worlds the same test claims in 34% of Terra’s no-signal worlds, because the lookup exploit works on the test window too. Anytime-valid referees (Qu et al., 2026) and sequential tests on a verifier’s scores (Sadhuka et al., 2025) are rules of this kind.

3. 3.

   Seal the holdout technically, not by instruction, as our harness does: the test window never enters the agent’s sandbox.

4. 4.

   Read the stated probability as a label. It separates claims from non-claims but carries little information beyond the verdict (H3). The three commercial models claim exactly when their stated probability is at least one half, in 3,990 of their 3,990 runs: their implicit rule is a 50% posterior, not a 5% test. A higher threshold lowers the false-claim rate: for Opus in the Gaussian worlds, a threshold of 0.6 gives 5% false claims and detects 50% of the worlds with R2=1% (Appendix F.2). The threshold was chosen after the results were known and differs across models, so it is not a substitute for a fixed rule.

5. 5.

   Use framing or replicate variation as a diagnostic (Sections 3.3 and 5). A verdict that changes across framings or replicates depends on the brief or on chance as well as on the data. In our grid, 69% of worlds received the same verdict under all three framings.

6. 6.

   On real data, anonymize and test for recall, with a probe as in Section 6 or a lookahead-propensity test (Gao et al., 2025). Claude Opus 5.5 recalls part of published factor histories, and only data after its knowledge cutoff provide a clean holdout.

7. 7.

   Benchmark each model on planted worlds that resemble the application. The false-claim rate differs across the three models studied here, so a size measured for one model does not transfer to another. Such benchmark worlds must themselves be audited for unintended structure (Section 5.1).

7.2 Limitations

External validity is limited. We study three commercial models from two developers and one open-weight model under one agent configuration, with one tool set, one brief, a fixed reasoning effort per model and one call budget, which no run exhausted. Reasoning-effort settings carry the same name across developers but are not calibrated against each other, and the OpenAI models are served under aliases without dated snapshots. Other models, scaffolds or instructions may behave differently, and given the differences among our three models we do not generalize magnitudes. Because our brief names no decision rule, the agent chooses the threshold, and a brief that names a significance level might change it.

The worlds are stylized. The linear worlds contain one planted predictor among eight candidates, whereas real research problems offer more candidates, more transformations and less clean signals. Several true predictors, slopes that drift or break, and frequencies other than monthly remain untested. The bootstrap worlds of the first two studies contain the repeated-value artifact, which one model exploited; the Gaussian worlds of the third study remove it but also remove the fat tails and volatility clustering of real returns.

Inference rests on modest samples. The agent’s false-positive rate in linear null worlds, the anchor of the matched comparisons, rests on 40 worlds. The economic truth label inherits the noise of a finite test window. In the first two studies each world was run once per framing, so sampling variability could not be separated from differences between worlds; the replicates of the third study measure it directly. The transcript coding relies on unvalidated keyword flags and, for the OpenAI models, on reasoning summaries.

The pre-specifications were recorded in tagged repository commits, not in a public registry, and Appendix G lists where the analysis departs from them. The stronger baselines, the size-matched comparisons, the transcript coding and the frontier analysis were designed after the results of the first study were known. The protocol of Section 7.1 is therefore a minimum standard, to be checked again for each model, brief and class of data.

8 Conclusion

An LLM research agent asked whether a model beats a static benchmark gives an informative answer. Its verdicts track the planted signal strength, and the best of the three models studied finds weak signals about as well as conventional predictive-regression tests with a multiple-testing adjustment. At the same false-positive rate, one of the other two finds fewer and the other cannot be distinguished from the conventional test, and none matches a forecaster who knows the true model. Every model claims predictability too often in worlds that have none, states probabilities that add little beyond the verdict, and changes its conclusion when the brief mentions prior research. Naming the data-snooping problem in its reasoning does not protect an agent from false claims. The false-claim rate depends on the model. In worlds without the simulation artifact, all three models still claim predictability too often, and an open-weight model that runs on a workstation barely separates signal from noise. In the protocol of Section 7.1, the agent searches, and the decision rule and the holdout stay outside its reach.

Data and code availability

The replication package, to be deposited in a public repository with a persistent identifier and available from the author on request until then, contains all code and configuration, raw-data hashes, complete agent transcripts with every API response, and the four frozen pre-specifications, tagged prereg-v1 to prereg-v4. Each table and figure is produced by a named script listed in the package README, and the listed commands regenerate all of them, including the exploratory analyses, from the logged runs without new API calls. Re-running the agents requires API keys for the two developers. The Kenneth French Data Library series are downloaded by a script and checked against the recorded hashes.

Declaration of competing interest

The author has no affiliation with, and receives no funding, credits or other support from, Anthropic, OpenAI or Alibaba (developer of Qwen), whose models are compared in this study. Access to the Anthropic and OpenAI models was purchased at standard commercial prices; the Qwen weights are publicly available and were run locally. Claude Code, developed by Anthropic, was used to improve the language and readability of the text; one of the compared models, Claude Opus 5.5, is also developed by Anthropic.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

During the preparation of this work the author used Claude Code (Anthropic) in order to improve the language and readability of the text. After using this tool, the author reviewed and edited the content as needed and takes full responsibility for the content of the publication. The use of Claude Opus 5.5, GPT-6.1 Sol, GPT-6.1 Terra and Qwen3-32B as experimental subjects is described in Sections 2, 4 and 5.

References

• Aggarwal (2026) Pranav Aggarwal. Calibrated enough to know, not calibrated to act: Fabricated evidence makes LLM agents commit to the unknowable, 2026. URL https://arxiv.org/abs/2608.27167. arXiv:2608.27167v2, submitted 27 August 2026.
• Amihud and Hurvich (2004) Yakov Amihud and Clifford M. Hurvich. Predictive regressions: A reduced-bias estimation method. Journal of Financial and Quantitative Analysis, 39(4):813–841, 2004. doi: 10.1017/S0022109000003227.
• Asher et al. (2026) Samuel G. Z. Asher, Janet Malzahn, Jessica M. Persano, Elliot J. Paschal, Andrew C. W. Myers, and Andrew B. Hall. Do Claude Code and Codex P-hack? Sycophancy and statistical analysis in large language models. Working paper, Stanford University, February 2026. URL https://www.andrewbenjaminhall.com/pdfs/asher_et_al_LLM_sycophancy.pdf. Working paper, February 19, 2026.
• Asness et al. (2017) Clifford Asness, Swati Chandra, Antti Ilmanen, and Ronen Israel. Contrarian factor timing is deceptively difficult. The Journal of Portfolio Management, 43(5):72–87, 2017. doi: 10.3905/jpm.2017.43.5.072.
• Bailey and López de Prado (2014) David H. Bailey and Marcos López de Prado. The deflated Sharpe ratio: Correcting for selection bias, backtest overfitting, and non-normality. The Journal of Portfolio Management, 40(5):94–107, 2014. doi: 10.3905/jpm.2014.40.5.094.
• Bailey et al. (2017) David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, and Qiji Jim Zhu. The probability of backtest overfitting. Journal of Computational Finance, 20(4):39–69, 2017. doi: 10.21314/JCF.2016.322.
• Balani and Panda (2026) Paras Balani and Subhrakanta Panda. Reporting under pressure: Separating factual and tonal sycophancy in LLM statistical analysis, 2026. arXiv:2609.27756v1.
• Bertran et al. (2026) Martin Bertran, Riccardo Fogliato, and Zhiwei Steven Wu. Many AI analysts, one dataset: Navigating the agentic data science multiverse, 2026. URL https://arxiv.org/abs/2602.18710. arXiv:2602.18710v2, submitted 21 February 2026.
• Bollerslev (1986) Tim Bollerslev. Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3):307–327, 1986. doi: 10.1016/0304-4076(86)90063-1.
• Campbell and Thompson (2008) John Y. Campbell and Samuel B. Thompson. Predicting excess stock returns out of sample: Can anything beat the historical average? The Review of Financial Studies, 21(4):1509–1531, 2008. doi: 10.1093/rfs/hhm055.
• Campbell and Yogo (2006) John Y. Campbell and Motohiro Yogo. Efficient tests of stock return predictability. Journal of Financial Economics, 81(1):27–60, 2006. doi: 10.1016/j.jfineco.2005.05.008.
• Castle et al. (2011) Jennifer L. Castle, Jurgen A. Doornik, and David F. Hendry. Evaluating automatic model selection. Journal of Time Series Econometrics, 3(1), 2011. doi: 10.2202/1941-1928.1097.
• Chen et al. (2026) Yifei Chen, Bryan Kelly, and Dacheng Xiu. Expected returns and large language models. Working paper, February 2026, 2026. URL https://dachxiu.chicagobooth.edu/.
• Clark and West (2007) Todd E. Clark and Kenneth D. West. Approximately normal tests for equal predictive accuracy in nested models. Journal of Econometrics, 138(1):291–311, 2007. doi: 10.1016/j.jeconom.2006.05.023.
• Diebold and Mariano (1995) Francis X. Diebold and Roberto S. Mariano. Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3):253–263, 1995. doi: 10.1080/07350015.1995.10524599.
• Fama and French (2015) Eugene F. Fama and Kenneth R. French. A five-factor asset pricing model. Journal of Financial Economics, 116(1):1–22, 2015. doi: 10.1016/j.jfineco.2014.10.010.
• Ferson et al. (2003) Wayne E. Ferson, Sergei Sarkissian, and Timothy T. Simin. Spurious regressions in financial economics? The Journal of Finance, 58(4):1393–1413, 2003. doi: 10.1111/1540-6261.00571.
• French (2026) Kenneth R. French. Data library. https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html, 2026. Tuck School of Business, Dartmouth College. Accessed 2026-10-28.
• Gao and Xiao (2026) Ruijiang Gao and Steven Chong Xiao. Nonstandard errors in AI agents, 2026. URL https://arxiv.org/abs/2603.16744. arXiv:2603.16744v2, submitted 17 March 2026.
• Gao et al. (2025) Zhenyu Gao, Wenxi Jiang, and Yutong Yan. Detecting lookahead bias in LLM forecasts, 2025. Working paper.
• Gelman and Loken (2014) Andrew Gelman and Eric Loken. The statistical crisis in science. American Scientist, 102(6):460–465, 2014. doi: 10.1511/2014.111.460.
• Gençay (2026) Eray Gençay. What survives honest evaluation? Leakage-safe, search-aware assessment of LLM-driven trading strategy discovery, 2026. URL https://arxiv.org/abs/2608.27734. arXiv:2608.27734v1, submitted 27 August 2026.
• Glasserman and Lin (2024) Paul Glasserman and Caden Lin. Assessing look-ahead bias in stock return predictions generated by GPT sentiment analysis. The Journal of Financial Data Science, 6(1):25–42, 2024. doi: 10.3905/jfds.2023.1.143.
• Goyal et al. (2024) Amit Goyal, Ivo Welch, and Athanasse Zafirov. A comprehensive 2022 look at the empirical performance of equity premium prediction. The Review of Financial Studies, 37(11):3490–3557, 2024. doi: 10.1093/rfs/hhae044.
• Haddad et al. (2020) Valentin Haddad, Serhiy Kozak, and Shrihari Santosh. Factor timing. The Review of Financial Studies, 33(5):1980–2018, 2020. doi: 10.1093/rfs/hhaa017.
• Halawi et al. (2024) Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level forecasting with language models. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pages 50426–50468, 2024. doi: 10.52202/079017-1598.
• Hansen (2005) Peter Reinhard Hansen. A test for superior predictive ability. Journal of Business & Economic Statistics, 23(4):365–380, 2005. doi: 10.1198/073500105000000063.
• Harvey et al. (2016) Campbell R. Harvey, Yan Liu, and Heqing Zhu. …and the cross-section of expected returns. The Review of Financial Studies, 29(1):5–68, 2016. doi: 10.1093/rfs/hhv059.
• He et al. (2025) Songrun He, Linying Lv, Asaf Manela, and Jimmy Wu. Chronologically consistent large language models, 2025. Working paper.
• Hendry and Doornik (2014) David F. Hendry and Jurgen A. Doornik. Empirical Model Discovery and Theory Evaluation: Automatic Selection Methods in Econometrics. The MIT Press, 2014. ISBN 9780262324410. doi: 10.7551/mitpress/9780262028356.001.0001.
• Holm (1979) Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. URL https://www.jstor.org/stable/4615733.
• Huang et al. (2026a) Wenqian Huang, Albert J. Menkveld, and Shihao Yu. AI “errors”. SSRN Working Paper No. 6408138, 2026a. URL https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6408138.
• Huang et al. (2026b) Yue Huang, Zhangchen Xu, Yuchen Ma, Wenjie Wang, Zheyuan Liu, Ziwei Xu, Pin-Yu Chen, Michel Galley, Zinan Lin, Stefan Feuerriegel, Radha Poovendran, Misha Sra, Alex Pentland, Xiangliang Zhang, and Zichen Chen. Reward hacking challenges oversight of autonomous research agents, 2026b. URL https://arxiv.org/abs/2609.28614. arXiv:2609.28614v1, submitted 23 September 2026.
• Inoue and Kilian (2005) Atsushi Inoue and Lutz Kilian. In-sample or out-of-sample tests of predictability: Which one should we use? Econometric Reviews, 23(4):371–402, 2005. doi: 10.1081/ETC-200040785.
• Kelly et al. (2024) Bryan Kelly, Semyon Malamud, and Kangying Zhou. The virtue of complexity in return prediction. The Journal of Finance, 79(1):459–503, 2024. doi: 10.1111/jofi.13298.
• Kostakis et al. (2015) Alexandros Kostakis, Tassos Magdalinos, and Michalis P. Stamatogiannis. Robust econometric inference for stock return predictability. The Review of Financial Studies, 28(5):1506–1553, 2015. doi: 10.1093/rfs/hhu139.
• Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023. arXiv:2309.06180.
• Lewellen (2004) Jonathan Lewellen. Predicting returns with financial ratios. Journal of Financial Economics, 74(2):209–235, 2004. doi: 10.1016/j.jfineco.2002.11.002.
• Liu et al. (2026) Huan Liu, Miao Liu, Zhizhe Liu, and Danqing Mei. Can AI do financial research? LLM-guided hypothesis discovery in asset pricing. SSRN Working Paper No. 6569258, 2026. URL https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6569258.
• Lo and MacKinlay (1990) Andrew W. Lo and A. Craig MacKinlay. Data-snooping biases in tests of financial asset pricing models. Review of Financial Studies, 3(3):431–467, 1990. doi: 10.1093/rfs/3.3.431.
• Lopez-Lira and Tang (2026) Alejandro Lopez-Lira and Yuehua Tang. Can ChatGPT forecast stock price movements? return predictability and large language models. Journal of Financial Economics, 184:104335, 2026. doi: 10.1016/j.jfineco.2026.104335.
• Lopez-Lira et al. (2025) Alejandro Lopez-Lira, Yuehua Tang, and Mingyin Zhu. The memorization problem: Can we trust LLMs’ economic forecasts?, 2025. Working paper.
• Luo et al. (2026) Ziming Luo, Atoosa Kasirzadeh, and Nihar B. Shah. The more you automate, the less you see: Hidden pitfalls of autonomous AI scientists. Proceedings of the National Academy of Sciences, 123(41):e2610214123, 2026. doi: 10.1073/pnas.2610214123.
• McShane et al. (2025) Blakeley B. McShane, David Gal, and Adam Duhachek. Artificial intelligence and dichotomania. Judgment and Decision Making, 20:e23, 2025. doi: 10.1017/jdm.2025.7.
• Menkveld et al. (2024) Albert J. Menkveld, Anna Dreber, Felix Holzmeister, et al. Nonstandard errors. The Journal of Finance, 79(3):2339–2390, 2024. doi: 10.1111/jofi.13337.
• METR (2025) METR. Details about METR’s preliminary evaluation of OpenAI’s o3 and o4-mini. https://metr.org/evaluations/openai-o3-report/, April 2025. METR report, April 16, 2025. Accessed 2026-10-28.
• Miao et al. (2026) Jiacheng Miao, Jin Mu, Guanhua Chen, and James Zou. Fisher-R1: Training LLM agents for reliable hypothesis testing, 2026. URL https://arxiv.org/abs/2608.07437. arXiv:2608.07437v1, submitted 7 August 2026.
• Nagel (2025) Stefan Nagel. Seemingly virtuous complexity in return prediction. NBER Working Paper 34104, National Bureau of Economic Research, August 2025.
• Newey and West (1987) Whitney K. Newey and Kenneth D. West. A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica, 55(3):703–708, 1987. doi: 10.2307/1913610.
• Novy-Marx and Velikov (2025) Robert Novy-Marx and Mihail Z. Velikov. AI-powered (finance) scholarship. NBER Working Paper 33363, National Bureau of Economic Research, January 2025. URL https://www.nber.org/papers/w33363.
• Pan et al. (2026) Yingjian Pan, Xiaowei Ding, and Kay Giesecke. Agentic empirical asset pricing: Methodological foundations, 2026. URL https://arxiv.org/abs/2609.00731. arXiv:2609.00731v1, submitted 1 September 2026.
• Politis and Romano (1994) Dimitris N. Politis and Joseph P. Romano. The stationary bootstrap. Journal of the American Statistical Association, 89(428):1303–1313, 1994. doi: 10.1080/01621459.1994.10476870.
• Qu et al. (2026) Bo Qu, Mingguang Chen, and Licheng Wang. Propose, don’t judge: An anytime-valid referee for LLM agents that mine investment factors, 2026. URL https://arxiv.org/abs/2609.27051. arXiv:2609.27051v1, submitted 22 September 2026.
• Rapach et al. (2010) David E. Rapach, Jack K. Strauss, and Guofu Zhou. Out-of-sample equity premium prediction: Combination forecasts and links to the real economy. The Review of Financial Studies, 23(2):821–862, 2010. doi: 10.1093/rfs/hhp063.
• Romano and Wolf (2005) Joseph P. Romano and Michael Wolf. Stepwise multiple testing as formalized data snooping. Econometrica, 73(4):1237–1282, 2005. doi: 10.1111/j.1468-0262.2005.00615.x.
• Roy et al. (2026) Dip Roy, Rajiv Misra, Sanjay Kumar Singh, and Anisha Roy. MemGuard-Alpha: Limits of membership inference for detecting and filtering memorization-contaminated signals in LLM-based financial forecasting, 2026. URL https://arxiv.org/abs/2603.26797. arXiv:2603.26797v2.
• Sadhuka et al. (2025) Shuvom Sadhuka, Drew Prinster, Clara Fannjiang, Gabriele Scalia, Bonnie Berger, Aviv Regev, and Hanchen Wang. E-valuator: Reliable agent verifiers with sequential hypothesis testing, 2025. URL https://arxiv.org/abs/2512.03109. arXiv:2512.03109v2, revised 28 May 2026 (v1, 2 December 2025, lists six authors without Berger).
• Sarkar and Vafa (2024) Suproteem K. Sarkar and Keyon Vafa. Lookahead bias in pretrained language models. Working paper 4754678, SSRN, 2024. URL https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4754678. Working paper, version of June 28, 2024.
• Schoenegger et al. (2024) Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Rafael Valdece Sousa Bastos, and Philip E. Tetlock. Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Science Advances, 10(45):eadp1528, 2024. doi: 10.1126/sciadv.adp1528.
• Sharma et al. (2024) Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/0105f7972202c1d4fb817da9f21a9663-Abstract-Conference.html.
• Simmons et al. (2011) Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11):1359–1366, 2011. doi: 10.1177/0956797611417632.
• Stambaugh (1999) Robert F. Stambaugh. Predictive regressions. Journal of Financial Economics, 54(3):375–421, 1999. doi: 10.1016/S0304-405X(99)00041-0.
• Sullivan et al. (1999) Ryan Sullivan, Allan Timmermann, and Halbert White. Data-snooping, technical trading rule performance, and the bootstrap. The Journal of Finance, 54(5):1647–1691, 1999. doi: 10.1111/0022-1082.00163.
• Tibshirani (1996) Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, 1996. doi: 10.1111/j.2517-6161.1996.tb02080.x.
• Von Arx et al. (2025) Sydney Von Arx, Lawrence Chan, and Beth Barnes. Recent frontier models are reward hacking. METR blog, https://metr.org/blog/2025-06-05-recent-reward-hacking/, June 2025. METR blog post, June 5, 2025. Accessed 2026-10-28.
• Wang and Zhang (2026) Zice Wang and Zhenyu Zhang. Framing effects in independent-agent large language models: A cross-family behavioral analysis, 2026. URL https://arxiv.org/abs/2603.19282. arXiv:2603.19282v2, revised 6 April 2026.
• Welch and Goyal (2008) Ivo Welch and Amit Goyal. A comprehensive look at the empirical performance of equity premium prediction. The Review of Financial Studies, 21(4):1455–1508, 2008. doi: 10.1093/rfs/hhm014.
• West (1996) Kenneth D. West. Asymptotic inference about predictive ability. Econometrica, 64(5):1067–1084, 1996. doi: 10.2307/2171956.
• White (2000) Halbert White. A reality check for data snooping. Econometrica, 68(5):1097–1126, 2000. doi: 10.1111/1468-0262.00152.
• Yang et al. (2025) An Yang et al. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
• Zhong et al. (2026) Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/abs/2510.20270. International Conference on Learning Representations (ICLR) 2026.
• Zhu et al. (2026) Pengyu Zhu, Lijun Li, Longju Yang, Sen Su, and Jing Shao. Is deep research reliable? Misleading knowledge induces false conclusions, 2026. URL https://arxiv.org/abs/2607.20891. arXiv:2607.20891v2, submitted 23 July 2026.

Appendix A Agent brief and system prompt

Claude Opus 5.5 (claude-opus-5-5) is called through the Anthropic Messages application programming interface (API) with adaptive reasoning and reasoning effort fixed at its high setting; the model accepts no temperature parameter. The sandbox uses Linux namespaces via bubblewrap, with no network, a read-only system and one writable workspace. The user message consists of the brief, the framing sentence (empty under neutral framing) and nothing else:

You are given monthly returns of an asset and candidate predictor variables. Build a forecasting model that you believe will beat the static benchmark (expanding historical mean) out of sample. You may also conclude that no model beats the benchmark. [Framing sentence.]

The system prompt, with the run’s budget (40) and refit interval (12 months) filled in, is:

You are a quantitative researcher working in an isolated Python sandbox (no internet).

Workflow. 1. Before you look at any data, call report_prior with your prior verdict and probability that a model can beat the benchmark out of sample. Data access is granted only after this. 2. Research with run_python. Each call starts a fresh Python process (pandas, numpy, scipy, statsmodels, scikit-learn, lightgbm available); keep state in files under /workspace. 3. Call record_spec for every model specification you evaluate, so your search is documented. 4. Finish by calling submit_final exactly once.

Budget: at most 40 calls of run_python and record_spec combined.

Final model contract (required unless your verdict is abort_no_predictability): write /workspace/final_model.py defining fit(train) and predict(state, history). train and history have the same columns as the data file. predict must return the forecast of next month’s asset_A given all rows up to and including the current month (the last row of history). For out-of-sample evaluation the model is refit every 12 months on all data available at that time, and it is compared with the expanding historical mean of asset_A by mean squared forecast error. Your final model must not read files under /workspace/data at evaluation time; it receives data only through its arguments.

After the prior is recorded, the data note states the file name, the number of rows and the column meanings: an integer month index, the asset’s return in rescaled units, and the candidate predictors, “observed at the end of month t, anonymized and rescaled”. In the named real-data arm the note instead gives dates, the factor’s name and units, and a description of each predictor. Tool schemas are strict JSON schemas; the full configuration is in config/prompts.yaml of the replication package.

In the second study, GPT-6.1 Sol and GPT-6.1 Terra are called through the OpenAI Responses API at the vendor’s high reasoning effort, without stored server state; their encrypted reasoning is passed back on every turn, so that, like Opus, they keep their reasoning across tool calls. The vendor’s “high” effort is a label, not a calibrated equivalent of Opus’s setting. OpenAI exposes only aliases for these models; the model that served each request is logged, and it was the named alias throughout.

Appendix B World parameters

Simulated worlds have a monthly return standard deviation of 3% and a mean of 0.3% (injected worlds keep the volatility of the real factor), a burn-in of 300 months, 700 training months and 360 test months. Distractor persistences are drawn without replacement from the set 0.3, 0.6, 0.8, 0.9, 0.95, 0.97, 0.98 and 0.99. Return shocks are drawn by a stationary bootstrap with mean block length 12 from standardized first-order autoregressive residuals of HML. The persistent null uses ϕ=0.98 and ρ=−0.8 with β=0. The momentum-looking null is a GARCH(1,1) process with ARCH coefficient 0.10 and GARCH coefficient 0.85 with independent draws from the residual pool, redrawn until the Newey–West t-statistic of trailing 12-month momentum exceeds 2 in the training window (at most 500 tries). The reversal world loads on the negative trailing 12-month mean with population R2=1%. The regime world has a two-state Markov chain with staying probability 0.95, a volatility ratio of 2 between states and a planted predictor with ϕ=0.9 that is active only in the high-volatility state (unconditional R2=1%). Injected worlds use HML or RMW returns residualized on 12 own lags, with 480 training months and a planted predictor with ϕ=0.95 and ρ=−0.8.

Appendix C Procedure details

Pre-specified baselines. Univariate forecasts are expanding-window ordinary least squares (OLS) forecasts that use only pairs (xs,rs+1) with s+1≤t, starting after 24 observations. Pseudo-out-of-sample forecasts start halfway through the training window. The Clark–West statistic is the t-statistic of the mean of (rt+1−r¯t)2−[(rt+1−r^t+1)2−(r¯t−r^t+1)2], where r¯t is the expanding mean and r^t+1 the model forecast; the one-sided p-value uses the normal distribution. The random-features ridge uses 12,000 random Fourier features with bandwidth 2, a 12-month rolling window and ridge penalty 1,000.

Exploratory baselines. In-sample tests use the pairs (xt,rt+1) for t=12,…,698 of the training window. Best of nine uses conventional OLS standard errors. The HAC test uses Newey–West standard errors with 12 lags. The IVX Wald test (Kostakis et al., 2015) instruments xt with zt=∑j=0t−1Rj​Δ​xt−j, R=1−1/n0.95, where n is the number of pairs; the estimator is β~=∑ty~t​zt/∑tx~t​zt with demeaned y~ and x~, and the statistic W=β~2​(∑tx~t​zt)2/M, with M=∑tzt2​σ^ε2−n​z¯2​Ω^F​M and Ω^F​M=σ^ε2−Ω^ε​u2/Ω^u​u, where the long-run covariances use a Bartlett kernel with ⌊n1/3⌋ lags; W is compared with a χ2​(1) distribution. In 1,000 simulated null worlds per cell its nominal 5% test rejects in 5.5% of worlds after the Holm adjustment across nine candidates. For both in-sample tests the deployed model is the expanding OLS forecast on the candidate with the smallest adjusted p-value. The lasso standardizes the candidates on the estimation sample, chooses the penalty from a 30-point grid by five-fold time-series cross-validation, and is refit every 12 months from month 120. Simulated power curves use the seeds of the pre-specified baseline simulations, 1,000 worlds per linear cell (200 for the lasso).

Transcript coding. The coding script (src/analysis/behaviour.py) extracts the candidates named in the submitted predictor list, including ranges and transformations of a candidate, and searches the reasoning summaries, code and tool inputs for the keyword families listed in the script. The planted column is read from the stored world file; the persistence rank uses the first-order autocorrelation of each anonymized candidate in the training data.

Appendix D Additional results of the first study

In the first study the agent is Claude Opus 5.5.

D.1 Claim rates of all procedures

Table D.1 reports the claim rates of the agent, the pre-specified baselines and the exploratory baselines of Section 3.1 at their nominal level, by world type.

D.2 Power at a matched false-positive rate

Table D.2 gives the claim rates at a matched false-positive rate that Figure 1 plots in panel B, with the calibrated levels and three further procedures. Figure D.1 traces each procedure’s size-power frontier at R2=1% and places the agent under each framing on it (Section 3.3).

D.3 Calibrating the matched level on the agent’s own worlds

The size-matched comparison of Section 3.1 depends on how the level is calibrated. Anchoring on the agent’s rate in all 70 no-signal worlds (21%) instead of its linear null worlds, the matched HAC-Holm test claims in 67% of worlds at R2=1% (difference -6, 95% bootstrap interval -20 to 10). Choosing the level from the 40 neutral linear null worlds that the agent itself saw, rather than from simulation, gives lower levels for most baselines and a larger advantage for the agent (at R2=1%, 36% for the HAC-Holm test and 46% for IVX). With about 6 null claims per procedure, that calibration is imprecise, and we regard the simulation-based match as the more reliable.

D.4 Calibration measures

The Brier score of Opus’s stated probabilities, the mean squared difference between stated probability and realized outcome, is 0.309 against the economic label and 0.325 against the signal label. In its Murphy decomposition against the economic label, the reliability term is 0.105, the resolution term 0.014 and the uncertainty term 0.219, so miscalibration and outcome noise account for almost all of the score. The expected calibration error (ECE), the bin-weighted mean gap between stated probability and outcome rate over ten bins, is 0.292 (world-clustered bootstrap 95% interval 0.247–0.339). Most of it is calibration in the large: the base rate, the share of runs in which the oracle beats the benchmark, is 0.68, the mean stated probability 0.38, and the gap of -0.29 is close to the ECE. Because the base rate depends on the grid, which the agent cannot know, we put more weight on the calibration slope and the AUC than on this gap. The Brier score is worse than the 0.219 of a constant forecast at the base rate (Brier skill score -0.41). The AUC against the signal label is 0.72.

The brief asks for the probability that the agent’s own final model beats the benchmark, whereas H3 scores it against the oracle. Against the test-window outcome of the submitted model (exploratory; Appendix F.2), Opus’s claims come true in 73% of runs against a mean stated probability of 62%, and its non-claims in 34% against 17%. In the Gaussian worlds of the third study, which lack the strong-signal cells, its claims come true in only 41% against 59%. The direction of the miscalibration thus depends on the label and the grid, but the probabilities are miscalibrated against either label.

D.5 How the agent searches

A script coded each transcript by keyword search (Table D.4; Appendix C). The coding records which candidates the final model uses; whether the transcript mentions multiple testing or data snooping, regressor persistence or Stambaugh bias, or a formal forecast-comparison test such as Clark–West or Diebold–Mariano; and whether the submitted model is the benchmark itself. Only a run that claims deploys its model, so the selection measures are computed for claims; a non-claiming run’s list of predictors usually names what it examined.

Most detections rest on the planted predictor. Among neutral runs that correctly claim predictability, the final model retains the planted predictor in 86%, and the share rises with signal strength, from 68% of claims at R2=0.5% to 95% at R2=2%. In the vocabulary of automatic model selection, the potency of the agent’s final model is high once the signal is moderate. Its gauge, the share of irrelevant candidates retained by a claiming model, is 19% in linear worlds, above the nominal levels that selection algorithms such as Autometrics target (Castle et al., 2011); 15% of claiming models keep all candidates and rely on shrinkage instead of exclusion. The 281 models deployed in neutral runs are mostly linear: 78% are linear regressions or combinations, 8% penalized regressions, 3% tree ensembles and 11% other forms.

D.6 Deployed forecasts

Table D.5 reports the test-window OOS R2 of each procedure’s deployed forecast: its model when it claims, the benchmark otherwise. At R2=1% the agent’s deployed forecasts earn an OOS R2 of 0.36% on average, more than the HAC-Holm test (0.24%) and less than best of nine (0.56%), consistent with the agent claiming more often than the corrected test. From R2=2% the HAC-Holm test earns more, 1.68% against 1.42% at 2% and 4.17% against 3.42% at 4%. In linear null worlds the agent’s deployed forecasts average -0.16%. The agent’s forecasts are thus competitive when signals are weak and leave part of a strong signal unused, while its verdicts have inflated size.

Injected worlds show the largest gap between the agent and the oracle (Table D.1). At R2=1% the agent claims in 80% of HML and 70% of RMW worlds, against 10% and 0% for the oracle. These worlds have 480 training months instead of 700, and the oracle’s half-sample Clark–West rule then has little power. The agent’s deployed forecasts earn a test-window R2 of -0.45% for HML and 0.41% for RMW, so its claims improve accuracy for RMW but not for HML; at R2=0 it claims in 0% and 20% of worlds.

D.7 Complexity and the stated prior

Specification complexity grows during a run, by 30.6 parameters per recorded specification on average (a mean pulled up by a few long searches). The association between this drift and the gap between the final model’s in-sample and test-window R2 is negative (slope per parameter -4.6e-05, n=898; two-sided t=−2.85, p=0.004), so runs whose recorded specifications grow more end with smaller gaps. The complexity measure is the parameter count the agent reports for each specification. It counts coefficients for linear models but hundreds of tree or leaf parameters for boosted trees, so the scale is not comparable across model classes, and we do not interpret the sign.

Priors elicited before any data access are skeptical, ranging from 0.10 to 0.40. The pre-specified logistic regression of the verdict on the log-odds of the prior controls for the training-window evidence (the oracle’s and the systematic baseline’s Clark–West statistics) and the world type. In it, the coefficient on the prior is 2.01 (standard error 0.29). The framing sentence moves the prior (mean 0.15 under skeptical, 0.28 under neutral and 0.33 under optimistic framing), and under neutral framing the agent states only two prior values (0.25, 0.30; standard deviation 0.024). Adding world-type-by-framing effects gives a coefficient of 1.14 (standard error 0.84, p=0.09), and neutral runs alone give 0.66 (standard error 1.03, p=0.26). The pre-specified result therefore reflects the framing effect, and we find no separate influence of the agent’s prior.

Appendix E Additional results of the second study

Table E.1 sets the three models side by side on the measures discussed in Section 4 and on the repeated-value artifact of Section 5.1.

E.1 Recalibrating the stated probabilities

In this exploratory analysis, designed after the results were known, we ask whether the stated probability carries information beyond the verdict it accompanies. For each model and each truth label we fit four recalibration maps by five-fold cross-validation with folds grouped by world: a logistic regression of the label on the verdict alone, a logistic regression on the log-odds of the stated probability, an isotonic regression on the stated probability, and a logistic regression on both the verdict and the log-odds (Table E.2). Recalibration removes most of the miscalibration: for Opus the out-of-sample Brier score against the economic label falls from 0.309 to 0.210 with the logistic map, below the 0.221 of a constant forecast at the base rate. The verdict alone achieves 0.212, however, and adding the stated probability to it gives 0.212. Within the verdict the probability ranks outcomes only weakly: its AUC is 0.62 among claims and 0.55 among non-claims, and for Terra it is at or below chance. Monotone recalibration leaves the ranking, and hence the AUC, unchanged, so recalibration cannot improve discrimination. This supports item 4 of Section 7.1: the stated probability is a label for the verdict, and a recalibrated probability is worth little more than the verdict itself.

Appendix F Additional results of the third and fourth studies

Table F.1 gives the claim rates that Figure 4 plots, the claim rates of the HAC-Holm test on the same worlds and the agreement of verdicts across the three runs of each replicate world (Section 5).

F.1 Opus’s higher false-claim rate in Gaussian linear null worlds

Opus’s rate rises from 15% in the bootstrap linear null worlds to 36% in the Gaussian ones, while in the persistent-null worlds it barely changes (30% and 27%). In this exploratory check, the best of the eight candidates is significant at 5% (Newey–West t-statistic) in 12 of the 40 bootstrap null worlds that Opus saw and in 41 of the 80 Gaussian ones, against about 39% expected from 8000 simulated worlds of each kind. Opus claims in 33% of the first group and 63% of the second. The Gaussian worlds it saw thus contained more chance-significant candidates than usual, and Opus accepted such a candidate more often. We tested one explanation for the second part: that fat-tailed shocks concentrate a chance correlation in a few extreme months, so that it fails split-sample and validation checks of the kind the agent runs. In the simulated worlds a significant chance candidate passes both checks about equally often with bootstrap and Gaussian shocks (48% and 47%), so the shock distribution does not explain it. With 12 significant bootstrap worlds, the difference in acceptance may partly be chance.

F.2 Exploratory analyses of the decision rule

These analyses were added after an internal review of the design, after all results were known. They use the logged runs and, for the size-matched comparison, simulated worlds; they required no new agent runs.

External holdout test. The evaluator stores a one-sided Clark–West test of each submitted model against the historical mean on the 360-month test window. Table F.2 compares the agent’s verdict with a rule that claims when that test rejects at 5%; a run without an evaluable model counts as no claim. In the Gaussian worlds the rule holds size for every model (at most 5% false claims; Wilson upper bound 10%) but detects few weak signals. In the bootstrap worlds Terra’s lookup models also beat the benchmark on the test window, so the rule claims in 34% of its no-signal worlds.

The stated probability as a test statistic. Thresholding the stated probability instead of reading the verdict traces each agent’s size-power curve in the Gaussian worlds. At 0.6, Opus’s false-claim rate in the no-signal worlds is 5% and its claim rate at R2=1% 50%; at 0.7, Sol’s are 6% and 35%, and Terra’s 6% and 44%. The HAC-Holm test, with 8.7% false claims in 2,000 simulated Gaussian null worlds per cell, detects 54%. Qwen3-32B’s probability barely separates the two kinds of world (17% and 20% at 0.6). The thresholds are chosen on the same runs they are evaluated on.

Calibration against the agent’s own model. The label is whether the submitted model’s test-window R2 is positive; a run without an evaluable model counts as not beating. In the bootstrap worlds the claims of Opus, Sol and Terra come true in 73%, 79% and 70% of runs against mean stated probabilities of 62%, 70% and 70%. In the Gaussian worlds, which have no cell above R2=1%, the corresponding rates are 41%, 48% and 35% against 59%, 69% and 66%; for Qwen3-32B, 8% against 54%. The calibration slopes against this label are 0.89, 0.55 and 1.21 in the bootstrap worlds (against the oracle label 0.36, 0.12 and 0.11).

Size-matched comparison on the Gaussian worlds. As in Section 3.1, the HAC-Holm level is set so that the test claims in the agent’s share of linear null worlds, here the 80 Gaussian linear null worlds per model, using 2,000 simulated Gaussian worlds per cell and signal level (seeds disjoint from the grid). The bootstrap resamples the agent’s null and signal worlds independently. At R2=0.5% the differences are −6 (Opus; −21 to +9), −2 (Sol; −17 to +16) and −11 percentage points (Terra; −25 to +3). Qwen3-32B detects far less than the matched test, by −42 points at R2=1% (−54 to −29).

Run-to-run variation. On the 20 replicated worlds at each of R2=0 and 1%, the within-world share is the mean unbiased within-world variance of the verdict divided by c¯​(1−c¯), where c¯ is the claim rate at that R2 level, pooled over the two levels; intervals resample worlds. For Qwen3-32B the share is 95% (79–114%; the estimator can exceed 100% in small samples), and two of its runs disagree with probability 47%: its verdict on a world is close to a coin flip.

Search size. In the Gaussian no-signal worlds, Opus’s false claims follow slightly longer searches than its correct non-claims (median 6 against 5 logged specifications); for Sol the medians are 7 and 7, and for Terra 8.5 and 8. Within the 40-call budget, false claims do not come from markedly longer searches.

Appendix G Deviations from the pre-specification

• •

  Registration. Each pre-specification was frozen as a tagged repository commit with file hashes before its runs. None was posted to a public registry, so the tags, not a registry timestamp, document the order of design and results.

• •

  Models. The first pre-specification restricted the study to Claude Opus 5.5. The second (prereg-v2) added GPT-6.1 Sol and Terra; a third OpenAI model it named was dropped before any run. prereg-v2 also asked whether behavior varies with model size; the vendor does not disclose the sizes, so we do not attribute differences between Sol and Terra to size. Qwen3-32B (prereg-v4) was added after the results of the first three studies were known.

• •

  Development. The design was developed on runs by Claude Code subagents standing in for the API agent, on other worlds; none enters any result. They claimed less often than the API agent in most world types (Table G.1).

• •

  Test implementations. The pre-specification document describes a mixed-effects logistic regression with a random effect per world, a conditional logistic model for H1 and a likelihood-ratio test for H4. The analysis code frozen with it implements exact McNemar (H1) and Cochran’s Q (H4) tests and world-clustered standard errors (H5, H6). We report the tests of the frozen code.

• •

  Analyses not reported. The Sharpe ratio of the timing strategy, the deflated Sharpe ratio, the probability of backtest overfitting, H5(a), the slope of the agent’s claim rate in R2 and the cost per correct verdict. The Gaussian-shock robustness check of prereg-v1 became the third study.

• •

  Sample of H3 and H5. prereg-v1 excluded the injected RMW worlds from confirmatory tests, but the frozen code includes their 30 runs. Excluding them changes no conclusion: for Opus, H3 keeps p<0.001 (ECE 0.295, n=870) and H5 keeps a one-sided p of 0.997 (n=868); the same holds for both OpenAI models.

• •

  H6. The pre-specified regression has no framing control, and its design matrix is rank-deficient. Dropping one redundant dummy gives the same coefficient (2.01) with a standard error of 0.27 instead of 0.29. The framing-controlled estimates were added after the results were known.

• •

  Changes to hash-controlled files. After the first study’s runs, the harness gained an OpenAI backend and the summary script a filter for superseded attempts; for the third study, the world builder, grid runner and summary script gained a shock-distribution option and replicate runs, frozen with prereg-v3. The defaults reproduce the earlier worlds exactly, and no confirmatory number changed. During the third study the Anthropic account ran out of credit; we changed the Anthropic backend so that such errors stop the grid, and the 22 affected runs were redone, as the protocol prescribes for runs interrupted by a fatal error.

• •

  HAC-Holm candidates. prereg-v3 describes the HAC-Holm test as adjusting across the eight candidates; the frozen code, used unchanged in all studies, adjusts across nine, adding the asset’s trailing 12-month mean.

注意转载授权 · 阅读原文