Long-MDR：面向多模态深度研究智能体的 128k 上下文长程强化学习训练配方

Abstract

The next generation of multimodal research agents must reason over long-lived research histories rather than short model completions. During a single task, an agent may repeatedly search the web, inspect visual evidence, revisit earlier hypotheses, and accumulate tens of thousands of tokens of multimodal context. Despite this trend, online RL for multimodal research agents remains largely confined to shorter contexts and interaction horizons. We push online RL training to 128k context and 75+ tool-interaction turns. To our knowledge, this is the first online multimodal deep-research RL study trained at 128k context, and the first trained with a 75 tool-turn horizon.

Scaling to this regime exposes several practical limitations of conventional RL training. Early in training, weak policies make poor use of large interaction budgets, causing expensive rollouts with little reward improvement. Later, policy entropy can collapse before performance has saturated, prematurely ending useful learning. We introduce Long-MDR, a three-component training recipe designed specifically for this setting: On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue. Together, these techniques improve both the learning efficiency and stability of long-horizon RL, enabling continued gains in a regime where direct training is slow and costly.

Introduction

A straightforward path toward more capable research agents is to give them more room to research. Longer contexts allow agents to retain more evidence, while larger interaction budgets permit additional searches, webpage visits, visual inspections, and revisions. Recent multimodal research agents increasingly exploit this additional inference-time compute, showing substantial gains as the number and diversity of tool interactions increase [3, 14]. This raises an apparently simple question: if longer research trajectories are useful at inference time, can we simply train reinforcement-learning agents with proportionally longer trajectories?

Our experiments show that the answer is no.

We study online multimodal deep-research RL with 128k context and 75+ tool-interaction turns. To our knowledge, this is the first online multimodal deep-research RL study trained at 128k context and the first trained with a 75 tool-turn horizon. When we directly extend a conventional RL configuration to this scale, two behaviors become immediately apparent.

• •

  First, reward rises extremely slowly during the early stage of training. The policy is still weak, yet it is allowed to generate long and costly trajectories, resulting in large amounts of ineffective exploration.

• •

  Second, even after the policy has learned useful research behavior, training can terminate prematurely through entropy collapse: exploration drops sharply and reward plateaus before the model has exhausted the benefits of further optimization.

These phenomena make long-horizon RL particularly unattractive for multimodal deep research. A short mathematical reasoning rollout primarily costs generated tokens. A deep-research rollout may additionally involve visual preprocessing, search engines, webpage retrieval, multimodal context encoding, and tens of sequential environment interactions. Therefore, a recipe that eventually learns but spends many updates in an unproductive regime can be practically infeasible.

Based on these observations, we develop Long-MDR, a three-component recipe for scaling online RL to long research trajectories. It combines On-Policy Distillation Warmup to accelerate early learning, Progressive Horizon Expansion to gradually increase the interaction budget, and Entropy-Triggered Rescue to recover from premature exploration collapse.

Our contributions are threefold.

• •

  First, we scale online multimodal deep-research RL to 128k context and 75 tool-interaction turns, which, to our knowledge, is the first online multimodal deep-research RL study at 128k context and the first with a 75 tool-turn training horizon.

• •

  Second, we introduce Long-MDR, a simple and practical three-component training recipe for this regime.

• •

  Third, we analyze training dynamics to motivate practical choices for learning efficiency and stability in long-horizon multimodal deep-research RL.

Table 1 compares Long-MDR with published reference results across six benchmarks.

Preliminary

REINFORCE Leave-One-Out (RLOO)

Given an input x, a rollout policy πold independently samples G≥2 trajectories, each assigned a scalar reward ri. RLOO [4] measures a trajectory’s quality relative to the others sampled for the same input:

The baseline excludes trajectory i, so it is independent of that trajectory conditional on x. Subtracting it can reduce variance without changing the expected on-policy policy gradient, and requires no learned value model. A positive Ai favors the sampled trajectory; a negative Ai discourages it. The same trajectory-level advantage is assigned to each sampled action.

RLOO subtracts a baseline without dividing by the group reward standard deviation, so the advantage retains the scale of the reward gap. This differs from the group-standard-deviation normalization used in GRPO [5]. For example, two rewards (0.80,0.30) give advantages (+0.50,−0.50), while a gap of 0.02 gives (+0.02,−0.02). If all trajectories receive the same reward, their advantages are zero.

General Policy Loss

RLOO provides the advantages; the policy loss determines how they change action probabilities. Let si​t be the history before action ai​t in trajectory i. For a language model, each action is a generated token. The ratio

compares the action probability under the policy being updated with its probability under the policy that generated the rollout. A common clipped surrogate from PPO [6] is

where ε>0 is the clipping width. For A>0, increasing ρ beyond 1+ε yields no further gain; for A<0, decreasing ρ below 1−ε yields no further gain. Clipping limits the incentive for a large favorable update.

The policy loss minimizes the negative batch average of this surrogate:

Here ℬ denotes the empirical distribution over sampled action positions in a fixed rollout batch; the averaging weights can be specified independently of the advantage estimator. The negative sign turns reward maximization into loss minimization. Rewards, advantages, and rollout-policy probabilities are held fixed across both mini-batch actor updates from a rollout batch; gradients flow only through πθ.

Method

Staged Long-Horizon RL Recipe

Long-MDR combines On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue for online RL at 128k context and 75+ tool-interaction turns. Building on the RLOO objective in Sec. 2, training stage s minimizes

Here ℒOPD supplies teacher guidance, ℒKL penalizes divergence from the reference policy, and ℋ is assistant-token entropy, encouraged by the negative sign. All terms use the sequence-mean aggregation in Sec. 4.1.1. The stage-dependent coefficients control supervision and regularization, while the tool horizon grows as training progresses. We detail the three techniques below.

Learn Before Exploring: On-Policy Distillation Warmup

The first challenge is the cold start of long-horizon online RL. Although the policy has already undergone SFT, its early research trajectories remain unreliable, while outcome-based RL provides only a scalar signal after a potentially long sequence of reasoning and tool interactions. As a result, the initial reward improvement can be slow, making this stage particularly costly for multimodal deep research.

We therefore augment the early stages of training with On-Policy Distillation (OPD) from a stronger teacher policy [7, 8]. The student still generates complete trajectories using its current policy; the teacher only evaluates tokens that were actually sampled by the student. We request teacher log-probabilities only when the terminal outcome score satisfies a<1; fully correct trajectories skip teacher inference.

For each response token at, we compute the student–teacher log-probability gap

where πold is the student policy that generated the rollout and πT is the teacher. We compute the gap once per rollout batch and hold it fixed across both mini-batch actor updates. Using the trajectory reward r and a clipping bound c>0, we convert this gap into a reward-adaptive token-level OPD advantage,

and set AtOPD=0 for a≥1.

This construction has two effects.

• •

  First, the factor 1−r makes teacher guidance stronger for lower-reward trajectories and progressively weaker as the trajectory approaches full reward.

• •

  Second, clipping bounds every token-level OPD advantage to [−c,c], preventing poorly scored trajectories from producing excessively large distillation updates.

When the teacher assigns higher probability to a sampled token than the student (kt<0), the resulting OPD advantage is positive and encourages that token; when the student is more confident than the teacher (kt>0), the advantage becomes negative.

The complete token-level OPD advantage is detached before differentiation. For trajectory i and assistant token t, let A^i​tOPD=sg⁡(Ai​tOPD), where Ai​tOPD is the reward-adaptive, clipped advantage defined above and sg denotes stop-gradient. With the probability ratio in Eq. (2), the explicit OPD objective is

Here mi​t is one for an unmasked generated assistant token and zero otherwise, Ti=∑tmi​t is its trajectory’s assistant-token count, and B is the number of trajectories in the optimizer batch. Fully correct trajectories have A^i​tOPD=0 and remain in the batch average. The student–teacher log-probability gap used to construct the advantage, the reward, and the rollout-policy probabilities are treated as constants when differentiating this loss; gradients flow only through ρi​t​(θ). Thus, Eq. (6) uses the same clipped surrogate as Eq. (3) and the sequence-mean token-mean aggregation in Sec. 4.1.1. We combine the two losses as

Because the trajectories themselves remain sampled from the current student policy, OPD preserves the online training distribution while supplying a denser learning signal during the expensive early phase of RL.

OPD token-scoring requests and the OPD loss are disabled after the short warmup.

We use OPD as a temporary bootstrap signal: it is active for the first k training steps and disabled thereafter. Figure shows the initial experiment with ten steps of OPD, which motivates limiting teacher guidance to a short warmup rather than retaining it throughout training.

Grow the Horizon with the Policy: Progressive Horizon Expansion

A large interaction budget is necessary for difficult deep-research tasks, but it is inefficient when applied uniformly from the beginning of RL. Early in training, the policy has not yet learned reliable search and evidence-gathering behavior, so additional tool rounds mostly extend unsuccessful trajectories rather than create better learning signals. In multimodal research, this cost compounds: every extra interaction may invoke an external tool, append new observations, and increase the context processed by subsequent generations.

We therefore use Progressive Horizon Expansion, gradually increasing the maximum tool-interaction budget as the policy improves. Training starts with a smaller tool budget and expands to the full 75-turn horizon as the policy becomes capable of using longer trajectories.

The key idea is to match exploration capacity to policy capability. Shorter horizons provide a denser and cheaper learning regime early in training: the policy can first acquire basic search, tool-use, and evidence-selection behavior without paying for dozens of low-value interactions. Once these behaviors become reliable, increasing the horizon exposes the policy to harder trajectories that require deeper search, repeated verification, and hypothesis revision.

This curriculum also changes the quality of the RL signal. Under a fixed large horizon, early trajectories often contain long tails of low-quality actions after the policy has already lost the correct research direction. These tokens consume rollout compute but contribute little useful information to the terminal reward. Restricting the early horizon reduces such unproductive tails and concentrates training on decisions the current policy can realistically improve. Longer trajectories are introduced only after the policy can make meaningful use of them.

Progressive Horizon Expansion therefore serves two complementary purposes: it reduces the cost of early exploration and improves the effective learning signal per rollout. Later stages train directly with the full 75-turn budget; Figure traces validation performance during the main run, while Figure shows that the policy makes greater use of tools in later intervals. These complementary observations motivate increasing available interaction capacity as research behavior develops. Figure provides a contrasting run with OPD and a fixed 75-turn cap from the start: its early validation is only comparable to the 25-turn warmup run and subsequently deteriorates. The larger initial budget alone thus yields little early validation benefit, motivating a shorter initial horizon followed by staged expansion.

Train Past Collapse: Entropy-Triggered Rollback and Rescue

A third difficulty appears later in training. As the policy improves, its entropy naturally decreases; however, in our long-horizon runs we observe a qualitatively different regime in which entropy drops sharply while task reward is still below its eventual achievable level. After entering this regime, additional updates produce little improvement and the policy increasingly repeats a narrow set of research behaviors. We refer to this phenomenon as premature entropy collapse.

This effect is particularly consequential for long-horizon research agents. In a short reasoning trajectory, reduced token-level diversity affects only a small number of subsequent decisions. In a training trajectory of 75+ tool-interaction turns, however, an early low-entropy decision can determine the entire downstream search path: which entity is investigated, which evidence source is trusted, and which branch of the search space remains reachable. As the policy becomes more deterministic, this branching structure collapses as well. Thus, a modest reduction in local action entropy can translate into a much larger reduction in trajectory-level exploration.

Importantly, we do not treat every decrease in entropy as a failure. Entropy should decline when the policy becomes more confident in genuinely useful behaviors. We therefore identify collapse only when a sustained entropy decrease is accompanied by a plateau in outcome reward. The trigger is based on training statistics. Once collapse is detected, we roll back to the most recent checkpoint before the collapse.

From this checkpoint, we restore both policy and optimizer states, then resume training with a stronger entropy bonus and a lower learning rate. Training continues until reward reaches a new plateau.

The key insight is that policy convergence and entropy collapse are not the same event. A policy may appear converged simply because the current optimization trajectory has become too narrow to discover alternative research strategies.

A natural alternative is to apply entropy regularization from the beginning of RL. We use a targeted intervention for two reasons.

• •

  First, early training already contains substantial exploration, and additional entropy pressure can interfere with the exploitation of newly discovered successful behaviors.

• •

  Second, the appropriate exploration pressure changes over the course of training: what is beneficial after collapse can be unnecessarily disruptive before it.

Our approach therefore treats entropy regularization as a state-dependent intervention rather than a permanent objective term. Figure visualizes the resulting entropy, KL, and validation trajectories after checkpoint resumption.

This distinction is especially important in expensive multimodal research RL. A collapsed policy can continue generating long trajectories and consuming tool calls while repeatedly following nearly identical search strategies, creating the appearance of continued training without meaningful exploration. Entropy-Triggered Rollback and Rescue prevents this failure mode and extends the useful optimization horizon beyond the point at which the original run would otherwise stagnate.

Experiments

Training Details

RL Setup

Outcome supervision.

Our reinforcement-learning objective is built around an outcome-based reward. Given a completed research trajectory, an LLM judge scores the final answer against the ground-truth answer on a continuous scale ai∈[0,1]. Unlike deterministic verifiers commonly used in mathematical reasoning or coding RL, the judge supports partial credit, which is important for multimodal research tasks whose answers often consist of lists, descriptions, or multiple partially recoverable facts. Its system prompt specifies multi-hop research and grading strategies. Repeated judgments can vary slightly. We therefore use the mean-only RLOO advantages in Eq. (1), preserving absolute reward gaps without amplifying small judge fluctuations through standard-deviation normalization.

Trajectory reward.

Answer correctness remains the primary signal, but long rollouts can also waste compute through stalled reasoning or repeated search. We therefore combine the judge score ai with behavioral penalties pi≥0 and a completion bonus bi≥0:

The adjustments cover five behaviors:

• •

  Idle behavior: penalize unsuccessful trajectories that stop after very few tool interactions.

• •

  Generation-cap exhaustion: penalize reasoning turns that consume their generation budget without producing a tool action or a correct final answer.

• •

  Phrase looping: penalize repeated fallback expressions that indicate stalled reasoning.

• •

  Search repetition: penalize exact TextSearch repeats or queries with high token-level Jaccard overlap.

• •

  Unforced success: reward a fully correct answer when the environment has not injected a forced-tool instruction.

Idle, cap, and loop penalties apply only to unsuccessful trajectories; repeated searches are penalized even when the final answer is correct. The adjustments target these behaviors, allowing successful agents to finish early or continue researching when needed. Multiple tool calls in any round set the entire reward to zero. Valid-trajectory rewards are floored at zero after adding the bonus and subtracting penalties, so a violating trajectory cannot receive a higher reward than a valid trajectory solely because of a negative shaped reward. Positive shaped rewards are retained, and no upper clipping is applied.

RLOO computes advantages from the final reward ri. OPD eligibility instead uses the raw outcome score ai<1, while its teacher signal is weighted by 1−ri. Thus, a fully correct trajectory skips teacher inference even if a behavioral penalty lowers its training reward. This reward definition is shared across the three training techniques; coefficients and thresholds are specified in Sec. 4.1.2.

Loss aggregation.

Long research trajectories vary substantially in length. We instantiate the general loss in Eq. (4) by first averaging over valid assistant tokens within each trajectory, then over the B trajectories in the optimizer batch:

This sequence-mean token-mean aggregation gives each trajectory equal weight in the outer average; a longer rollout does not receive more weight solely because it contains more assistant tokens. We apply the same averaging to the distillation, KL, and entropy terms in Sec. 3.

Training Configuration

Initialization and runtime.

We initialize from Qwen3.5-9B [32] and perform one epoch of SFT on approximately 27,000 mixed-difficulty demonstrations comprising multimodal deep-research trajectories and search-agent data. The final corpus contains approximately 300M full-sequence tokens, including inputs and tool observations, and averages approximately 25 effective evidence-gathering rounds per trajectory across the corpus. Effective rounds count tool interactions that contribute evidence to the investigation. Before training, SFT demonstrations and RL prompts are screened against all six evaluation benchmarks using exact-text and image-hash matching followed by near-duplicate review. Online RL also uses 16 GPUs, with tensor parallelism =1; a separately served Qwen3.8-27B [33] supplies both outcome judgments and OPD teacher scores.

Rollouts and stage transitions.

We sample with temperature 1.0 and top-p=0.9. The reported checkpoint trajectory has four consecutive stages: steps 1–5, 6–15, 16–20, and 21–25. Each rollout step collects 32 prompts × 16 trajectories =512 trajectories, split into two disjoint mini-batches of 256 sequences, each used for one actor update. We hold πold, rewards, RLOO advantages, and OPD advantages fixed across both updates. After the first update, the second mini-batch is evaluated against the same rollout-policy probabilities, so its likelihood ratios can differ from one and enter the clipped regime. Training tool budgets are 25 turns at steps 1–5, 50 at steps 6–15, and 75 at steps 16–25. Each model context is capped at 128k tokens; the tool budget is an upper bound, and successful trajectories may end earlier. Step 25 denotes the reported checkpoint index; it does not count the total sampling work across the original and resumed branches, which has not been aggregated.

Evaluation budget.

All Long-MDR scores in Table 1 use a maximum of 50 tool-interaction turns, compared with the final 75-turn training budget. During formal evaluation, access to benchmark repositories and public answer pages is blocked. The BC-V³ validation value of 0.423 (42.3%) in Figures , , and uses a 75-turn evaluation budget, whereas Table 1 reports 35.7% under the 50-turn cutoff. These values therefore correspond to different interaction budgets. In Figure , BC-V³ @8 denotes correctness averaged over eight sampled responses per question and then over questions.

Table 2 separates the token context available during training from the tool-interaction limits used in training and evaluation.

OPD warmup and policy updates.

In the main run, OPD is enabled for the first k=5 rollout steps and disabled afterward. The initial comparison in Figure instead keeps OPD active for ten steps and is a separate experiment beyond their shared first five steps. We set λOPD=0.05 during warmup and zero afterward, and use the clipping bound c=2 in Sec. 3.1. OPD token-scoring requests are restricted to trajectories with raw outcome score ai<1; the sampled-token advantage is weighted by 1−ri, using the final trajectory reward, and detached before optimization. Fully correct trajectories skip OPD token scoring, and both these requests and the OPD term are disabled after warmup. Outcome judging continues throughout RL. Policy updates use mean-only RLOO and the assistant-token, sequence-mean aggregation in Sec. 4.1.1. The initial RL learning rate is 2×10−6.

Reward configuration.

Let zi=0 if any interaction contains multiple tool calls and zi=1 otherwise. The adjustment Δi=bi−pi from Sec. 4.1.1 is instantiated as follows:

The indicators follow the conditions in Sec. 4.1.1; the idle threshold is fewer than three tools on an unsuccessful trajectory. The zero floor is applied after all reward adjustments. Under these coefficients, valid rewards lie in [0,1.1], while a trajectory with a multiple-tool-call violation receives zero.

Entropy monitoring and rescue.

We track assistant-token entropy using the sequence-mean aggregation in Eq. (8) alongside the batch mean of the raw outcome score ai, ensuring that behavioral adjustments do not distort the plateau signal. Let H¯u and a¯u denote their moving averages over a window of w updates at step u. We automatically identify premature collapse when a sharp entropy decrease coincides with an outcome plateau:

for P consecutive checks, governed by window size w, patience P, and tolerances δH,δa>0. Once triggered, we restore both policy and optimizer states from the preceding pre-collapse checkpoint, increase the entropy coefficient to 5×10−3, and reduce the learning rate to 1.5×10−6.

Training Dynamics

We next examine how the policy evolves throughout the four-stage training process. The main training statistics include outcome reward, validation performance, policy entropy, trajectory length, and tool usage. Across training stages, the interaction budget is progressively increased and, later, entropy rescue is activated. These trends matter computationally as well as statistically: an unproductive training step still requires collecting a costly batch of multimodal trajectories.

The first stage exhibits the characteristic cold-start regime of long-horizon RL. Starting from the SFT policy, reward is initially low and improves only gradually under sparse outcome supervision. OPD warmup substantially shortens this regime by providing token-level guidance on suboptimal trajectories. Once useful search behavior begins to emerge, the policy transitions from relying primarily on teacher-assisted corrections toward improvement driven by the online reward signal. Importantly, the reward increase is accompanied by improved validation performance rather than merely higher shaped training reward, suggesting that the early gains correspond to genuinely better research behavior.

Figure reports the initial ten-step OPD/no-OPD experiment. OPD performs better at both validation checkpoints, but its validation score declines with further training despite lower OPD loss. Only its first five OPD steps are shared with the main run in Figure . This motivates a finite warmup: the main run uses k=5 and disables OPD afterward.

As the training tool horizon expands from 25 to 50 and subsequently to 75 turns, the agent is exposed to progressively harder and longer trajectories. Each horizon increase initially raises average trajectory length and tool usage, but does not cause a sustained degradation in reward. Instead, the policy gradually learns to exploit the additional interaction capacity, and performance continues to improve after each expansion. This behavior supports the motivation for Progressive Horizon Expansion: the full interaction budget becomes useful only after the policy has acquired sufficiently reliable short- and medium-horizon research skills. Starting directly at the largest horizon would pay the computational cost of these long trajectories before the policy is capable of benefiting from them.

Figures and connect the main-run validation trend to tool-use behavior: mean tool counts rise from 14.3 over steps 1–5 to 21.7 over steps 21–25, with fluctuations. Figure provides a contrasting OPD run with a fixed 75-turn cap from the outset. Its early validation is numerically close to the 25-turn OPD run in Figure , then declines despite decreasing OPD loss and entropy. A large initial interaction budget therefore offers little early validation benefit, motivating the progressive schedule. Its computational motivation is to reserve the largest trajectory budgets for stages where the policy can exploit them.

An important pattern emerges later in training. While reward continues to improve initially, policy entropy steadily decreases and eventually enters a sharp collapse regime. At this point, reward and validation performance stop improving despite continued optimization. Meanwhile, the policy becomes increasingly concentrated on a narrow set of research actions and trajectories. The later gains after rescue show that useful learning can resume beyond this plateau.

Entropy-Triggered Rescue supports this interpretation. After rolling back to a pre-collapse checkpoint and resuming training with stronger entropy regularization, policy entropy recovers and reward begins to improve again. The resumed gains occur despite starting from an earlier checkpoint, highlighting the value of revisiting a checkpoint with restored exploration. Instead, restoring exploration allows the policy to discover research strategies that were no longer reachable from the collapsed policy. The subsequent increase in validation performance further indicates that the recovered entropy corresponds to useful behavioral diversity.

Figure highlights the phase where we apply the intervention in Sec. 3.3. The resumed continuation shows eventual entropy recovery, slower KL growth, and higher validation performance than the original continuation. With expensive rollouts, this intervention has a practical role: it redirects subsequent data collection toward a policy that can recover useful exploration when continued optimization has stopped improving validation.

Figure makes the cost of this training regime concrete. The generation/rollout stage alone averages 14.8 minutes per step over steps 1–5 and 25.3 minutes over steps 6–25, reaching 36.2 minutes at step 17; parameter updates are excluded. A single step can therefore spend over half an hour collecting experience before policy optimization. At this cost, a prolonged cold start or a validation plateau can consume substantial training time with limited progress. These measurements strengthen the practical motivation for our recipe: establish useful behavior before allocating large interaction budgets, and address later exploration failures before collecting many more expensive rollouts.

Figure shows that mean tool calls in both trajectory-outcome groups and response length rise early and then fluctuate. Incorrect trajectories use more tool calls on average than correct trajectories at every plotted step, highlighting the interaction cost of unsuccessful search. Figure further shows changing tool selection: image search remains dominant, fetch usage is generally higher later, and Web usage recovers after a mid-training decline, while RIS and Crop do not follow the same late increase. We assess changes in tool use and response length alongside validation gains.

Overall, the training dynamics reveal three distinct phases: bootstrap, expansion, and rescue. OPD primarily accelerates the emergence of useful behavior during the bootstrap phase; progressive horizon expansion increases the complexity of the research problems the policy can tackle; and entropy rescue extends optimization after exploration would otherwise collapse. The components address different sources of wasted training effort: weak initial behavior, premature exposure to long trajectories, and stalled exploration. Their coordination is particularly valuable in a regime where rollout stages routinely take tens of minutes.

Related work

Search-R1 and R1-Searcher train reasoning–search policies with outcome-based RL [18, 19]. WebDancer, WebSailor, DeepResearcher, Tongyi DeepResearch, and REDSearcher explore supervised initialization, task synthesis, and interactive RL for research agents [20, 21, 22, 2, 12]. Rufus-Air includes a text-based Search Agent RL stage with up to 100 tool calls per rollout [34].

Multimodal agents combine visual evidence with retrieval and browsing. Vision-DeepResearch, OpenSearch-VL, and MMSearch-R1 train these capabilities through RL [3, 15, 25]. WebWatcher, POINTS-Seeker, and WeAgent-MMSearch develop multimodal browsing, visual memory, and persistent image references, respectively [29, 11, 13]. HyperEyes, SearchEyes, and VSearcher study retrieval efficiency, simulated search environments, and long-horizon RL [9, 16, 17]. DeepVoyager-VL uses synthesized trajectories without RL [10], while LMM-Searcher extends inference to 100 turns through context management and distillation [14]. MMSearch-OPD [35] combines controllable refinement of multimodal questions with joint RL and OPD.

Our focus is online multimodal RL at 128k context and 75+ tool-interaction turns. The recipe connects teacher feedback on student-generated sequences [7, 8], curriculum learning [24], and exploration control [1, 23] through OPD warmup, progressive horizon expansion, and entropy-triggered rescue.

Conclusion

We study online reinforcement learning for multimodal deep-research agents in a substantially longer regime than prior work, scaling training to 128k context and 75+ tool-interaction turns. At this scale, simply extending a conventional RL recipe is inefficient: early policies make poor use of long and expensive trajectories, while later optimization can plateau under entropy collapse before performance has fully saturated.

To address these challenges, we introduce a practical training recipe combining On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue. Together, these techniques accelerate early learning, reduce wasted long-horizon rollouts, sustain exploration during later training, and enable stable optimization over extended multimodal research histories.

Our results show that scaling multimodal research agents to longer horizons is not merely a matter of increasing context length or inference-time tool budgets. The training process itself must adapt to the changing cost and optimization dynamics of long trajectories. We hope this study provides a practical foundation for training future research agents that can learn effectively from increasingly long, interactive, and multimodal research processes.

References

• [1] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025. arXiv:2503.14476v2.
• [2] Tongyi DeepResearch Team. Tongyi DeepResearch Technical Report. 2025; revised 2026. arXiv:2510.24701v3.
• [3] Wenxuan Huang et al. Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models. 2026. arXiv:2601.22060.
• [4] Arash Ahmadian et al. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. 2024. arXiv:2402.14740.
• [5] Zhihong Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. arXiv:2402.03300.
• [6] John Schulman et al. Proximal Policy Optimization Algorithms. 2017. arXiv:1707.06347.
• [7] Rishabh Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR, 2024. arXiv:2306.13649.
• [8] Kevin Lu and Thinking Machines Lab. On-Policy Distillation. 2025. Technical note.
• [9] Guankai Li et al. HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents. 2026. arXiv:2605.07177.
• [10] Huanyao Zhang et al. DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents. 2026. arXiv:2608.01827v1.
• [11] Yikun Liu et al. POINTS-Seeker: Towards Training a Multimodal Agentic Search Model from Scratch. 2026. arXiv:2604.14029v1.
• [12] Zheng Chu et al. REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents. 2026. arXiv:2602.14234.
• [13] Zongkai Liu et al. WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents. 2026. arXiv:2608.28062.
• [14] Yifan Du et al. Towards Long-horizon Agentic Multimodal Search. 2026. arXiv:2604.12890.
• [15] Shuang Chen et al. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. 2026. arXiv:2605.05185.
• [16] Zhengbo Jiao et al. SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulation. 2026. arXiv:2607.05943.
• [17] Ruiyang Zhang et al. VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning. 2026. arXiv:2603.02795.
• [18] Bowen Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. 2025. arXiv:2503.09516.
• [19] Huatong Song et al. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. 2025. arXiv:2503.05592.
• [20] Jialong Wu et al. WebDancer: Towards Autonomous Information Seeking Agency. 2025. arXiv:2505.22648.
• [21] Kuan Li et al. WebSailor: Navigating Super-human Reasoning for Web Agent. 2025. arXiv:2507.02592.
• [22] Yuxiang Zheng et al. DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments. 2025. arXiv:2504.03160.
• [23] Ganqu Cui et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. 2025. arXiv:2505.22617.
• [24] Yoshua Bengio et al. Curriculum Learning. ICML, 2009. Official ICML paper.
• [25] Jinming Wu et al. MMSearch-R1: Incentivizing LMMs to Search. 2025. arXiv:2506.20670.
• [26] Dongzhi Jiang et al. MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines. ICLR, 2025. arXiv:2409.12959.
• [27] Xianfu Cheng et al. SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models. ICCV, 2025. arXiv:2502.13059.
• [28] Mingyang Fu et al. LiveVQA: Live Visual Knowledge Seeking. 2025. arXiv:2504.05288v1.
• [29] Xinyu Geng et al. WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent. 2025. arXiv:2508.05748.
• [30] Huanyao Zhang et al. BrowseComp-V³: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents. 2026. arXiv:2602.12876.
• [31] Zhengbo Zhang et al. VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents. 2026. arXiv:2603.16289.
• [32] Qwen Team. Qwen3.5-9B. 2026. Official model card.
• [33] Qwen Team. Qwen3.8-27B. 2026. Official model card.
• [34] Chia-Yuan Chang et al. Rufus-Air: An Open LLM Post-Training Recipe. 2026. arXiv:2609.29421v1.
• [35] Huanyao Zhang, Jiepeng Zhou, Jiaye Lin, Ontai Tang, Jingtian Hao, Mugeng Liu, Ru Peng, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, and Wentao Zhang. MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents. 2026. arXiv:2610.22240.

Appendix A Case Study: A 75-Round Search Trajectory

This case is a training rollout sampled at step 16, when the training tool budget is 75 turns. It is not a formal evaluation trajectory contributing to Table 1.

Question. At the hotel where the event shown took place, how many red life rings hang on the white railing beside the cliff-side infinity pool in the sunset photograph?

Ground-truth answer. 2.

Model answer. There are 2 red life rings hanging on the white railing beside the cliff-side infinity pool at the Hotel du Cap-Eden-Roc in the sunset photograph.

Tool usage. The 75 numbered entries contain 35 TextSearch, 21 ImageSearch, 9 ReverseImageSearch, 6 WebVisit, 4 CropImage, and 0 FetchImage calls.

Input and outcome.

Initial searches pursue person identities and wedding or breakup coverage. The last two entries search for the pool scene using visual attributes and then target Hotel du Cap-Eden-Roc; the final answer matches the ground truth.

Reading the trace. Parameters and returned excerpts are reproduced with formatting changes only. The markers [bin] and [REDACTED], existing ellipses, and excerpts ending mid-sentence or mid-token are retained from the raw log. No omitted text is reconstructed. Red text marks selected actions discussed after the table.

Case Study: Opportunistic Search.

Table A.1 presents a 75-tool-call trajectory selected for its unusual search behavior. The agent initially explores several person identities and event locations, repeatedly revisiting weakly supported hypotheses. Meanwhile, it searches for scene attributes such as the infinity pool, white railing, and red life rings from call 12 onward. These routes remain interleaved throughout the episode, with recurring person-based queries and even identical cropping parameters at calls 17 and 71. The trajectory thus reveals persistent exploration accompanied by substantial repetition and limited consolidation of earlier attempts.

A particularly interesting behavior is the emergence of benchmark-directed answer lookup. At call 30, a search using the input image’s URL returns a BrowseComp-V3 page. The agent then incorporates the benchmark name into its queries and visits GitHub and Hugging Face pages with the explicit goal of locating the question and its answer. Similar attempts recur at calls 56–59. This sequence illustrates how a retrieved result can redirect the search toward the provenance of the task itself: the benchmark becomes another potential source of an answer. The agent also issues queries containing candidate counts, including “two” and “3” red life rings. Together, these actions show the policy using entity hypotheses, scene descriptions, candidate answers, and task provenance as alternative retrieval cues.

Task-relevant retrieval occurs near the end. At call 74, the query "red life rings" infinity pool cliff returns a result associated with Hotel du Cap-Eden-Roc. A hotel-specific image search follows at call 75, after which the agent answers 2, matching the ground truth. This late progress emerges from a scene-based search route that the agent had already attempted earlier. The case illustrates the coexistence of persistent exploration, opportunistic answer lookup, repeated hypotheses, and eventual retrieval progress. Its interest lies in the diversity of search behaviors underlying a single correct final response.

注意转载授权 · 阅读原文