second brain
source
← 首页

external-source

UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning

UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.20089 · published 2026-09-17 · submitted 2026-09-17

Abstract

Self-evolving methods reduce the need for human-annotated trajectories by allowing tool-using agents to generate their own training data. Yet existing methods typically separate trajectory generation from evaluation, relying on static verifiers that cannot adapt to emerging failure modes or self-consistency signals that may reinforce errors shared across trajectories. Jointly adapting planning, execution, and evaluation offers a promising alternative, but introduces a fundamental coordination challenge: each component continuously changes the data or feedback used to train the others. We address this challenge with \textbf{UnifiedPlayers}, a cooperative framework comprising a Planning Player that generates tasks, an Execution Player that produces multi-turn trajectories with Python tool calls, and an Evaluation Player that constructs executable verifiers. We design role-specific rewards that coordinate the three players toward a shared learning objective under GRPO. Across two model backbones and twelve reasoning benchmarks, UnifiedPlayers outperforms the strongest prior baseline by at least 3.5% on mathematical reasoning and 3.9% on general reasoning tasks. Moreover, the learned verifier achieves 84.2% adversarial detection accuracy, while its reward signal exhibits 2.03$\times$ higher per-question variance than a self-consistency baseline, providing more discriminative verifications. These results highlight cooperation among specialized players as a promising path toward self-enhanced tool-integrated agents.

Authors

Wenjie Liao, Liangjie Zhao, Zehong Cao

Key claims (verbatim-leaning English extract)

Structure (section headings from HTML)

Body excerpts (original English)

UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Preliminaries 3 Method 3.1 Unified Cooperation under Executable Feedback 3.2 Cooperative Players with Role-Specific Objectives Planning Player. Execution Player. Evaluation Player. Reward alignment across players. 3.3 Alternating Cooperative Optimization Player-wise optimization. Block-wise optimization stability. 4 Experiments 4.1 Experimental Setup Implementation Details. Baseline Methods and Evaluation Datasets. 4.2 Main Performance Results 4.3 Ablation Studies Individual Player contributions. Fresh vs. stale feedback. Component ablations. 4.4 Optimization Stability 4.5 Mechanism Analysis and Qualitative Case Study 5 Conclusion References A Related Work A.1 Tool-Integrated Reasoning A.2 Self-Generated Data and Adaptive Exploration A.3 Executable Verification and Multi-Player Optimization B Methodology Details B.1 Cross-Player Interaction Records and Executable Feedback B.2 The Objectives of UnifiedPlayers B.3 Complete Policy Optimization Definitions B.4 Update Order within UnifiedPlayers Planning block. Execution block. Evaluation block. C Evaluation Metrics C.1 Player Coordination Protocol Factorial configurations. Paired contrasts. Fresh versus stale feedback. Scope of the comparisons. C.2 Coupled Learning Dynamics D Experimental Setup Details D.1 Hyperparameter Settings D.2 Optimization Stability Diagnostics D.3 Sandbox Configuration E Prompt Templates E.1 Planning Player Prompt E.2 Execution Player Prompt E.3 Evaluation Player Prompt F Perturbation Engine Details F.1 Five Mutation Rules F.2 Before-After Examples G Baselines and Benchmarks G.1 Baselines Overview G.2 Evaluation Benchmarks H Additional Results and Analysis H.1 Per-Iteration Training Process H.2 Compute Cost Comparison H.3 Perturbation Rule Subsets H.4 Verifier Pool Size (M) Sweep H.5 Reward Coefficient Sensitivity I Case Analysis I.1 Generated Questions across Iterations I.2 Execution Player Trajectory Examples I.3 Illustrative Verifier Checks Input parsing. Verifiers for Trajectory 2 (GCF plus eleven). Verifiers for Trajectory 4 (sum of first N N odd numbers). Verifiers for Trajectory 5 (Greek army battalions). I.4 Perturbation Pairs Pairs on Trajectory 1 (divisor counting). Pairs on Trajectory 2 (GCF plus eleven). Pairs on Trajectory 4 (sum of first N N odd numbers). Pairs on Trajectory 5 (Greek army battalions). License: arXiv.org perpetual non-exclusive license arXiv:2609.20089v1 [cs.AI] 17 Sep 2026 UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning Wenjie Liao Affiliation: Waseda University Email: jie3040@akane.waseda.jp Liangjie Zhao Affiliation: Adelaide University Email: zhaoliangjie55@gmail.com Zehong Cao Affiliation: Adelaide University Email: Jimmy.Cao@adelaide.edu.au Abstract Self-evolving methods reduce the need for human-annotated trajectories by allowing tool-using agents to generate their own training data. Yet existing methods typically separate trajectory generation from evaluation, relying on static verifiers that cannot adapt to emerging failure modes or self-consistency signals that may reinforce errors shared across trajectories. Jointly adapting planning, execution, and evaluation offers a promising alternative, but introduces a fundamental coordination challenge: each component continuously changes the data or feedback used to train the others. We address this challenge with UnifiedPlayers , a cooperative framework comprising a Planning Player that generates tasks, an Execution Player that produces multi-turn trajectories with Python tool calls, and an Evaluation Player that constructs executable verifiers. We design role-specific rewards that coordinate the three players toward a shared learning objective under GRPO. Across two model backbones and twelve reasoning benchmarks, UnifiedPlayers outperforms the strongest prior baseline by at least 3.5% on mathematical reasoning and 3.9% on general reasoning tasks. Moreover, the learned verifier achieves 84.2% adversarial detection accuracy, while its reward signal exhibits 2.03 × \times higher per-question variance than a self-consistency baseline, providing more discriminative verifications. These results highlight cooperation among specialized players as a promising path toward self-enhanced tool-integrated agents. Figure 1: Overall performance of UnifiedPlayers . (a) Across three outer iterations on Qwen3-4B, planning difficulty U ⁡ ( q ) U(q) , execution accuracy, and evaluation adversarial-rejection rate rise monotonically under cooperative policy optimization. These trends do not isolate individual player contributions. (b) UnifiedPlayers outperforms tool-augmented Qwen3-4B across five benchmarks. 1 Introduction Large language model agents increasingly plan, act, and use tools across multi-step interactions with their environments ( Yao et al., 2023 ; Schick et al., 2023 ; Wu et al., 2024 ; Qin et al., 2024 ) . Training these agents requires execution trajectories that capture tool calls, observations, error recovery, and final decisions ( Yao et al., 2025 ; Barres et al., 2026 ; Jin et al., 2025 ) . Constructing reliable evaluation data requires expert input and careful review, as illustrated by Humanity’s Last Exam ( Phan et al., 2026 ) . Existing corpora such as Self-Instruct and UltraChat focus on instruction following and dialogue rather than executable tool interactions ( Wang et al., 2023b ; Ding et al., 2023 ) . Obtaining realistic trajectories at the scale required for reinforcement learning (RL) therefore remains a bottleneck for agent training ( Liu et al., 2024 ; Prabhakar et al., 2025 ; Xu et al., 2025 ) . Self-generated data offers a way to reduce this dependence on manual trajectories. A broad line of work lets language models synthesize instructions, solutions, or interaction data for their own improvement ( Zelikman et al., 2022 ; Wang et al., 2023b ; Qiao et al., 2024 ; Zhao et al., 2025 ) . TaskCraft expands atomic tasks into structurally complex trajectories ( Shi et al., 2026 ) . AReaL-SEA generates interactions together with per-instance executable verifiers and reuses them as RL rewards ( Gao et al., 2026 ) . These approaches reduce the need for manually authored training trajectories. More recent work trains task generators to adapt with solvers. For example, Agent0 ( Xia et al., 2025 ) trains its Curriculum and Executor policies with group-relative policy optimization (GRPO) ( Shao et al., 2024 ) . The Curriculum proposes tool-integrated tasks, while the Executor solves them through multi-turn code execution ( Gou et al., 2024 ; Feng et al., 2026 ; Chen et al., 2025 ) . Their supervision remains self-referential, because the Curriculum reward uses Executor self-consistency and tool-call frequency, while the Executor learns from majority-vote pseudo-labels over the same samples ( Wang et al., 2023a ) . Appendix A compares recent tool-integrated RL, self-evolving agents, and verifier-based training. Neither signal verifies execution: agreement on a wrong answer can therefore produce incorrect labels and misleading advantage updates ( Huang et al., 2026 ) . Therefore, existing work primarily adapts task generation and execution, while evaluation signals remain derived from Executor outputs or LLM judgments ( Zheng et al., 2023 ) . Simply making evaluation trainable does not resolve this limitation: once planning, execution, and evaluation adapt jointly, each policy changes the learning environment of the other two. Specifically, Planning shifts the task distribution, Execution changes the resulting trajectories and failure modes, and Evaluation alters the rewards that guide both. We refer to this circular dependency as cyclic interdependence . Without explicit coordination, such interdependence can induce degenerate learning loops: Planning may favor easy, repetitive, or unverifiable tasks; Execution may reinforce confidently incorrect trajectories accepted by the evaluator; and Evaluation may collapse to trivial rules that accept all trajectories, reject all trajectories, or repeatedly generate similar verifier patterns. These failures can mutually reinforce one another, increasing internal rewards without producing genuine improvements in tool-integrated reasoning. We address this issue with UnifiedPlayers , an executable-feedback-mediated cooperative framework in which all players are unified through role-specific rewards grounded in a shared source of executable evidence ( Guo et al., 2025 ) . The Planning Player ( Task-Setter ) generates tool-using tasks, the Execution Player ( Solver ) produces multi-turn trajectories with Python tool calls, and the Evaluation Player ( Critic-Coder ) constructs executable verifiers. Specifically, for each reasoning task, we design a fixed perturbation engine to generate adversarial trajectory variants, while a sandbox executes every verifier against both the original and perturbed trajectories. The resulting verdict matrix provides shared executable evidence of task verifiability, trajectory correctness, and verifier discrimination. We summarize this shared evidence differently for each player to construct its role-specific reward, with an additional code-diversity term for the Evaluation Player to discourage repetitive verifier patterns. To stabilize learning under cyclic interdependence, we optimize one player at a time with GRPO while keeping the other two fixed, ensuring that the policies generating its training data and feedback remain unchanged within each update block. Based on our experiments across two backbones and twelve benchmarks, UnifiedPlayers exceeds the strongest baseline by at least 3.5% on mathematical reasoning and 3.9% on general reasoning tasks. Its verifiers achieve 84.2% discrimination accuracy, while the resulting rewards exhibit 2.03 × \times the per-question variance of Agent0’s offline self-consistency feedback ( Xia et al., 2025 ) . Factorial comparisons show Qwen3-4B-Base attribute gains of 2.5% and 2.3% on mathematical and general reasoning to effective planning, respectively, and 3.3% on both to evaluation outcomes. Moreover, fresh feedback improves performance by 1.5% and 1.7% over a compute-matched control using feedback from the previous iteration, demonstrating the benefit of our designed evaluation. Our contributions are threefold. (1) We introduce UnifiedPlayers, a cooperative multi-player RL framework unifying planning, tool-integrated execution, and evaluation through shared executable feedback. (2) We derive role-specific rewards from a shared verdict matrix produced by executing learned verifiers and alternate policy updates while holding the other players fixed. (3) We train executable evaluation with adversarial trajectory perturbations and improve reasoning across two backbones and twelve benchmarks, with accurate verifiers and more discriminative rewards. 2 Preliminaries Multi-turn tool-integrated reasoning. Given a reasoning task q q , an agent interacts with an environment over multiple turns to produce a trajectory τ = ( a 1 , o 1 , a 2 , o 2 , … , a T ) \tau=(a_{1},o_{1},a_{2},o_{2},\ldots,a_{T}) ( Yao et al., 2023 ; Schick et al., 2023 ) . Each action a t a_{t} is either a tool call or a final answer that terminates the trajectory. Tools may include external APIs ( Qin et al., 2024 ) or code interpreters ( Feng et al., 2026 ) . In our UnifiedPlayers setting, tool calls execute Python code in a sandbox and return stdout, stderr, and exit status as observations o t o_{t} , which are supported by related applications such as multi-hop question answering ( Jin et al., 2025 ) and function calling ( Patil et al., 2025 ; Lu et al., 2026 ) . Reinforcement learning with verifiable rewards. Reinforcement learning with verifiable rewards (RLVR) trains the policy using explicit correctness checks instead of learned reward predictions ( Lambert et al., 2025 ; Guo et al., 2025 ) . A verifier v ⁡ ( τ ) ∈ { 0 , 1 } v(\tau)\in{0,1} evaluates τ \tau and provides a binary terminal reward. Checks include code unit tests, math answer matching ( Cobbe et al., 2021 ; Shao et al., 2024 ) , and environment state comparisons ( Yao et al., 2025 ) , grounding rewards in observable outcomes. A recent work AReaL-SEA ( Gao et al., 2026 ) , generates verification functions alongside synthetic tasks and uses them to compute RL rewards. In our UnifiedPlayers setting, the Evaluation Player learns to generate verifiers for new Execution Player trajectories and errors. 3 Method Figure 2: UnifiedPlayers Overview. Planning generates tasks, Execution produces tool-integrated trajectories, and Evaluation writes executable verifiers. A fixed perturbation engine and Python sandbox map verifier outcomes on original and adversarial trajectories to a shared verdict matrix. The resulting role-specific rewards support alternating GRPO updates for all players. 3.1 Unified Cooperation under Executable Feedback As shown in Figure 2 , UnifiedPlayers coordinates three trainable policies through shared executable feedback. The Planning Player π T \pi_{T} generates a task q q . Given q q , the Execution Player π S \pi_{S} samples K K trajectories with tool calls, and the Evaluation Player π C \pi_{C} generates M M executable verifiers. A fixed perturbation engine creates P P variants per trajectory. The sandbox runs each verifier that parses and compiles on all original and perturbed trajectories. Let ℐ valid ​ ( q ) \mathcal{I}{\mathrm{valid}}(q) index these valid verifiers and M q = | ℐ valid ​ ( q ) | M{q}=|\mathcal{I}{\mathrm{valid}}(q)| . Their binary outcomes form the shared verdict matrix 𝐙 ⁡ ( q ) = [ 𝐙 orig ∣ 𝐙 adv ] \mathbf{Z}(q)=[\mathbf{Z}^{\mathrm{orig}}\mid\mathbf{Z}^{\mathrm{adv}}] , where rows index valid verifiers, columns index trajectories, and one denotes acceptance. We first collect each task’s evidence in a cooperative interaction record for policy optimization, ξ ⁡ ( q ) = ( q , 𝒯 ⁡ ( q ) , 𝒱 ⁡ ( q ) , 𝒯 adv ​ ( q ) , 𝐙 ⁡ ( q ) , 𝝆 red ​ ( q ) ) , \xi(q)=\bigl(q,\mathcal{T}(q),\mathcal{V}(q),\mathcal{T}^{\mathrm{adv}}(q),\mathbf{Z}(q),\bm{\rho}^{\mathrm{red}}(q)\bigr), (1) where 𝒯 ⁡ ( q ) \mathcal{T}(q) , 𝒱 ⁡ ( q ) \mathcal{V}(q) , and 𝒯 adv ​ ( q ) \mathcal{T}^{\mathrm{adv}}(q) contain the original trajectories, valid verifiers, and perturbed trajectories, respectively. 𝝆 red ​ ( q ) \bm{\rho}^{\mathrm{red}}(q) contains redundancy scores computed from verifier source code. Each player derives its role-specific reward from a different summary of the shared verdict matrix: original-trajectory acceptance for Execution, original acceptance and perturbation rejection for Evaluation, and task verifiability for Planning. This shared evidence coordinates the players, which retain separate policies and are optimized in turn with GRPO.Appendix B.1 provides the complete construction of the verdict matrix and interaction records. 3.2 Cooperative Players with Role-Specific Objectives Planning Player. We use a fixed question prompt to generate a nontrivial multistep task q q and a boxed answer used only for data filtering ( Huang et al., 2026 ) . Its reward favors tasks near the Execution Player’s capability boundary, valid discriminative verifiers, and novelty, weighted by w unc w{\text{unc}} , w ver w_{\text{ver}} , and w rep w_{\text{rep}} , respectively: r T ​ ( q ) = w unc ​ U ​ ( q ) + w ver ​ V ​ ( q ) − w rep ​ R ​ ( q ) . r_{T}(q)=w_{\text{unc}}U(q)+w_{\text{ver}}V(q)-w_{\text{rep}}R(q). (2) Here, U ⁡ ( q ) = min ⁡ ( p ^ , 1 − p ^ ) U(q)=\min(\hat{p},1-\hat{p}) measures uncertainty across K K sampled Execution trajectories, where p ^ \hat{p} denotes the fraction sharing the majority answer, while R ⁡ ( q ) R(q) penalizes repetition of previously generated questions. We define verifier writability as: V ( q ) = m ( q ) ⋅ 1 M q ∑ i ∈ ℐ valid ​ ( q ) ρ i adv , V(q)=m(q)\cdot\frac{1}{M_{q}}\sum_{i\in\mathcal{I}{\mathrm{valid}}(q)}\rho{i}^{\mathrm{adv}}, (3) where ℐ valid ​ ( q ) \mathcal{I}{\mathrm{valid}}(q) denotes the set of valid verifiers among M M generated verifiers, M q = | ℐ valid ​ ( q ) | M{q}=|\mathcal{I}{\mathrm{valid}}(q)| , m ⁡ ( q ) = M q / M m(q)=M{q}/M is the valid verifier fraction, and ρ i adv \rho_{i}^{\mathrm{adv}} is verifier i i ’s adversarial rejection rate. We set V ⁡ ( q ) = 0 V(q)=0 if no verifier is valid or no valid verifier rejects a perturbation. Execution Player. Upon receiving a multistep task q q from the Planning Player, the Execution Player interleaves reasoning with Python tool calls, using each tool output as the next observation until it produces a boxed answer or reaches the turn limit. Valid verifiers then evaluate the complete trajectory, with their mean acceptance defining the reward as: r S ​ ( τ k ) = 1 M q ​ ∑ i ∈ ℐ valid ​ ( q ) v i ​ ( τ k ) . r_{S}(\tau_{k})=\frac{1}{M_{q}}\sum_{i\in\mathcal{I}{\mathrm{valid}}(q)}v{i}(\tau_{k}). (4) This averages the corresponding column of 𝐙 orig \mathbf{Z}^{\mathrm{orig}} . Executed checks replace majority-vote pseudo-labels ( Xia et al., 2025 ) , so agreement among Execution samples does not guarantee a high reward. Evaluation Player. It generates M M Python verifier candidates, each mapping a trajectory to a pass/fail verdict and the earliest failing step. Parsing and syntax checks determine the valid verifier set ℐ valid ​ ( q ) \mathcal{I}{\mathrm{valid}}(q) before sandbox execution. For each valid verifier v i v{i} , the verdict matrix yields the original-pass rate ρ i gold = K − 1 ​ ∑ k Z orig ​ i , k \rho_{i}^{\mathrm{gold}}=K^{-1}\sum_{k}Z^{\mathrm{orig}}{i,k} and adversarial-rejection rate ρ i adv = ( K ​ P ) − 1 ​ ∑ k , j ⁡ ( 1 − Z i , k , j adv ) \rho_{i}^{\mathrm{adv}}=(KP)^{-1}\sum{k,j}(1-Z^{\mathrm{adv}}{i,k,j}) , where K K is the number of original trajectories and P P is the number of perturbations per trajectory. We further compute redundancy ρ i red \rho{i}^{\mathrm{red}} as the maximum source-code similarity to any other valid verifier. The Evaluation Player reward is r C ​ ( v i ) = α ​ ρ i gold + β ​ ρ i adv − γ ​ ρ i red , r_{C}(v_{i})=\alpha\rho^{\text{gold}}{i}+\beta\rho^{\text{adv}}{i}-\gamma\rho^{\text{red}}{i}, (5) where α \alpha , β \beta , and γ \gamma weight original acceptance, adversarial rejection, and verifier redundancy, respectively. The first two terms reward accepting originals and rejecting perturbations, while the last discourages repetitive verifier code. An accept-all verifier receives no adversarial reward, a reject-all verifier no original-pass reward, and repeated templates incur redundancy penalties. Reward alignment across players. We model UnifiedPlayers as a stochastic game 𝒢 = ( 𝒫 , 𝝅 , 𝒳 , 𝐉 ) \mathcal{G}=(\mathcal{P},\bm{\pi},\mathcal{X},\mathbf{J}) with three players denoted by 𝒫 = T , S , C \mathcal{P}={T,S,C} . Their joint policy is 𝝅 = ( π T , π S , π C ) \bm{\pi}=(\pi{T},\pi_{S},\pi_{C}) and utility vector is 𝐉 = ( J T , J S , J C ) \mathbf{J}=(J_{T},J_{S},J_{C}) , while the interaction process 𝒳 \mathcal{X} comprises tasks, trajectories, verifiers, perturbations, and sandbox verdicts. Each player maximizes its own expected reward while the other two policies remain fixed during its update. Cooperation therefore does not require a shared scalar reward; instead, the players are coupled through shared executable evidence. The role-specific rewards align these interactions by discouraging degenerate behavior from each player. For Planning, easy tasks reduce uncertainty U ⁡ ( q ) U(q) and repeated tasks incur the penalty R ⁡ ( q ) R(q) , encouraging challenging and novel tasks. For Execution, trajectories rejected by valid verifiers receive lower rewards, encouraging solutions that satisfy executable checks. For Evaluation, accept-all verifiers receive no adversarial-rejection reward, reject-all verifiers receive no original-acceptance reward, and repetitive verifiers incur a redundancy penalty. Consequently, improving one player changes the evidence and learning signal available to the others, creating a cooperative learning process despite their distinct rewards. Appendix B.2 formalizes the corresponding player objectives. 3.3 Alternating Cooperative Optimization UnifiedPlayers maintains separate parameters and role-specific rewards for each player, in contrast to multi-agent GRPO ( Chen et al., 2026 ) , which uses shared parameters and a global reward augmented with role-specific format rewards. In each outer iteration, we sequentially update Planning, Execution, and Evaluation while holding the other two fixed. Updated Planning immediately generates tasks for Execution, and updated Execution generates trajectories for Evaluation. Because Evaluation is last, its updated checkpoint first supplies verifier evidence in the next iteration. Later blocks therefore use the latest preceding-player checkpoints while co-player policies remain fixed within each block. Appendix B.4 details the complete update and evidence-refresh schedule. For an active player p p in block b b , the dataset 𝒟 p ( b ) \mathcal{D}{p}^{(b)} contains groups of G p G{p} candidate outputs sampled under the same context. Specifically, Planning samples candidate tasks from a fixed questioner prompt, Execution samples trajectories for a given task q q , and Evaluation samples verifiers conditioned on q q and the trajectory schema. Their rewards are computed from the interaction records in Eq. 1 , with each group containing outputs only from the active player. The advantage A ^ p \widehat{A}{p} is obtained by normalizing each candidate’s reward by the mean and standard deviation of its group (Appendix B.3 ). For each token, let ϱ p \varrho{p} be its probability under the current policy divided by its probability under the behavior policy, conditioned on the same context and preceding tokens. Let ϱ ¯ p \bar{\varrho}{p} denote this ratio clipped to [ 1 − ϵ clip , 1 + ϵ clip ] [1-\epsilon{\mathrm{clip}},1+\epsilon_{\mathrm{clip}}] . The GRPO objective is 𝒥 p ( b ) ​ ( θ p ) = 𝔼 ⁡ [ min ⁡ ( ϱ p ​ A ^ p , ϱ ¯ p ​ A ^ p ) − β p ​ D p KL ] . \mathcal{J}{p}^{(b)}(\theta{p})=\mathbb{E}!\left[\min(\varrho_{p}\widehat{A}{p},\bar{\varrho}{p}\widehat{A}{p})-\beta{p}D_{p}^{\mathrm{KL}}\right]. (6) The expectation averages over groups in 𝒟 p ( b ) \mathcal{D}{p}^{(b)} , candidates within each group, and tokens within each candidate. Here, D p KL D{p}^{\mathrm{KL}} measures the divergence between the current policy and its reference checkpoint saved at the start of each outer iteration, with β p \beta_{p} controlling the KL penalty. Candidate and token indices are omitted for brevity. Let b ℓ , p b_{\ell,p} denote the update block of player p p in iteration ℓ \ell . We summarize the three player-wise objectives as: 𝒥 UP ( ℓ ) = 1 3 ​ ∑ p ∈ T , S , C 𝒥 p ( b ℓ , p ) ​ ( θ p ) . \mathcal{J}{\mathrm{UP}}^{(\ell)}=\frac{1}{3}\sum{p\in{T,S,C}}\mathcal{J}{p}^{(b{\ell,p})}(\theta_{p}). (7) This mean serves only as an iteration-level summary: each objective is optimized independently within its corresponding block, without combining rewards or averaging gradients across players. Since group normalization makes advantage magnitudes incomparable across players or systems, we instead assess optimization stability using realized KL, clipping fraction, gradient statistics, policy entropy, and checkpoint-level performance changes. Player-wise optimization. During player p p ‘s update block, we treat its sampled outputs, sandbox outcomes, and resulting rewards as fixed records, denoted by sg ⁡ [ 𝒟 p ( b ) ] \operatorname{sg}[\mathcal{D}{p}^{(b)}] . The other two player policies, behavior policies, and reference checkpoints are also held fixed. Under these conditions, Eq. 6 reduces to a clipped GRPO surrogate over only the active policy π p \pi{p} . Since the three players have separate parameters, the iteration-level objective in Eq. 7 therefore satisfies: ∇ θ p 𝒥 UP ( ℓ ) = 1 3 ​ ∇ θ p 𝒥 p ( b ℓ , p ) ​ ( θ p , sg ⁡ [ 𝒟 p ( b ℓ , p ) ] ) . \nabla_{\theta_{p}}\mathcal{J}{\mathrm{UP}}^{(\ell)}=\frac{1}{3}\nabla{\theta_{p}}\mathcal{J}{p}^{(b{\ell,p})}\left(\theta_{p};\operatorname{sg}\left[\mathcal{D}{p}^{(b{\ell,p})}\right]\right). (8) Thus, each block updates only the active player’s parameters from its fixed interaction records, without propagating gradients through the other players or the data-generation process. This establishes the block-wise optimization structure, but does not imply monotonic improvement. Block-wise optimization stability. The stationarity here is conditional on fixed interaction records and auxiliary evidence, including the verifier pool and question history used to construct rewards. Within an update block, these quantities are held fixed, so the reward targets associated with a given record remain unchanged during optimization. This stationarity is local to the block: once executable evidence is refreshed for the next block or iteration, the resulting records and reward targets may change. We further characterize how far an active policy can move from its reference within such a block. Clipping and KL regularization discourage large policy updates but do not impose a hard constraint. For a fixed context distribution, let D ¯ KL , p \bar{D}{\mathrm{KL},p} and T ¯ p \bar{T}{p} denote the average KL divergence and total variation distance, respectively, between the updated and reference output distributions. If D ¯ KL , p ≤ δ p \bar{D}{\mathrm{KL},p}\leq\delta{p} , Pinsker’s inequality gives T ¯ p ≤ δ p / 2 . \bar{T}{p}\leq\sqrt{\delta{p}/2}. This conditional result assumes output-distribution KL and does not follow from the token-level KL training diagnostic. Table 1: Results on mathematical reasoning benchmarks. Best per backbone is in bold . Model Name AMC MATH GSM8K Minerva Olympiad AIME24 AIME25 AVG Qwen3-4B-Base Vanilla ✗ ✗ ✗ 45.4 68.6 88.5 37.3 41.2 11.0 6.33 42.6 Tool-augmented ✓ ✗ ✗ 45.9 72.7 88.8 38.1 42.4 12.5 7.86 44.0 - Absolute Zero ✓ ✗ ✗ 50.3 76.8 88.8 40.6 42.9 12.7 13.0 46.4 - R-Zero ✗ ✗ ✗ 57.7 80.2 90.5 52.4 43.6 12.3 4.45 48.7 - Socratic-Zero ✗ ✓ ✗ 57.7 80.3 88.5 47.3 49.2 19.4 13.8 50.9 - Agent0 ✓ ✗ ✗ 60.4 79.9 91.1 56.7 48.4 17.3 14.9 52.7 - UnifiedPlayers (ours) ✓ ✗ ✓ 64.5 83.9 95.1 59.6 51.3 21.8 18.7 56.4 MiMo-7B-Base Vanilla ✗ ✗ ✗ 58.3 75.6 89.9 47.3 46.6 30.7 22.3 53.0 Tool-augmented ✓ ✗ ✗ 63.6 77.9 91.1 51.4 50.2 34.4 26.9 56.5 - Absolute Zero ✓ ✗ ✗ 64.2 77.0 91.6 50.0 49.1 33.8 26.6 56.0 - R-Zero ✗ ✗ ✗ 65.0 80.5 93.3 57.2 51.3 33.0 28.6 58.4 - Socratic-Zero ✗ ✓ ✗ 66.5 79.3 90.4 52.5 55.8 39.1 34.0 59.7 - Agent0 ✓ ✗ ✗ 67.3 81.4 94.2 57.9 54.1 39.5 32.4 61.0 - UnifiedPlayers (ours) ✓ ✗ ✓ 71.4 84.9 96.2 61.5 58.1 43.0 36.8 64.6 4 Experiments 4.1 Experimental Setup Implementation Details. We evaluate UnifiedPlayers on two open-source base models, Qwen3-4B-Base ( Yang et al., 2025 ) and MiMo-7B-Base ( Xiaomi LLM-Core Team, 2025 ) , with all three policies initialized from the same backbone. The framework is implemented on top of VeRL ( Sheng et al., 2025 ) with a sandboxed Python interpreter provided by VeRL-Tool ( Jiang et al., 2025 ) that executes the Execution Player’s code blocks in a subprocess and returns their standard output as the next observation. For each Planning Player task, we sample K = 8 K=8 Execution Player trajectories and ask the Evaluation Player to write M = 4 M=4 Python verifiers, each restricted to the standard library with a strict per-call timeout. The perturbation engine applies all five mutation rules independently, and the sandbox runs every verifier against the original and the five adversarial variants. The Evaluation Player reward then combines these verdicts with coefficients α = 0.4 \alpha=0.4 , β = 0.5 \beta=0.5 , γ = 0.1 \gamma=0.1 for the original-pass, adversarial-rejection, and redundancy terms. Each outer iteration consists of three sequential GRPO update blocks, one per policy, and we run three outer iterations. Table 2: Results on general-domain reasoning benchmarks. Best per backbone is in bold . Model Name SuperGPQA MMLU-Pro BBEH GPQA-D HumanEval AVG Qwen3-4B-Base Vanilla ✗ ✗ ✗ 21.1 37.6 7.6 36.5 65.7 33.7 Tool-augmented ✓ ✗ ✗ 25.8 43.6 8.5 37.2 67.1 36.4 - Absolute Zero ✓ ✗ ✗ 26.4 52.9 8.7 38.4 68.4 39.0 - R-Zero ✗ ✗ ✗ 28.3 51.4 10.2 39.6 69.6 39.8 - Socratic-Zero ✗ ✓ ✗ 28.5 54.2 10.8 40.1 70.2 40.8 - Agent0 ✓ ✗ ✗ 30.4 55.1 11.5 41.7 71.8 42.1 - UnifiedPlayers (ours) ✓ ✗ ✓ 33.3 59.6 15.8 45.2 76.7 46.1 MiMo-7B-Base Vanilla ✗ ✗ ✗ 24.9 42.3 8.5 26.1 52.3 30.8 Tool-augmented ✓ ✗ ✗ 26.4 44.2 9.4 27.4 54.1 32.3 - Absolute Zero ✓ ✗ ✗ 27.6 48.6 10.0 29.2 55.8 34.2 - R-Zero ✗ ✗ ✗ 28.8 48.4 10.7 31.0 57.0 35.2 - Socratic-Zero ✗ ✓ ✗ 29.2 49.1 11.1 31.7 57.6 35.7 - Agent0 ✓ ✗ ✗ 30.5 51.3 12.2 33.5 59.2 37.3 - UnifiedPlayers (ours) ✓ ✗ ✓ 33.4 57.0 14.9 37.3 63.5 41.2 Table 3: Player attribution on Qwen3-4B-Base with trainable Execution. Scores summarize three paired seeds (mean ± \pm SD). GPU-hours per seed. Stale feedback lags one outer iteration. Configuration Planning Evaluation Math AVG General AVG GPU-h Execution only Frozen Frozen 50.6 ± 1.1 50.6\pm 1.1 40.5 ± 0.8 40.5\pm 0.8 312 Planning + Execution Trainable Frozen 52.9 ± 0.6 52.9\pm 0.6 42.3 ± 0.5 42.3\pm 0.5 360 Execution + Evaluation Frozen Trainable 53.7 ± 1.4 53.7\pm 1.4 43.3 ± 1.0 43.3\pm 1.0 384 Full UnifiedPlayers Trainable Trainable 56.4 ± 0.8 \mathbf{56.4\pm 0.8} 46.1 ± 0.4 \mathbf{46.1\pm 0.4} 432 Planning marginal Average effect + 2.5 ± 0.4 +2.5\pm 0.4 + 2.3 ± 0.3 +2.3\pm 0.3 – Evaluation marginal Average effect + 3.3 ± 0.8 +3.3\pm 0.8 + 3.3 ± 0.5 +3.3\pm 0.5 – Interaction Difference-in-differences + 0.4 ± 0.3 +0.4\pm 0.3 + 1.0 ± 0.4 +1.0\pm 0.4 – Coordination control Stale-feedback UnifiedPlayers Trainable Trainable 54.9 ± 0.9 54.9\pm 0.9 44.4 ± 0.7 44.4\pm 0.7 432 Fresh-feedback gain Matched capacity + 1.5 ± 0.3 +1.5\pm 0.3 + 1.7 ± 0.2 +1.7\pm 0.2 Matched Baseline Methods and Evaluation Datasets. Our six baselines comprise two untrained references, Vanilla and Tool-augmented, three other self-evolving frameworks, Absolute Zero ( Zhao et al., 2025 ) , R-Zero ( Huang et al., 2026 ) , and Socratic-Zero ( Wang et al., 2025 ) , and the two-player Agent0 framework ( Xia et al., 2025 ) . Vanilla evaluates the original backbone without tools, whereas Tool-augmented enables the same Python tool interface at inference without RL training. We use our reimplementations for all baselines and evaluate them with the same backbone-specific prompts, decoding settings, benchmark splits, and tool sandbox whenever the method uses tools. We evaluate seven math benchmarks (AMC, MATH ( Hendrycks et al., 2021 ) , GSM8K ( Cobbe et al., 2021 ) , Minerva ( Lewkowycz et al., 2022 ) , Olympiad-Bench ( He et al., 2024 ) , AIME24, AIME25) and five general-domain benchmarks (SuperGPQA ( M-A-P Team et al., 2025 ) , MMLU-Pro ( Wang et al., 2024 ) , BBEH ( Kazemi et al., 2025 ) , GPQA-D ( Rein et al., 2024 ) , HumanEval ( Chen et al., 2021 ) ), using greedy pass@1 except mean@32 for AMC and AIME. 4.2 Main Performance Results Tables 1 and 2 report results across seven mathematical and five general-domain reasoning benchmarks. UnifiedPlayers achieves the best average performance on both model backbones, with consistent gains across individual benchmarks. On mathematical reasoning, UnifiedPlayers obtains average scores of 56.4 with Qwen3-4B-Base and 64.6 with MiMo-7B-Base, outperforming Agent0, the strongest baseline, by 3.7 and 3.6, respectively. With Qwen3-4B-Base, it further surpasses the Tool-augmented reference by 12.4 on average, with substantial gains on challenging benchmarks such as AIME24 and AIME25. Figure 1 (b) further shows an average improvement of 10.9 over this reference across five representative math benchmarks. Furthermore, across the five general-domain benchmarks, UnifiedPlayers achieves averages of 46.1 and 41.2 on Qwen3-4B-Base and MiMo-7B-Base, respectively, improving over Agent0 by 4.0 and 3.9. A particularly notable gain appears on BBEH with Qwen3-4B-Base, where all prior baselines score below 12. Importantly, although the Planning Player generates only mathematical tasks during training, the consistent improvements across general-domain benchmarks indicate transfer beyond the training distribution. 4.3 Ablation Studies Individual Player contributions. Table 3 reports a 2 × 2 2\times 2 factorial study on Qwen3-4B-Base, where Execution is always trained while Planning and Evaluation are independently trained or frozen under the same generation settings. All configurations use three outer iterations, matched task and sampling budgets, and three paired seeds, without reallocating saved compute to Execution. Averaged over the other player’s setting, training Planning improves mathematical and general reasoning by 2.5 and 2.3, respectively, while training Evaluation yields larger gains of 3.3 on both domains. Their interaction contributes an additional 0.4 on math and 1.0 on general reasoning, suggesting complementarity without establishing statistically significant super-additivity. Appendix C.1 details the paired contrasts and coordination control. Table 4: Qwen3-4B-Base Execution diagnostics. Grad. reports median pre-clipping norm and CV. Max drop measures maximum adjacent-checkpoint accuracy decline (points). Configuration Grad. Med./CV KL Clip Entropy Neg. Steps Max Drop Execution only 0.86 / 0.37 0.018 0.16 0.91 0.24 1.1 Planning + Execution 0.79 / 0.35 0.016 0.13 0.91 0.20 0.9 Execution + Evaluation 0.77 / 0.32 0.016 0.15 0.90 0.17 0.8 Full UnifiedPlayers 0.73 / 0.31 0.014 0.09 0.87 0.19 0.6 Figure 3: Reward and verifier comparison. (a, b) UnifiedPlayers has 2.03 times the per-question reward variance of offline Agent0-based self-consistency; (c) Verifier accuracy in distinguishing original from perturbed trajectories approaches GPT-4o without external API calls during training. Table 5: Component ablations of UnifiedPlayers. Method Math AVG General AVG Full UnifiedPlayers 56.4 46.1 w/o Perturbation Engine 51.6 41.3 Single Verifier ( M = 1 M=1 ) 54.3 44.2 w/o V ⁡ ( q ) V(q) in Planning reward 54.2 43.9 w/o Diversity Penalty 54.2 44.6 Single Iteration 47.4 38.0 Fresh vs. stale feedback. We examine whether sharing updated checkpoints within each iteration improves cross-player coordination. UnifiedPlayers immediately uses each updated player to generate fresh feedback for the next, whereas the stale-feedback control delays all updated checkpoints until the next iteration. With matched players, compute, rewards, sampling, and update budgets, fresh feedback improves mathematical and general reasoning by 1.5 and 1.7, respectively. These results support the benefit of within-iteration feedback refresh, while isolating feedback freshness rather than comparing against all independent training strategies. Component ablations. All players remain trainable while we vary feedback components and the training schedule (Table 5 ). One outer iteration causes the largest drop, 9.0 on math and 8.1 on general reasoning. Removing the Perturbation Engine lowers both averages by 4.8. Using one verifier or removing V ⁡ ( q ) V(q) from the Planning reward reduces performance by about two points. Tables 3 and 5 distinguish adaptive player contributions from component effects. Both roles contribute, while adversarial verifier training and repeated outer iterations have the largest component effects. These comparisons identify no player as the sole contributor. Appendix H provides iteration results and analyses of perturbation subsets, verifier-pool size, and reward weights. 4.4 Optimization Stability Because group normalization rescales advantages, we compare Execution Player diagnostics instead (Table 4 ). Full UnifiedPlayers has lower gradient variability, realized KL, clipping frequency, and maximum checkpoint decline than all three controls. Its median pre-clipping gradient norm is 0.73, KL is 0.014, and clipping fraction is 0.09. Evaluated every 50 updates on a fixed 256-question pool excluded from model selection, its largest checkpoint decline is 0.6. However, entropy retention is lower at 87%, and negative steps are slightly more frequent than with Execution + Evaluation. These findings suggest reduced variability on several metrics, but do not establish uniformly better stability or convergence. Further details appear in Appendix D.2 . 4.5 Mechanism Analysis and Qualitative Case Study Across three Qwen3-4B-Base iterations (Figure 1 (a)), Execution accuracy rises from 52.5% to 63.3%, Evaluation adversarial rejection from 31% to 79%, and Planning uncertainty U ⁡ ( q ) U(q) from 0.18 to 0.39, indicating harder tasks. Table 3 isolates player contributions and fresh coordination. On a shared question pool, UnifiedPlayers yields 2.03 × \times the reward variance of offline Agent0 self-consistency (Figure 3 (a,b)), while the Evaluation Player generates over 95% of top-scoring verifiers. Figure 3 (c) shows 84.2% accuracy in distinguishing original from perturbed trajectories, approaching GPT-4o’s 90.8% without external training APIs. This indicates differentiated feedback. Figure 4: Representative outputs from UnifiedPlayers. (a) The Planning Player generates increasingly challenging tasks across three outer iterations. (b) In a separate held-out MATH-500 example, the Execution Player reasons about a digit-constrained multiple, calls Python, and returns 2220. (c) The Evaluation Player writes an executable verifier that accepts the original trajectory and rejects an arg_mutation variant that changes 2220 to 2200. Figure 4 illustrates the players. Panel (a) shows harder tasks across iterations. Panel (b) shows Execution solving a held-out MATH-500 problem with Python, returning 2220. Panel (c) shows an Evaluation verifier accepting the original trajectory but rejecting an arg_mutation from 2220 to 2200. Together, they show UnifiedPlayers’ task–execution–evaluation cycle. 5 Conclusion UnifiedPlayers coordinates Planning, Execution, and Evaluation through shared executable feedback, while maintaining separate rewards, parameters, and alternating GRPO updates. Across two backbones and twelve benchmarks, it consistently outperforms the strongest baselines, while learned verifiers reach 84.2% discrimination accuracy without external API calls during training. Factorial and fresh-versus-stale comparisons further support the contributions of adaptive players and within-iteration feedback. Future work will explore adaptive adversarial generation, larger models and broader agent settings, and more efficient sandbox evaluation. References Barres et al. (2026) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan τ 2 \tau^{2} -Bench: evaluating conversational agents in a dual-control environment . In International Conference on Machine Learning , External Links: Link Cited by: §1 . Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code . arXiv preprint arXiv:2107.03374 . External Links: Link Cited by: §G.2 , §4.1 . Chen et al. (2026) Y. Chen, J. Feng, W. Yang, M. Zhong, Z. Shi, R. Li, X. Wei, Y. Gao, Y. Wu, Y. Hu, Z. Pu, and J. Mao Self-compression of chain-of-thought via multi-agent reinforcement learning . arXiv preprint arXiv:2601.21919 . External Links: Link Cited by: §3.3 . Chen et al. (2025) Z. Chen, J. Yang, T. Xiao, R. Zhou, L. Zhang, X. Xi, X. Shi, W. Wang, and J. Wang Reinforcement learning for tool-integrated interleaved thinking towards cross-domain generalization . arXiv preprint arXiv:2510.11184 . External Links: Link Cited by: §A.1 , §1 . Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168 . External Links: Link Cited by: §A.3 , §G.2 , §2 , §4.1 . Ding et al. (2023) N. Ding, Y. Chen, B. Xu, Y. Qin, S. Hu, Z. Liu, M. Sun, and B. Zhou Enhancing chat language models by scaling high-quality instructional conversations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 3029–3051 . External Links: Document , Link Cited by: §1 . Dong et al. (2025) G. Dong, Y. Chen, X. Li, J. Jin, H. Qian, Y. Zhu, H. Mao, G. Zhou, Z. Dou, and J. Wen Tool-Star: empowering LLM-brained multi-tool reasoner via reinforcement learning . arXiv preprint arXiv:2505.16410 . External Links: Link Cited by: §A.1 . Feng et al. (2026) J. Feng, S. Huang, X. Qu, G. Zhang, Y. Qin, B. Zhong, C. Jiang, J. Chi, and W. Zhong ReTool: reinforcement learning for strategic tool use in LLMs . In International Conference on Learning Representations , External Links: Link Cited by: §A.1 , §1 , §2 . Gao et al. (2026) J. Gao, J. Chen, C. He, S. Xu, D. Jin, and Y. Wu From self-evolving synthetic data to verifiable-reward RL: post-training multi-turn interactive tool-using agents . arXiv preprint arXiv:2601.22607 . External Links: Link Cited by: §A.2 , §A.3 , §1 , §2 . Gou et al. (2024) Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, M. Huang, N. Duan, and W. Chen ToRA: a tool-integrated reasoning agent for mathematical problem solving . In International Conference on Learning Representations , pp. 48362–48395 . External Links: Link Cited by: §A.1 , §1 . Guo et al. (2026) B. Guo, S. Shi, Y. Zhao, T. Ma, G. Zhu, Y. He, P. Zhang, and X. Li E3-TIR: enhanced experience exploitation for tool-integrated reasoning . In Findings of the Association for Computational Linguistics: ACL 2026 , pp. 24575–24596 . External Links: Document , Link Cited by: §A.2 . Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning . Nature 645 ( 8081 ), pp. 633–638 . External Links: Document , Link Cited by: §A.3 , §1 , §2 . He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun OlympiadBench: a challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 3828–3850 . External Links: Document , Link Cited by: §G.2 , §4.1 . Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset . In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , Vol. 1 . External Links: Link Cited by: §G.2 , §4.1 . Huang et al. (2026) C. Huang, W. Yu, X. Wang, H. Zhang, Z. Li, R. Li, J. Huang, H. Mi, and D. Yu R-Zero: self-evolving reasoning LLM from zero data . In International Conference on Learning Representations , External Links: Link Cited by: §A.2 , §G.1 , §1 , §3.2 , §4.1 . Hugging Face H4 (2024) Hugging Face H4 AIME 2024 . Note: Hugging Face datasetAccessed September 17, 2026 External Links: Link Cited by: §G.2 . Jiang et al. (2025) D. Jiang, Y. Lu, Z. Li, Z. Lyu, P. Nie, H. Wang, A. Su, H. Chen, K. Zou, C. Du, T. Pang, and W. Chen VerlTool: towards holistic agentic reinforcement learning with tool use . arXiv preprint arXiv:2509.01055 . External Links: Link Cited by: §D.3 , §4.1 . Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. O. Arik, D. Wang, H. Zamani, and J. Han Search-R1: training LLMs to reason and leverage search engines with reinforcement learning . In Conference on Language Modeling , External Links: Link Cited by: §1 , §2 . Kazemi et al. (2025) M. Kazemi, B. Fatemi, H. Bansal, J. Palowitch, C. Anastasiou, S. V. Mehta, L. K. Jain, V. Aglietti, D. Jindal, P. Chen, N. Dikkala, G. Tyen, X. Liu, U. Shalit, S. Chiappa, K. Olszewska, Y. Tay, V. Q. Tran, Q. V. Le, and O. Firat BIG-Bench extra hard . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 26473–26501 . External Links: Document , Link Cited by: §G.2 , §4.1 . Lambert et al. (2025) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. Tülu 3: pushing frontiers in open language model post-training . In Conference on Language Modeling , External Links: Link Cited by: §2 . Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra Solving quantitative reasoning problems with language models . In Advances in Neural Information Processing Systems , Vol. 35 , pp. 3843–3857 . External Links: Document , Link Cited by: §G.2 , §4.1 . Li et al. (2025a) P. Li, Z. Gao, B. Zhang, Y. Mi, X. (. Ma, C. Shi, T. Yuan, Y. Wu, Y. Jia, S. Zhu, and Q. Li Iterative tool usage exploration for multimodal agents via step-wise preference tuning . In Advances in Neural Information Processing Systems , Vol. 38 , pp. 59496–59528 . External Links: Document , Link Cited by: §A.2 . Li et al. (2025b) X. Li, H. Zou, and P. Liu ToRL: scaling tool-integrated RL . arXiv preprint arXiv:2503.23383 . External Links: Link Cited by: §A.1 . Lin and Xu (2025) H. Lin and Z. Xu Understanding tool-integrated reasoning . arXiv preprint arXiv:2508.19201 . External Links: Link Cited by: §A.1 . Liu et al. (2024) Z. Liu, T. Hoang, J. Zhang, M. Zhu, T. Lan, S. Kokane, J. Tan, W. Yao, Z. Liu, Y. Feng, R. Murthy, L. Yang, S. Savarese, J. C. Niebles, H. Wang, S. Heinecke, and C. Xiong APIGen: automated pipeline for generating verifiable and diverse function-calling datasets . In Advances in Neural Information Processing Systems , Vol. 37 , pp. 54463–54482 . External Links: Document , Link Cited by: §1 . Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization . In International Conference on Learning Representations , External Links: Link Cited by: §D.1 . Lu et al. (2026) S. Lu, Z. Wang, H. Zhang, Q. Wu, L. Gan, C. Zhuang, J. Gu, and T. Lin Don’t just fine-tune the agent, tune the environment . In International Conference on Learning Representations , External Links: Link Cited by: §2 . M-A-P Team et al. (2025) M-A-P Team, X. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, et al. SuperGPQA: scaling LLM evaluation across 285 graduate disciplines . In Advances in Neural Information Processing Systems , Vol. 38 . External Links: Document , Link Cited by: §G.2 , §4.1 . Math AI (2023) Math AI AMC23 . Note: Hugging Face datasetAccessed September 17, 2026 External Links: Link Cited by: §G.2 . Math AI (2025) Math AI AIME 2025 . Note: Hugging Face datasetAccessed September 17, 2026 External Links: Link Cited by: §G.2 . Patil et al. (2025) S. G. Patil, H. Mao, F. Yan, C. C. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez The berkeley function calling leaderboard (BFCL): from tool use to agentic evaluation of large language models . In International Conference on Machine Learning , Proceedings of Machine Learning Research , Vol. 267 , pp. 48371–48392 . External Links: Link Cited by: §2 . Phan et al. (2026) L. Phan, A. Gatti, N. Li, A. Khoja, R. Kim, R. Ren, J. Hausenloy, O. Zhang, M. Mazeika, D. Hendrycks, et al. A benchmark of expert-level academic questions to assess AI capabilities . Nature 649 ( 8099 ), pp. 1139–1146 . External Links: Document , Link Cited by: §1 . Prabhakar et al. (2025) A. Prabhakar, Z. Liu, M. Zhu, J. Zhang, T. M. Awalgaonkar, S. Wang, Z. Liu, H. Chen, T. Hoang, J. C. Niebles, S. Heinecke, W. Yao, H. Wang, S. Savarese, and C. Xiong APIGen-MT: agentic pipeline for multi-turn data generation via simulated agent-human interplay . In Advances in Neural Information Processing Systems , Vol. 38 . External Links: Document , Link Cited by: §1 . Qiao et al. (2024) S. Qiao, N. Zhang, R. Fang, Y. Luo, W. Zhou, Y. Jiang, C. Lv, and H. Chen AutoAct: automatic agent learning from scratch for QA via self-planning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 3003–3021 . External Links: Document , Link Cited by: §A.2 , §1 . Qin et al. (2024) Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun ToolLLM: facilitating large language models to master 16000+ real-world APIs . In International Conference on Learning Representations , pp. 9695–9717 . External Links: Link Cited by: §1 , §2 . Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof Q&A benchmark . In Conference on Language Modeling , External Links: Link Cited by: §G.2 , §4.1 . Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools . In Advances in Neural Information Processing Systems , Vol. 36 , pp. 68539–68551 . External Links: Document , Link Cited by: §1 , §2 . Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models . arXiv preprint arXiv:2402.03300 . External Links: Link Cited by: §A.3 , §1 , §2 . Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient RLHF framework . In Proceedings of the Twentieth European Conference on Computer Systems , pp. 1279–1297 . External Links: Document , Link Cited by: §4.1 . Shi et al. (2026) D. Shi, J. Cao, Q. Chen, W. Sun, W. Li, H. Lu, F. Dong, T. Qin, K. Zhu, M. Liu, Y. E. Jiang, J. Yang, G. Zhang, J. Liu, C. Zhang, J. Wang, and W. Zhou TaskCraft: automated generation of agentic tasks . In International Conference on Learning Representations , External Links: Link Cited by: §A.2 , §1 . Wang et al. (2025) S. Wang, Z. Jiao, Z. Zhang, Y. Peng, X. Ze, B. Yang, W. Wang, H. Wei, and L. Zhang Socratic-Zero: bootstrapping reasoning via data-free agent co-evolution . arXiv preprint arXiv:2509.24726 . External Links: Link Cited by: §A.2 , §G.1 , §4.1 . Wang et al. (2023a) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models . In International Conference on Learning Representations , External Links: Link Cited by: §1 . Wang et al. (2023b) Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi Self-instruct: aligning language models with self-generated instructions . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 13484–13508 . External Links: Document , Link Cited by: §A.2 , §1 , §1 . Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen MMLU-Pro: a more robust and challenging multi-task language understanding benchmark . In Advances in Neural Information Processing Systems , Vol. 37 , pp. 95266–95290 . External Links: Document , Link Cited by: §G.2 , §4.1 . Wei et al. (2025) Y. Wei, X. Yu, Y. Weng, T. Pan, A. Li, and L. Du AutoTIR: autonomous tools integrated reasoning via reinforcement learning . arXiv preprint arXiv:2507.21836 . External Links: Link Cited by: §A.1 . Wu et al. (2024) Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang AutoGen: enabling next-gen LLM applications via multi-agent conversations . In Conference on Language Modeling , External Links: Link Cited by: §1 . Xia et al. (2025) P. Xia, K. Zeng, J. Liu, C. Qin, F. Wu, Y. Zhou, C. Xiong, and H. Yao Agent0: unleashing self-evolving agents from zero data via tool-integrated reasoning . arXiv preprint arXiv:2511.16043 . External Links: Link Cited by: §A.2 , §E.1 , §G.1 , §1 , §1 , §3.2 , §4.1 . Xiaomi LLM-Core Team (2025) Xiaomi LLM-Core Team MiMo: unlocking the reasoning potential of language model—from pretraining to posttraining . arXiv preprint arXiv:2505.07608 . External Links: Link Cited by: §4.1 . Xu et al. (2025) Z. Xu, A. Meza Soria, S. Tan, A. Roy, A. S. Agrawal, R. Poovendran, and R. Panda TOUCAN: synthesizing 1.5M tool-agentic data from real-world MCP environments . arXiv preprint arXiv:2510.01179 . External Links: Link Cited by: §1 . Xue et al. (2026) Z. Xue, L. Zheng, Q. Liu, Y. Li, X. Zheng, Z. Ma, and B. An SimpleTIR: end-to-end reinforcement learning for multi-turn tool-integrated reasoning . In International Conference on Learning Representations , pp. 8424–8449 . External Links: Link Cited by: §A.1 . Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report . arXiv preprint arXiv:2505.09388 . External Links: Link Cited by: §4.1 . Yao et al. (2025) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ \tau -bench: a benchmark for tool-agent-user interaction in real-world domains . In International Conference on Learning Representations , pp. 9965–10017 . External Links: Link Cited by: §1 , §2 . Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models . In International Conference on Learning Representations , External Links: Link Cited by: §1 , §2 . Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman STaR: bootstrapping reasoning with reasoning . In Advances in Neural Information Processing Systems , Vol. 35 , pp. 15476–15488 . External Links: Document , Link Cited by: §A.2 , §1 . Zhang et al. (2026a) S. Zhang, Y. Dong, J. Zhang, J. Kautz, B. Catanzaro, A. Tao, Q. Wu, Z. Yu, and G. Liu Nemotron-Research-Tool-N1: exploring tool-using language models with reinforced reasoning . In International Conference on Learning Representations , pp. 91437–91453 . External Links: Link Cited by: §A.1 , §A.3 . Zhang et al. (2026b) S. Zhang, Y. Xiong, X. Chen, Z. Jia, R. Huang, J. Xu, and J. Zhang RAPO: expanding exploration for LLM agents via retrieval-augmented policy optimization . In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 , pp. 6546–6557 . External Links: Document , Link Cited by: §A.2 . Zhao et al. (2025) A. Zhao, Y. Wu, T. Wu, Q. Xu, Y. Yue, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang Absolute zero: reinforced self-play reasoning with zero data . In Advances in Neural Information Processing Systems , Vol. 38 , pp. 105816–105879 . External Links: Document , Link Cited by: §A.2 , §G.1 , §1 , §4.1 . Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-Bench and Chatbot Arena . In Advances in Neural Information Processing Systems , Vol. 36 , pp. 46595–46623 . External Links: Document , Link Cited by: §A.3 , §1 . Appendix A Related Work A.1 Tool-Integrated Reasoning Tool-integrated reasoning enables language models to combine natural-language reasoning with external computation. Early systems such as ToRA use tool-augmented trajectories to improve mathematical reasoning, while ReTool and TIRL apply reinforcement learning to multi-turn tool use ( Gou et al., 2024 ; Feng et al., 2026 ; Chen et al., 2025 ) . Recent methods extend this direction through different reward and training designs. ToRL studies scalable reinforcement learning for computational tool use, and AutoTIR combines outcome and process signals to encourage autonomous tool invocation ( Li et al., 2025b ; Wei et al., 2025 ) . Tool-Star supports multi-tool reasoning through hierarchical rewards and self-critic feedback ( Dong et al., 2025 ) , while Nemotron-Research-Tool-N1 uses format and functional-correctness rewards to train tool-using reasoning models ( Zhang et al., 2026a ) . Other studies focus on the optimization challenges of multi-turn tool use. SimpleTIR removes void-turn trajectories that provide ineffective learning signals during end-to-end reinforcement learning ( Xue et al., 2026 ) . Understanding Tool-Integrated Reasoning analyzes the capabilities required for effective tool use and proposes adaptive optimization mechanisms for more stable training ( Lin and Xu, 2025 ) . These methods primarily improve the tool-using execution policy. UnifiedPlayers instead studies the coupled setting in which task generation, tool-integrated execution, and executable evaluation are all adaptive. A.2 Self-Generated Data and Adaptive Exploration Self-generated training reduces dependence on manually constructed demonstrations. Self-Instruct, STaR, and AutoAct generate instructions, reasoning traces, or agent interactions for subsequent training ( Wang et al., 2023b ; Zelikman et al., 2022 ; Qiao et al., 2024 ) . More recent self-evolving methods jointly adapt parts of the data-generation process. Absolute Zero, R-Zero, and Socratic-Zero generate reasoning tasks or feedback without external training data ( Zhao et al., 2025 ; Huang et al., 2026 ; Wang et al., 2025 ) . Agent0 jointly trains a Curriculum policy and an Executor, while TaskCraft generates structurally complex agent tasks ( Xia et al., 2025 ; Shi et al., 2026 ) . AReaL-SEA synthesizes interactions together with executable verification functions and uses their outcomes as reinforcement-learning rewards ( Gao et al., 2026 ) . Related work also improves exploration and experience selection. SPORT forms a multimodal loop of task synthesis, trajectory exploration, verifier feedback, and step-wise preference tuning ( Li et al., 2025a ) . E3-TIR combines complementary experiences to increase trajectory diversity and mitigate mode collapse in tool-integrated reinforcement learning ( Guo et al., 2026 ) . RAPO retrieves prior interaction trajectories to expand exploration under sparse rewards ( Zhang et al., 2026b ) . These approaches improve the tasks or experiences used to train an agent. UnifiedPlayers focuses on a different question: how three adaptive policies can coordinate when each changes the evidence or feedback available to the others. A.3 Executable Verification and Multi-Player Optimization Reinforcement learning with verifiable rewards replaces learned reward predictions with explicit correctness checks, including answer matching, unit tests, and environment-state validation ( Cobbe et al., 2021 ; Shao et al., 2024 ; Guo et al., 2025 ) . This approach provides reliable supervision when a suitable verifier is available, but manually defining checks for newly generated tasks can be costly. AReaL-SEA addresses this issue by generating task-specific verification functions, while Nemotron-Research-Tool-N1 incorporates rule-based functional checks into tool-use training ( Gao et al., 2026 ; Zhang et al., 2026a ) . Learned evaluators offer broader coverage but may introduce biases or incorrect judgments ( Zheng et al., 2023 ) . UnifiedPlayers treats executable evaluation as an adaptive role rather than a fixed reward mechanism. The Evaluation Player generates verifiers that are executed on original and perturbed trajectories. Their verdicts form shared evidence from which separate Planning, Execution, and Evaluation rewards are derived. The players retain separate parameters and objectives and are updated in alternating blocks. Thus, cooperation refers to the use of common executable evidence, not to a shared scalar reward or shared policy parameters. This design complements execution-centered tool-integrated reinforcement learning and two-player self-evolving systems by explicitly modeling the interaction among adaptive task generation, execution, and evaluation. Appendix B Methodology Details B.1 Cross-Player Interaction Records and Executable Feedback Each task produces a cross-player interaction record that connects the three policies through executable evidence. The Planning Player supplies task q q , the Execution Player samples K K original trajectories, and the Evaluation Player generates M M candidate Python verifiers. A fixed perturbation engine then produces P P variants of each trajectory. After parsing and compilation, the sandbox executes every valid verifier on the original and perturbed trajectories. We call the resulting pass/fail outcomes executable feedback because they are obtained by running verifier code rather than from unexecuted critiques or another player’s reward. Let ℐ valid ​ ( q ) \mathcal{I}{\mathrm{valid}}(q) denote the valid verifier indices and M q = | ℐ valid ​ ( q ) | M{q}=|\mathcal{I}{\mathrm{valid}}(q)| . For a completed evaluation, v i ​ ( τ ) v{i}(\tau) is the binary acceptance verdict. We record Z i , k orig = v i ​ ( τ k ) , Z i , k , j adv = v i ​ ( τ k , j adv ) . Z^{\mathrm{orig}}{i,k}=v{i}(\tau_{k}),\qquad Z^{\mathrm{adv}}{i,k,j}=v{i}(\tau^{\mathrm{adv}}{k,j}). (9) Flattening the adversarial trajectory indices ( k , j ) (k,j) gives 𝐙 ⁡ ( q ) = [ 𝐙 orig ∣ 𝐙 adv ] ∈ { 0 , 1 } M q × K ⁡ ( 1 + P ) . \mathbf{Z}(q)=[\mathbf{Z}^{\mathrm{orig}}\mid\mathbf{Z}^{\mathrm{adv}}]\in{0,1}^{M{q}\times K(1+P)}. (10) Each row records one verifier’s judgments, while each column records the verifier pool’s judgments of one trajectory. The columns therefore provide feedback about Execution outputs, the rows provide feedback about Evaluation outputs, and summaries across both inform Planning. For M q > 0 M_{q}>0 , the Execution reward averages the original-trajectory column corresponding to its output. The Evaluation reward uses the row summaries ρ i gold = 1 K ​ ∑ k = 1 K Z i , k orig , ρ i adv = 1 K ​ P ​ ∑ k = 1 K ∑ j = 1 P ( 1 − Z i , k , j adv ) . \rho_{i}^{\mathrm{gold}}=\frac{1}{K}\sum_{k=1}^{K}Z^{\mathrm{orig}}{i,k},\qquad\rho{i}^{\mathrm{adv}}=\frac{1}{KP}\sum_{k=1}^{K}\sum_{j=1}^{P}(1-Z^{\mathrm{adv}}{i,k,j}). (11) Here, ρ i gold \rho{i}^{\mathrm{gold}} measures acceptance of original trajectories and ρ i adv \rho_{i}^{\mathrm{adv}} measures rejection of their perturbations. The superscript gold \mathrm{gold} does not imply independently verified correctness: an original trajectory may contain an error, and a perturbation may preserve its correctness. The Planning reward combines execution uncertainty, verifier writability, and question repetition. Its verifier-writability term can equivalently be written as V ⁡ ( q ) = 1 M ​ ∑ i ∈ ℐ valid ​ ( q ) ρ i adv . V(q)=\frac{1}{M}\sum_{i\in\mathcal{I}{\mathrm{valid}}(q)}\rho{i}^{\mathrm{adv}}. (12) This form incorporates the valid-verifier fraction and gives V ⁡ ( q ) = 0 V(q)=0 when no valid verifier is produced. Evaluation additionally uses source-code redundancy, which is computed separately from the verdict matrix. The cross-player record ξ ⁡ ( q ) \xi(q) in Eq. 1 collects q q , the original trajectories 𝒯 ⁡ ( q ) \mathcal{T}(q) , valid verifiers 𝒱 ⁡ ( q ) \mathcal{V}(q) , perturbed trajectories 𝒯 adv ​ ( q ) \mathcal{T}^{\mathrm{adv}}(q) , verdict matrix 𝐙 ⁡ ( q ) \mathbf{Z}(q) , and redundancy scores 𝝆 red ​ ( q ) \bm{\rho}^{\mathrm{red}}(q) . Planning, Execution, and Evaluation derive different rewards from this shared record. It therefore coordinates the players through common executable evidence without introducing a shared scalar reward, a new sequential rollout, or gradient propagation across policies. B.2 The Objectives of UnifiedPlayers Let 𝒳 \mathcal{X} denote the interaction process induced by the three policies, the perturbation engine, and the sandbox. The conditional expected utilities are J T ​ ( π T , π S , π C ) \displaystyle J_{T}(\pi_{T};\pi_{S},\pi_{C}) = 𝔼 𝒳 ​ [ r T ​ ( q ) ] , \displaystyle=\mathbb{E}{\mathcal{X}}[r{T}(q)], (13) J S ​ ( π S , π T , π C ) \displaystyle J_{S}(\pi_{S};\pi_{T},\pi_{C}) = 𝔼 𝒳 ​ [ r S ​ ( τ ) ] , \displaystyle=\mathbb{E}{\mathcal{X}}[r{S}(\tau)], (14) J C ​ ( π C , π T , π S ) \displaystyle J_{C}(\pi_{C};\pi_{T},\pi_{S}) = 𝔼 𝒳 ​ [ r C ​ ( v ) ] . \displaystyle=\mathbb{E}{\mathcal{X}}[r{C}(v)]. (15) Each expectation includes the sampled tasks, trajectories, verifier pools, and perturbations needed to evaluate that role’s reward. The Planning objective also depends on the question history used by its repetition penalty. These objectives are coupled through shared evidence. Planning changes the task distribution, Execution changes the trajectories being checked, and Evaluation changes the executable judgments. Cooperation therefore does not imply identical rewards or improvement in every utility after each update. We assume neither a common scalar potential nor a shared policy parameterization. The mean of block surrogates in Eq. 7 is a reporting convention, not a shared training loss. Its factor of 1 / 3 1/3 does not prescribe gradient scaling in the optimizer. Each block optimizes the active player’s own surrogate. B.3 Complete Policy Optimization Definitions For active player p ∈ { T , S , C } p\in{T,S,C} in block b b , a group contains candidate outputs generated under the same input context 𝒢 p ​ ( x ) = { ( a p , g , r p , g ) } g = 1 G p , ( x , 𝒢 p ​ ( x ) ) ∈ 𝒟 p ( b ) . \mathcal{G}{p}(x)={(a{p,g},r_{p,g})}{g=1}^{G{p}},\qquad(x,\mathcal{G}{p}(x))\in\mathcal{D}{p}^{(b)}. (16) For Planning, x x is the fixed questioner prompt and a T , g a_{T,g} is a generated task with its expected answer. For Execution, x x is task q q and a S , g a_{S,g} is a sampled trajectory. For Evaluation, x x contains q q and the trajectory schema, and a C , g a_{C,g} is a verifier candidate. Other players’ outputs provide reward evidence but are not members of the active player’s group. Let r ¯ p \bar{r}{p} and s p s{p} denote the mean and standard deviation of the rewards in this group. The candidate advantage is A ^ p , g = r p , g − r ¯ p s p + ϵ , ϵ > 0 . \widehat{A}{p,g}=\frac{r{p,g}-\bar{r}{p}}{s{p}+\epsilon},\qquad\epsilon>0. (17) The constant ϵ \epsilon stabilizes the denominator. Under this expression, equal rewards yield zero advantages. This removes the reward-dependent term for that group, but the KL penalty can still contribute to optimization. The current policy π θ p \pi_{\theta_{p}} is optimized against outputs sampled by the behavior policy π p old \pi_{p}^{\mathrm{old}} . The separate reference policy π p ref \pi_{p}^{\mathrm{ref}} is saved at the start of the outer iteration and defines the KL penalty. Behavior-policy probabilities remain fixed when optimizing the records they generated. They are not interchangeable with reference-policy probabilities. Let a p , g , t a_{p,g,t} be a token at a position included in the candidate loss, and let h p , g , t h_{p,g,t} be its observed history. For Execution, this history includes returned tool observations. The likelihood ratio is ϱ p , g , t ​ ( θ p ) = π θ p ​ ( a p , g , t ∣ h p , g , t ) π p old ​ ( a p , g , t ∣ h p , g , t ) . \varrho_{p,g,t}(\theta_{p})=\frac{\pi_{\theta_{p}}(a_{p,g,t}\mid h_{p,g,t})}{\pi_{p}^{\mathrm{old}}(a_{p,g,t}\mid h_{p,g,t})}. (18) Its clipped value is ϱ ¯ p , g , t = clip ⁡ ( ϱ p , g , t , 1 − ϵ clip , 1 + ϵ clip ) . \bar{\varrho}{p,g,t}=\operatorname{clip}(\varrho{p,g,t},1-\epsilon_{\mathrm{clip}},1+\epsilon_{\mathrm{clip}}). (19) Writing D p , g , t KL D^{\mathrm{KL}}{p,g,t} for the KL penalty relative to π p ref \pi{p}^{\mathrm{ref}} at the same history, the token surrogate is ℓ p , g , t = min ⁡ ( ϱ p , g , t ​ A ^ p , g , ϱ ¯ p , g , t ​ A ^ p , g ) − β p ​ D p , g , t KL . \ell_{p,g,t}=\min!\left(\varrho_{p,g,t}\widehat{A}{p,g},\bar{\varrho}{p,g,t}\widehat{A}{p,g}\right)-\beta{p}D^{\mathrm{KL}}{p,g,t}. (20) The candidate advantage is shared across its token positions. If L p , g L{p,g} is the number of positions included in its loss, the full block objective is 𝒥 p ( b ) ​ ( θ p ) = 𝔼 ( x , 𝒢 p ​ ( x ) ) ∼ 𝒟 p ( b ) ​ [ 1 G p ​ ∑ g = 1 G p 1 L p , g ​ ∑ t = 1 L p , g ℓ p , g , t ] . \mathcal{J}{p}^{(b)}(\theta{p})=\mathbb{E}{(x,\mathcal{G}{p}(x))\sim\mathcal{D}{p}^{(b)}}\left[\frac{1}{G{p}}\sum_{g=1}^{G_{p}}\frac{1}{L_{p,g}}\sum_{t=1}^{L_{p,g}}\ell_{p,g,t}\right]. (21) Thus, tokens are averaged within each candidate, candidates are averaged within each group, and groups are averaged over the block data. Rewards and advantages are not normalized across different roles. B.4 Update Order within UnifiedPlayers An outer iteration contains three role blocks in the order Planning, Execution, and Evaluation. A block contains multiple optimizer steps, not a single update. Table 6 specifies 200 Planning updates, 300 Execution updates, and 200 Evaluation updates per outer iteration. Training uses three outer iterations. Planning block. The Execution and Evaluation policies remain fixed. Candidate tasks are assessed using their trajectories, verifier judgments, and the question repetition penalty. Only the Planning policy receives parameter updates. Execution block. The updated Planning policy supplies fresh tasks. Planning and Evaluation remain fixed while Execution generates trajectories and is optimized using acceptance judgments from the verifier pool. Evaluation block. Planning and the updated Execution policy remain fixed. Evaluation generates verifier candidates, which are scored on original and perturbed trajectories. Only Evaluation receives parameter updates. Its resulting checkpoint is available to the other players in the next outer iteration. In

[... middle truncated for ingest size ...]

n over the Tool-augmented reference by these costs gives approximately 0.029 points per GPU-hour for UnifiedPlayers and 0.027 for Agent0. These ratios describe the reported runs and should not be interpreted as a comparison of total system cost. H.3 Perturbation Rule Subsets This analysis extends the “w/o Perturbation Engine” comparison in Table 5 by separating the rules used in the full configuration into two subsets. We divide the five perturbation rules into local and global subsets to examine whether their effects are redundant. Local edits modify individual reasoning or execution steps and include arg_mutation , step_drop , and step_swap . Global edits change the trajectory structure and include early_terminate and tool_substitute . Table 15: Effect of perturbation-rule subsets on Qwen3-4B-Base. Math AVG measures downstream task performance, and Adv. Reject measures the adversarial rejection rate of the Evaluation Player. Perturbation Rules Math AVG Adv. Reject Full capacity (5 rules) 56.4 0.79 Local edits only (3 rules) 55.2 0.68 Global edits only (2 rules) 54.9 0.71 No perturbation 51.6 0.21 Table 15 shows that the local subset retains a higher math average than the global subset, whereas the global subset retains a higher adversarial rejection rate. The full five-rule configuration performs best on both measures. Removing all perturbations lowers Math AVG from 56.4 to 51.6 and adversarial rejection from 0.79 to 0.21. These results suggest that local and global rules provide complementary negative examples, although the subset comparison does not isolate the contribution of each individual rule. H.4 Verifier Pool Size (M) Sweep This sweep extends the “Single Verifier” comparison in Table 5 by varying the verifier-pool size over M ∈ { 1 , 2 , 4 , 8 } M\in{1,2,4,8} . UnifiedPlayers uses M = 4 M=4 Evaluation Player candidates per task in the main experiments. Table 16 reports downstream performance and local compute cost on Qwen3-4B-Base. Table 16: Effect of Evaluation Player pool size M M on Qwen3-4B-Base. The bold M = 4 M=4 row is the setting used by UnifiedPlayers in Tables 1 and 2 . M M Math AVG General AVG Local GPU-hours 1 54.3 44.2 356 2 55.4 45.3 388 4 (used) 56.4 46.1 432 8 57.0 46.4 519 Within the tested range, larger verifier pools improve both benchmark averages while increasing reported local GPU-hours. Increasing M M from 4 to 8 raises Math AVG from 56.4 to 57.0 and General AVG from 46.1 to 46.4, while GPU-hours increase from 432 to 519. The main setting M = 4 M=4 therefore provides a lower-cost alternative with smaller performance differences than those observed between M = 1 M=1 and M = 4 M=4 . This comparison characterizes the tested settings and does not establish an optimal pool size. H.5 Reward Coefficient Sensitivity This analysis extends the “w/o Diversity Penalty” comparison in Table 5 and tests a broader range of Evaluation Player reward weights. The reward in Section 3.2 combines gold-pass, adversarial-rejection, and redundancy terms with default coefficients ( α , β , γ ) = ( 0.4 , 0.5 , 0.1 ) (\alpha,\beta,\gamma)=(0.4,0.5,0.1) . We vary these weights on Qwen3-4B-Base and report the resulting performance in Table 17 . Table 17: Sensitivity to the Evaluation Player reward coefficients ( α , β , γ ) (\alpha,\beta,\gamma) on Qwen3-4B-Base. The bold default row is the setting used by UnifiedPlayers in Tables 1 and 2 . Configuration 𝜶 \alpha 𝜷 \beta 𝜸 \gamma Math AVG General AVG Default (used) 0.4 0.5 0.1 56.4 46.1 Heavier gold-pass 0.6 0.3 0.1 55.7 45.4 Heavier adversarial 0.2 0.7 0.1 56.1 45.9 Strong diversity 0.4 0.5 0.3 55.3 45.5 No diversity 0.4 0.5 0.0 54.2 44.6 All four variants remain within 2.2 points of the default on math and 1.5 points on general reasoning. Removing the diversity term produces the largest decrease in this sweep. Increasing its weight to γ = 0.3 \gamma=0.3 also reduces performance relative to the default. Among the two rebalanced settings, the heavier adversarial weight performs slightly better than the heavier gold-pass weight on both task families. These results show moderate sensitivity within the tested range. They do not establish robustness to reward settings outside this sweep or isolate the effect of one coefficient when the others change. Appendix I Case Analysis This section provides qualitative examples of UnifiedPlayers’ three policies and the perturbation engine in action. Appendix I.1 shows representative questions produced by the Planning Player across the three outer iterations. Appendix I.2 walks through complete Execution Player trajectories on real MATH-500 problems. Appendix I.3 presents illustrative task-specific checks and discusses their limitations. Appendix I.4 shows pairs of original and perturbed Execution Player trajectories together with the verifier’s verdict. I.1 Generated Questions across Iterations To illustrate how the Planning Player’s question distribution evolves, we sample twelve problems from each outer iteration on Qwen3-4B-Base. Iter 1 problems exercise basic algebraic identities and elementary number theory, Iter 2 problems require multi-step reasoning over geometric or combinatorial constructs, and Iter 3 problems approach competition-level difficulty. Iter 1 (algebra and elementary number theory) Q1. Solve 2 ​ x 2 − 5 ​ x − 3 = 0 2x^{2}-5x-3=0 . Express the sum of the roots as a fraction. Answer: 5 / 2 \boxed{5/2} Q2. Find the smallest positive integer n n such that 3 ​ n + 7 3n+7 is divisible by 11 11 . Answer: 5 \boxed{5} Q3. If a / b = 3 / 4 a/b=3/4 and b / c = 5 / 6 b/c=5/6 , find a / c a/c . Answer: 5 / 8 \boxed{5/8} Q4. Compute 7 3 + 3 2 7^{3}+3^{2} . Answer: 352 \boxed{352} Q5. Find the greatest common factor of 24 24 and 36 36 . Answer: 12 \boxed{12} Q6. Solve 3 ​ x + 7 = 22 3x+7=22 for x x . Answer: 5 \boxed{5} Q7. What is the least common multiple of 6 6 and 8 8 ? Answer: 24 \boxed{24} Q8. Find the sum of all positive divisors of 12 12 . Answer: 28 \boxed{28} Q9. If sin ⁡ θ = 3 / 5 \sin\theta=3/5 with θ \theta in the first quadrant, find cos ⁡ θ \cos\theta . Answer: 4 / 5 \boxed{4/5} Q10. What is the perimeter of a regular hexagon with side length 5 5 ? Answer: 30 \boxed{30} Q11. Find the smallest prime greater than 50 50 . Answer: 53 \boxed{53} Q12. If 2 x = 64 2^{x}=64 , what is x x ? Answer: 6 \boxed{6} Iter 2 (multi-step reasoning) Q1. How many integers between 1 1 and 1000 1000 are divisible by 3 3 or 5 5 but not by 15 15 ? Answer: 401 \boxed{401} Q2. Let f ⁡ ( x ) = x 3 − 6 ​ x 2 + 11 ​ x − 6 f(x)=x^{3}-6x^{2}+11x-6 . Find all integer roots. Answer: 1 , 2 , 3 \boxed{1,2,3} Q3. A circle of radius r r is inscribed in a triangle with sides 13 , 14 , 15 13,14,15 . Find r r . Answer: 4 \boxed{4} Q4. Find the area of a circle inscribed in a square of side 10 10 . Answer: 25 ​ π \boxed{25\pi} Q5. Solve the system x + y = 10 x+y=10 , x − y = 4 x-y=4 . Give ( x , y ) (x,y) . Answer: ( 7 , 3 ) \boxed{(7,3)} Q6. How many distinct arrangements of the letters in MISSISSIPPI are there? Answer: 34650 \boxed{34650} Q7. Find ( 3 + 4 ​ i ) ​ ( 2 − i ) (3+4i)(2-i) where i 2 = − 1 i^{2}=-1 . Answer: 10 + 5 ​ i \boxed{10+5i} Q8. Find the sum of all perfect squares less than or equal to 100 100 . Answer: 385 \boxed{385} Q9. In how many ways can three letters be chosen from { A , B , C , D , E } {A,B,C,D,E} ? Answer: 10 \boxed{10} Q10. What is the determinant of ( 2 3 4 5 ) \begin{pmatrix}2&3\ 4&5\end{pmatrix} ? Answer: − 2 \boxed{-2} Q11. Find the sum of the infinite geometric series with first term 4 4 and ratio 1 / 3 1/3 . Answer: 6 \boxed{6} Q12. How many positive integers less than 100 100 have exactly three positive divisors? Answer: 4 \boxed{4} Iter 3 (competition-level) Q1. Find the smallest positive integer n n such that 1 2 + 2 2 + ⋯ + n 2 1^{2}+2^{2}+\cdots+n^{2} is divisible by 100 100 . Answer: 24 \boxed{24} Q2. Find the sum of all positive integers n ≤ 1000 n\leq 1000 such that n 2 + 1 n^{2}+1 is divisible by 5 5 . Answer: 200000 \boxed{200000} Q3. Let p p be a prime such that p 2 + 11 p^{2}+11 has exactly six positive divisors. Find p p . Answer: 3 \boxed{3} Q4. Find the units digit of 7 100 7^{100} . Answer: 1 \boxed{1} Q5. Find the remainder when 2 100 2^{100} is divided by 7 7 . Answer: 2 \boxed{2} Q6. What is the smallest positive integer k k such that k ! k! is divisible by 1000 1000 ? Answer: 15 \boxed{15} Q7. Find the number of ordered triples ( a , b , c ) (a,b,c) of positive integers with a + b + c = 12 a+b+c=12 . Answer: 55 \boxed{55} Q8. Find the sum of all distinct prime factors of 2 12 − 1 2^{12}-1 . Answer: 28 \boxed{28} Q9. Find the number of triples ( a , b , c ) (a,b,c) of positive integers with a + b + c = 20 a+b+c=20 and a ≤ b ≤ c a\leq b\leq c . Answer: 33 \boxed{33} Q10. Compute ∑ k = 1 10 1 k ⁡ ( k + 1 ) \sum_{k=1}^{10}\frac{1}{k(k+1)} . Answer: 10 / 11 \boxed{10/11} Q11. Find the smallest n n such that n ! n! has at least 10 10 trailing zeros. Answer: 45 \boxed{45} Q12. How many integer solutions ( x , y ) (x,y) satisfy x 2 + y 2 = 100 x^{2}+y^{2}=100 ? Answer: 12 \boxed{12} I.2 Execution Player Trajectory Examples We show eight Execution Player trajectories on real MATH-500 problems drawn from Number Theory, Algebra, and Counting & Probability. Each trajectory interleaves natural-language reasoning with Python tool calls and ends with a \boxed{...} answer. Trajectory 1: divisor counting (test/number_theory/572.json) Question. How many positive whole-number divisors does 196 196 have? Reasoning. 196 = 4 × 49 = 2 2 × 7 2 196=4\times 49=2^{2}\times 7^{2} . Any divisor has the form 2 a ⋅ 7 b 2^{a}\cdot 7^{b} with 0 ≤ a ≤ 2 0\leq a\leq 2 and 0 ≤ b ≤ 2 0\leq b\leq 2 , so the divisor count is ( 2 + 1 ) ​ ( 2 + 1 ) = 9 (2+1)(2+1)=9 . Code. count = sum(1 for d in range(1, 197) if 196 % d == 0) print(count) Output. 9 . Final. 9 \boxed{9} . Trajectory 2: GCF plus eleven (test/number_theory/45.json) Question. What is the result when the greatest common factor of 6432 6432 and 132 132 is increased by 11 11 ? Reasoning. Compute gcd ⁡ ( 6432,132 ) \gcd(6432,132) by the Euclidean algorithm and add 11 11 . Code. from math import gcd print(gcd(6432, 132) + 11) Output. 23 . Final. 23 \boxed{23} . Trajectory 3: smallest perfect cube (test/number_theory/515.json) Question. What is the smallest positive perfect cube that can be written as the sum of three consecutive integers? Reasoning. Three consecutive integers sum to 3 ​ ( a + 1 ) 3(a+1) . For this to be a perfect cube k 3 k^{3} , k k must be a multiple of 3 3 . The smallest is k = 3 k=3 , giving 8 + 9 + 10 = 27 8+9+10=27 . Code. for a in range(1, 100): s = a + (a + 1) + (a + 2) r = round(s ** (1/3)) if r ** 3 == s: print(s); break Output. 27 . Final. 27 \boxed{27} . Trajectory 4: sum of first N N odd numbers (test/algebra/769.json) Question. The sum of the first N N positive odd numbers is 121 121 . What is the value of N N ? Reasoning. The sum of the first N N positive odd numbers is N 2 N^{2} . So N 2 = 121 N^{2}=121 gives N = 11 N=11 . Code. N = int(121 ** 0.5) print(N) Output. 11 . Final. 11 \boxed{11} . Trajectory 5: Greek army battalions (test/counting_and_probability/666.json) Question. The Greek army has 5 5 upper-class soldiers and 10 10 lower-class soldiers. A battalion requires 4 4 upper-class soldiers and 8 8 lower-class soldiers. How many different battalions can be sent? Reasoning. Choose 4 4 from 5 5 upper and 8 8 from 10 10 lower independently. The count is ( 5 4 ) ​ ( 10 8 ) = 5 × 45 = 225 \binom{5}{4}\binom{10}{8}=5\times 45=225 . Code. from math import comb print(comb(5, 4) * comb(10, 8)) Output. 225 . Final. 225 \boxed{225} . Trajectory 6: modular product (test/number_theory/627.json) Question. If n ≡ 2 ( mod 7 ) n\equiv 2\pmod{7} , find the remainder when ( n + 2 ) ​ ( n + 4 ) ​ ( n + 6 ) (n+2)(n+4)(n+6) is divided by 7 7 . Reasoning. Substitute n = 2 n=2 as the representative residue. Then ( n + 2 ) ​ ( n + 4 ) ​ ( n + 6 ) = 4 ⋅ 6 ⋅ 8 = 192 (n+2)(n+4)(n+6)=4\cdot 6\cdot 8=192 , and 192 = 7 ⋅ 27 + 3 192=7\cdot 27+3 , so the remainder is 3 3 . Code. print((4 * 6 * 8) % 7) Output. 3 . Final. 3 \boxed{3} . Trajectory 7: GCD of three integers (test/number_theory/357.json) Question. Find the greatest common divisor of 3339 3339 , 2961 2961 , and 1491 1491 . Reasoning. Apply the Euclidean algorithm pairwise. The gcd of three numbers is gcd ⁡ ( gcd ⁡ ( 3339 , 2961 ) , 1491 ) \gcd(\gcd(3339,2961),1491) . Code. from math import gcd print(gcd(gcd(3339, 2961), 1491)) Output. 21 . Final. 21 \boxed{21} . Trajectory 8: seven bags of gold coins (test/number_theory/1055.json) Question. You have seven bags of gold coins with the same count per bag. After finding a bag of 53 53 coins, you redistribute all coins evenly into eight bags. You also have more than 200 200 coins. What is the smallest number of coins you could have had before finding the bag? Reasoning. Let b b be the per-bag count, so total before is 7 ​ b 7b and total after is 7 ​ b + 53 7b+53 , which must be divisible by 8 8 . We need 7 ​ b + 53 ≡ 0 ( mod 8 ) 7b+53\equiv 0\pmod{8} and 7 ​ b > 200 7b>200 . Code. for b in range(1, 1000): if (7 * b + 53) % 8 == 0 and 7 * b > 200: print(7 * b); break Output. 203 . Final. 203 \boxed{203} . I.3 Illustrative Verifier Checks For Trajectories 1, 2, 4, and 5 in Appendix I.2 , we present three illustrative checks per task. These listings show individual acceptance conditions, not complete verifier implementations or verbatim training outputs. Each function is named check_condition and returns a Boolean result with a diagnostic message. This display format is distinct from the formal check(trajectory) interface, which returns (passed, fail_step) as specified in Appendix E.3 . The listings do not implement failure-step localization. Input parsing. The trajectory schema stores final_answer as a string. The numerical checks illustrated in Figure 4 therefore require integer conversion before divisibility or minimality tests: ans = int(trajectory.final_answer) Verifier 1.1: brute-force re-count def check_condition(trajectory): expected = sum(1 for d in range(1, 197) if 196 % d == 0) if int(trajectory.final_answer) != expected: return False, f"wrong count: expected {expected}" return True, "passed" Verifier 1.2: code-output cross-check def check_condition(trajectory): outs = [s.result for s in trajectory.steps if s.type == "output"] if not outs or str(outs[-1]).strip() != str(trajectory.final_answer).strip(): return False, "code output and final answer mismatch" return True, "passed" Verifier 1.3: prime factorization def check_condition(trajectory): n, count, p = 196, 1, 2 while p * p <= n: e = 0 while n % p == 0: n //= p; e += 1 count *= (e + 1); p += 1 if n > 1: count = 2 if int(trajectory.final_answer) != count: return False, f"expected {count}" return True, "passed" Verifiers for Trajectory 2 (GCF plus eleven). Verifier 2.1: direct gcd + 11 \gcd+11 def check_condition(trajectory): from math import gcd expected = gcd(6432, 132) + 11 if int(trajectory.final_answer) != expected: return False, f"wrong value: expected {expected}" return True, "passed" Verifier 2.2: Euclidean algorithm step-by-step def check_condition(trajectory): a, b = 6432, 132 while b: a, b = b, a % b if int(trajectory.final_answer) != a + 11: return False, f"gcd is {a}, expected {a + 11}" return True, "passed" Verifier 2.3: subtract-and-verify def check_condition(trajectory): from math import gcd val = int(trajectory.final_answer) if val - 11 != gcd(6432, 132): return False, "answer minus 11 should equal gcd" return True, "passed" Verifiers for Trajectory 4 (sum of first N N odd numbers). Verifier 4.1: closed-form N2̂ def check_condition(trajectory): N = int(trajectory.final_answer) if N * N != 121: return False, f"N^2 = {NN}, expected 121" return True, "passed" Verifier 4.2: brute-force sum def check_condition(trajectory): N = int(trajectory.final_answer) s = sum(2 * k - 1 for k in range(1, N + 1)) if s != 121: return False, f"sum of first {N} odd numbers is {s}" return True, "passed" Verifier 4.3: integer square root def check_condition(trajectory): import math N = int(trajectory.final_answer) if N != math.isqrt(121): return False, f"expected sqrt(121) = {math.isqrt(121)}" return True, "passed" Verifiers for Trajectory 5 (Greek army battalions). Verifier 5.1: math.comb def check_condition(trajectory): from math import comb expected = comb(5, 4) * comb(10, 8) if int(trajectory.final_answer) != expected: return False, f"wrong count: expected {expected}" return True, "passed" Verifier 5.2: enumeration via itertools def check_condition(trajectory): from itertools import combinations upper = len(list(combinations(range(5), 4))) lower = len(list(combinations(range(10), 8))) if int(trajectory.final_answer) != upper * lower: return False, f"wrong count: expected {upper * lower}" return True, "passed" Verifier 5.3: factorial formula def check_condition(trajectory): from math import factorial as f expected = (f(5) // (f(4) * f(1))) * (f(10) // (f(8) * f(2))) if int(trajectory.final_answer) != expected: return False, f"wrong count: expected {expected}" return True, "passed" These checks have different limitations. Verifier 1.2 checks agreement between the recorded output and final answer, but accepts an incorrect value when both are 8 rather than 9. Verifier 4.1 checks only N 2 = 121 N^{2}=121 and therefore accepts N = − 11 N=-11 , although the task requires a positive integer. Most other listings check the final answer without validating every reasoning step. Agreement and answer checks alone therefore do not establish trajectory correctness. The examples illustrate possible failure modes of individual checks, not a guarantee that a verifier pool eliminates them. I.4 Perturbation Pairs We show four illustrative early_terminate examples, one for each task considered in Appendix I.3 . Each example replaces the trajectory suffix with an incorrect final answer. We report the Boolean result and diagnostic message from the corresponding task check. These examples illustrate rejection of incorrect answers, not failure-step localization or coverage of all perturbation rules. The five rules and their before-after examples are described in Appendix F.1 and Appendix F.2 . Pairs on Trajectory 1 (divisor counting). Pair 1: early_terminate on Trajectory 1 Original final two steps. Reasoning “The answer is 9 9 ” followed by 9 \boxed{9} . Perturbed final two steps. Reasoning step discarded after the code output of 9 , replaced by injected 8 \boxed{8} drawn from the off-by-one distractor pool. Verifier 1.1 verdict. ✗ rejected — “wrong count: expected 9”. Pairs on Trajectory 2 (GCF plus eleven). Pair 2: early_terminate on Trajectory 2 Original. Trajectory ends in 23 \boxed{23} after the code output of 23 . Perturbed. Reasoning step discarded after the code output of 23 , replaced by injected 20 \boxed{20} . Verifier 2.1 verdict. ✗ rejected — “wrong value: expected 23”. Pairs on Trajectory 4 (sum of first N N odd numbers). Pair 3: early_terminate on Trajectory 4 Original. Trajectory ends in 11 \boxed{11} . Perturbed. Reasoning step discarded after the code output of 11 , replaced by injected 10 \boxed{10} . Verifier 4.1 verdict. ✗ rejected — “ N 2 = 100 N^{2}=100 , expected 121”. Pairs on Trajectory 5 (Greek army battalions). Pair 4: early_terminate on Trajectory 5 Original. Trajectory ends in 225 \boxed{225} . Perturbed. Reasoning step discarded after the code output of 225 , replaced by injected 200 \boxed{200} . Verifier 5.1 verdict. ✗ rejected — “wrong count: expected 225”. Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" ( ) button, located in the page header. Tip: Yo