second brain
source
← 首页

external-source

Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.18366 · published 2026-09-16 · submitted 16 Sep 2026

Abstract

Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark $B_{\mathrm{rel}}$ to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed target agent. Task holdout varies semantic tasks but leaves the benchmark protocol fixed, so a "bad genius" Proposer can produce a cheating harness whose released-benchmark gain depends on a benchmark-wide shortcut. We introduce Counterfactual Harness Search and Evolution (CHASE), which casts harness evolution as constraint generation over validity-preserving benchmark counterfactuals. After each Proposer update, a Challenger searches for an executable protocol transformation with large gain destruction. A validity firewall checks that task semantics are preserved, while a confirmation set determines whether the counterfactual enters a finite archive. We formalize an exact shortcut-neutralized benchmark $B_0$ and establish statistical guarantees linking finite counterfactual archives to $B_0$ and characterizing sequential Challenger search. We evaluate CHASE on a synthetic benchmark and on OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol changes.

Authors

Guojun Zhu, Xunheng Huang, Peng Yin, Jiahui Xie, Sanguo Zhang, Doudou Zhou

Key claims (verbatim-leaning English extract)

Structure (section headings from HTML)

(section headings from HTML)

Body excerpts (original English)

Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark B rel B_{\mathrm{rel}} to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed target agent. Task holdout varies semantic tasks but leaves the benchmark protocol fixed, so a “bad genius” Proposer can produce a cheating harness whose released-benchmark gain depends on a benchmark-wide shortcut. We introduce C ounterfactual Ha rness S earch and E volution ( CHASE ), which casts harness evolution as constraint generation over validity-preserving benchmark counterfactuals. After each Proposer update, a Challenger searches for an executable protocol transformation with large gain destruction. A validity firewall checks that task semantics are preserved, while a confirmation set determines whether the counterfactual enters a finite archive. We formalize an exact shortcut-neutralized benchmark B 0 B_{0} and establish statistical guarantees linking finite counterfactual archives to B 0 B_{0} and characterizing sequential Challenger search. We evaluate CHASE on a synthetic benchmark and on OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol changes. Reliable agent evaluation is central to progress in agentic AI, with benchmarks playing a key role in determining which systems appear capable, safe, and ready for deployment. A reported score, however, does not reflect the target agent alone, but also depends on the harness through which it operates and the benchmark protocol under which it is evaluated. A harness specifies the executable context around the agent, including what information is stored and retrieved, how tools and workspace are exposed, and how outputs are handled. Evaluation is increasingly concerned not only with performance under a fixed harness, but also with the performance attainable after optimizing the harness around a fixed target agent. Meta-Harness demonstrates that an agentic Proposer can search directly over this code using harness source code, scores, and execution traces ( 15 ) . Subsequent work studies held-out evaluation, optimizer quality, priority ranking, and reliable harness selection ( 29 ; 18 ; 27 ; 37 ) . Harness-Bench further shows that performance and failure modes vary materially across agent–harness configurations under shared tasks ( 35 ) . Together, these results make harness optimization–and, more broadly, harness evolution–a substantive evaluation target ( 38 ; 39 ) . Repeated use of benchmark feedback creates a distinct generalization problem. The ‘‘Bad genius’’ 1 1 1 The title refers to the film Bad Genius , where a gifted student helps others cheat through clever schemes. Proposer is optimized for benchmark score, not for preserving what that score is intended to measure, and can modify the harness using benchmark feedback and execution traces. Classical shortcut learning concerns predictive rules induced by data or task artifacts ( 9 ; 25 ) . Harness evolution adds an executable route: the Proposer can write the shortcut into the agent’s protocol. Existing methods ban task identifiers, filenames, and per-task repair recipes, then evaluate on held-out tasks ( 29 ; 36 ) . These methods limit a task-specific shortcut —for example, detecting one question ID and opening a known document—but not a benchmark-wide shortcut , such as searching one document channel first because the released protocol places supporting evidence there more often. As illustrated in Figure 1(a) , task holdout therefore leaves benchmark-wide shortcuts intact: tasks change, but the protocol correlation does not. For example, 58.1% of questions in the benchmark OfficeQA Full mention numerical scales such as millions or billions. In its 697-document corpus, the next nonblank line after 95.2% of unit statements begins a table (Supplementary C.1 ). The Proposer may therefore propose a harness that induces the agent to inspect only the text immediately preceding a table for its unit. Task holdout alone does not evaluate whether the observed gain survives once this shared protocol correlation is broken. The natural next step is therefore to retain held-out tasks while varying the protocol as well. Figure 1(b)–(c) illustrates this comparison. Under the released protocol in Figure 1(b) , the shortcut “search channel A first” still places the supporting evidence first. Figure 1(c) then reassigns the same documents across channels, removing this shortcut advantage while keeping the question, answer, and documents fixed. The relevant quantity is not the resulting score change alone, since the benchmark intervention may affect both the evolved and initial harnesses, but how much of the evolved harness’s gain over H 0 H_{0} disappears. A large loss of gain reveals benchmark dependence that held-out tasks alone cannot expose. This raises a question: can we construct a benchmark that neutralizes all benchmark-wide shortcuts while preserving the underlying tasks? If every shortcut mechanism were known, one could construct an ideal shortcut-neutralized benchmark B 0 B_{0} and optimize the harness on it. In realistic agent benchmarks, however, benchmark-wide shortcuts can arise from many parts of the protocol and remain hidden from both manual inspection and automated checks, making them difficult to pre-specify with fixed rules. We therefore introduce a separate Challenger that searches online, after each harness update, for executable benchmark transformations, each producing a counterfactual benchmark designed to expose shortcuts in the current harness. Whereas prior work introduces auxiliary agents as debuggers or critics to guide harness repair ( 17 ; 16 ) , our Challenger attacks shortcut gains at their source—the benchmark protocol. Each valid counterfactual confirmed on held-out tasks becomes a constraint on subsequent harness evolution, limiting how much future gains can depend on the same protocol correlation. Our contributions are fourfold. First, we distinguish task-specific shortcuts from benchmark-wide harness shortcuts and define a gain-destruction estimand that assesses whether observed harness gains remain reliable under benchmark-preserving counterfactuals. Second, we propose Counterfactual Harness Search and Evolution (CHASE) , in which a Challenger searches for executable benchmark counterfactuals. Third, we establish finite-archive guarantees that certify how much harness gain survives shortcut neutralization through counterfactual-guided evolution and when Challenger search can no longer expose a large violation. Fourth, across synthetic benchmark Syn-Ledger and OfficeQA, we show that CHASE yields confirmed counterfactuals and substantially better generalization to protocol variants than other baselines. Harness optimization. Meta-Harness optimizes prompts, memory, tools, and control code using evaluation feedback ( 15 ) . HarnessOpt-Bench and priority ranking evaluate the optimizer itself under bounded target-agent calls ( 27 ; 18 ) . Harness-Bench isolates configuration-level harness effects across model backends ( 35 ) . HarnessLens reduces verification cost by selecting behavior-relevant tasks for each candidate and confirming promising edits on an additional task batch ( 33 ) . AutoSaddler further combines failure-trace diagnosis with validation-based harness updates ( 21 ) . These works characterize the optimization object, optimizer quality, and verification efficiency. CHASE addresses a different question: whether feedback from B rel B_{\mathrm{rel}} selects a harness whose gain depends on protocol-level correlations shared across tasks. Harness generalization. Held-out-task evaluation shows that optimizing and evaluating harnesses on the same tasks can substantially overstate their gains ( 29 ; 8 ) . HarnessCompass uses a fixed Generalization Gate to allow only task-agnostic modifications, together with component-wise feedback to guide harness updates ( 36 ) . HarnessEvolve uses reference trajectories, quality and performance gates, and held-out validation to filter and select harness updates ( 13 ) . EvoSafeHarness further uses fresh-context adversarial review to reject benchmark-specific safety rules during harness optimization ( 16 ) . Harness continual learning evaluates new harness updates for retention of previously acquired behavior ( 14 ) . More broadly, continually updated benchmarks, lifelong test sets, and overfitting alarms reduce repeated reliance on a fixed evaluation set ( 24 ; 12 ) . Overall, existing methods mainly vary tasks or test content, apply fixed filters to harness edits, or validate candidate updates before adoption. The former leaves the benchmark protocol unchanged, while the latter can only cover shortcut patterns anticipated by the filter. CHASE instead searches for counterfactuals against the current harness while preserving task semantics, without relying on a fixed filter over harness edits. Benchmark validity. Agent scores are properties of an agent–harness–environment–protocol stack rather than the target agent alone ( 38 ; 39 ; 35 ) . Recent benchmark audits show that agent scores can be inflated by benchmark-wide shortcuts, search-time contamination, broken tasks, and scoring errors ( 26 ; 30 ; 7 ) . Contamination tests and refreshed benchmarks address exposure of benchmark content ( 20 ; 31 ; 32 ) . Auditing Harness Tampering studies a related problem in self-improving agents, where harness edits can produce apparent performance gains without genuine capability improvement, and develops audits to detect and locate such edits ( 28 ) . HackProbe similarly detects reward hacking during self-evolution and uses the resulting signal for candidate reselection ( 34 ) . These works diagnose or reduce specific sources of invalid benchmark gains. CHASE instead tests protocol dependence directly by searching for valid protocol changes that destroy the gain of the current harness and turning confirmed counterfactuals into constraints on subsequent evolution. Counterfactual-guided optimization. Counterfactual-guided model repair repeatedly finds a counterfactual to the current model and updates the model to remove it ( 2 ; 3 ) . Model-written evaluations and automated red teaming use one model to generate tests that expose failures of another ( 22 ; 23 ) . Metamorphic testing evaluates a system under transformations that keep the task semantics unchanged ( 11 ; 5 ) . CHASE combines these ideas for harness evolution: after each Proposer update, the Challenger searches for a valid benchmark counterfactual that destroys the current harness’s gain, and a confirmed counterfactual constrains subsequent harness updates. Fix a target agent 𝖠 \mathsf{A} , and recall from Section 1 that B rel B_{\mathrm{rel}} denotes the released benchmark used for harness optimization. To describe its evaluation target, let U U denote the semantic task state and let Y = ψ ⁡ ( U ) Y=\psi(U) be the intended target, where ψ ⁡ ( ⋅ ) \psi(\cdot) is the target rule. The released protocol configuration V rel V_{\rm rel} contains file names, directory layout, metadata, tool aliases, demonstration order, feedback format, and other benchmark-specific details, while Q rel Q_{\rm rel} specifies how the agent interacts with these components. Thus, B rel B_{\mathrm{rel}} combines a distribution P P over semantic tasks U U and their induced targets Y = ψ ⁡ ( U ) Y=\psi(U) , the released protocol ( V rel , Q rel ) (V_{\rm rel},Q_{\rm rel}) , the available tools and resource budgets, and a scorer. Write B rel = ( U , V rel , Q rel , Y ) B_{\mathrm{rel}}=(U,V_{\rm rel},Q_{\rm rel},Y) . More generally, for any benchmark variant B B sharing the semantic-task distribution P P and any harness H H , let τ H , B ∼ p 𝖠 ( ⋅ ∣ H , B , U ) \tau^{H,B}\sim p_{\mathsf{A}}(\cdot\mid H,B,U) denote the resulting execution trajectory and let r B ​ ( τ , U ) ∈ [ 0 , 1 ] r_{B}(\tau,U)\in[0,1] denote the score assigned by benchmark B B . Suppose harness optimization runs for T T rounds. Let H 0 H_{0} be the initial harness. At each round t = 1 , … , T t=1,\ldots,T , a Proposer—for example, GPT-5.6 Sol configured as a coding agent—observes the current harness, previous scores, and execution traces, edits H t − 1 H_{t-1} , and returns H t H_{t} . Let D s = { U i } i = 1 n s D_{s}={U_{i}}{i=1}^{n{s}} denote the finite search-task set. If τ t ​ i H t , B \tau_{ti}^{H_{t},B} is the rollout of ( 𝖠 , H t ) (\mathsf{A},H_{t}) on U i U_{i} , define the observed score R ^ B , D s ​ ( 𝖠 , H t ) \widehat{R}{B,D{s}}(\mathsf{A},H_{t}) and the population score R B ​ ( 𝖠 , H ) R_{B}(\mathsf{A},H) : The outer loop therefore evaluates H 0 , H 1 , … , H T H_{0},H_{1},\ldots,H_{T} using the scores. After breaking ties by a fixed rule, the search-optimal harness among the evaluated candidates is: The Proposer’s direct objective is to increase R ^ B rel , D s ​ ( 𝖠 , H t ) \widehat{R}{B{\mathrm{rel}},D_{s}}(\mathsf{A},H_{t}) by modifying the harness, rather than to ensure that each task is solved through the intended capability. Because it can inspect prompts, memory, retrieval logic, code, tool calls, and execution traces, the resulting harness may achieve a higher search score by exploiting a benchmark-wide shortcut. If a shortcut mechanism is known, an exact transformation may construct a neutralized benchmark B 0 := Φ 0 ​ ( B rel ) B_{0}:=\Phi_{0}(B_{\mathrm{rel}}) through Φ 0 ​ ( B rel ) = ( U , V 0 , Q 0 , Y ) \Phi_{0}(B_{\mathrm{rel}})=(U,V_{0},Q_{0},Y) . A valid Φ 0 \Phi_{0} removes the specified protocol-level correlation while leaving U U , Y = ψ ⁡ ( U ) Y=\psi(U) , and the semantic-task distribution P P unchanged. Write R 0 := R B 0 R_{0}:=R_{B_{0}} and R rel := R B rel R_{\rm rel}:=R_{B_{\mathrm{rel}}} . Define G rel ​ ( H , H 0 ) = R rel ​ ( 𝖠 , H ) − R rel ​ ( 𝖠 , H 0 ) G_{\rm rel}(H;H_{0})=R_{\rm rel}(\mathsf{A},H)-R_{\rm rel}(\mathsf{A},H_{0}) and G 0 ​ ( H , H 0 ) = R 0 ​ ( 𝖠 , H ) − R 0 ​ ( 𝖠 , H 0 ) G_{0}(H;H_{0})=R_{0}(\mathsf{A},H)-R_{0}(\mathsf{A},H_{0}) . Then, we define Δ BS ​ ( H , H 0 ) \Delta_{\rm BS}(H;H_{0}) with: where G rel ​ ( H , H 0 ) G_{\mathrm{rel}}\left(H;H_{0}\right) is the improvement observed during harness optimization, while G 0 ​ ( H , H 0 ) G_{0}\left(H;H_{0}\right) is the improvement that survives shortcut neutralization. The remaining Δ BS ​ ( H , H 0 ) \Delta_{\mathrm{BS}}\left(H;H_{0}\right) is the extra gain obtained by the “bad genius” Proposer through increased reliance on the benchmark-wide shortcut rather than improved task-solving capability. Task holdout probes a different axis ( 29 ) . Let D h D_{h} be a test-task set disjoint from D s D_{s} , and let R B , D ( 𝖠 , H ) := 𝔼 U ∼ P ^ D 𝔼 τ ∼ p 𝖠 ( ⋅ ∣ H , B , U ) [ r B ( τ , U ) ] R_{B,D}(\mathsf{A},H):=\mathbb{E}{U\sim\widehat{P}{D}}\mathbb{E}{\tau\sim p{\mathsf{A}}(\cdot\mid H,B,U)}[r_{B}(\tau,U)] denote expected average score on a finite task set D D , where P P is replaced by the empirical distribution P ^ D \widehat{P}{D} . The excess search-set gain is: A task-specific shortcut can make Δ TS ​ ( H , H 0 ) > 0 \Delta{\rm TS}(H;H_{0})>0 . A benchmark-wide shortcut can persist in both D s D_{s} and D h D_{h} because both use the same ( V rel , Q rel ) (V_{\rm rel},Q_{\rm rel}) . Therefore Δ TS ​ ( H , H 0 ) ≈ 0 \Delta_{\rm TS}(H;H_{0})\approx 0 does not imply Δ BS ​ ( H , H 0 ) ≈ 0 \Delta_{\rm BS}(H;H_{0})\approx 0 . The goal is G rel ​ ( H , H 0 ) ≈ G 0 ​ ( H , H 0 ) > 0 G_{\rm rel}(H;H_{0})\approx G_{0}(H;H_{0})>0 together with Δ BS ​ ( H , H 0 ) ≈ 0 \Delta_{\rm BS}(H;H_{0})\approx 0 . The exact B 0 B_{0} above defines the target estimand when the shortcut mechanism and its neutralization are known. In general, neither is available. CHASE therefore replaces the single B 0 B_{0} with a family of validity-preserving counterfactual benchmarks. Index each protocol transformation by b b . Each Φ b \Phi_{b} acts on benchmark configurations; its action on the released benchmark preserves the same semantic task and maps: Write Valid ⁡ ( Φ b ) = 1 \operatorname{Valid}(\Phi_{b})=1 when the transformation preserves the semantic task, target, evidence, resources, and scoring semantics while changing only the declared protocol surface. The valid family is ℬ val = { B b = Φ b ​ ( B rel ) : Valid ⁡ ( Φ b ) = 1 } \mathcal{B}{\mathrm{val}}={B{b}=\Phi_{b}(B_{\mathrm{rel}}):\operatorname{Valid}(\Phi_{b})=1} , and the identity transformation includes B rel B_{\mathrm{rel}} in ℬ val \mathcal{B}{\mathrm{val}} . Validity Firewall. The benchmark-specific firewall sets Valid ⁡ ( Φ b ) = 1 \operatorname{Valid}(\Phi{b})=1 only when all executable checks comparing B rel B_{\mathrm{rel}} with Φ b ​ ( B rel ) \Phi_{b}(B_{\mathrm{rel}}) pass; otherwise, the proposal is rejected. These checks include preserving the semantic task and correct answer and restricting changes to the declared protocol surface. The complete set of checks is given in Supplementary B.3 . For any candidate H H and B b ∈ ℬ val B_{b}\in\mathcal{B}{\mathrm{val}} , let R b = R B b R{b}=R_{B_{b}} and define G b ​ ( H , H 0 ) = R b ​ ( 𝖠 , H ) − R b ​ ( 𝖠 , H 0 ) G_{b}(H;H_{0})=R_{b}(\mathsf{A},H)-R_{b}(\mathsf{A},H_{0}) . Taking B b = B rel B_{b}=B_{\mathrm{rel}} or B b = B 0 B_{b}=B_{0} gives G rel ​ ( H , H 0 ) G_{\rm rel}(H;H_{0}) or G 0 ​ ( H , H 0 ) G_{0}(H;H_{0}) , respectively. The released gain destroyed by B b B_{b} is Δ b ​ ( H , H 0 ) = G rel ​ ( H , H 0 ) − G b ​ ( H , H 0 ) \Delta_{b}(H;H_{0})=G_{\rm rel}(H;H_{0})-G_{b}(H;H_{0}) , which is the central estimand. When B b = B 0 B_{b}=B_{0} , the contrast reduces to Δ BS ​ ( H , H 0 ) \Delta_{\rm BS}(H;H_{0}) , so the counterfactual family directly extends exact neutralization. Negative Δ b \Delta_{b} means that the harness gain increases rather than decreases under the counterfactual. CHASE then alternates between a Proposer and a Challenger. Since ℬ val \mathcal{B}{\mathrm{val}} is an unknown infinite family, before round t t , the Proposer has access to a finite archive 𝒜 t − 1 ⊂ ℬ val \mathcal{A}{t-1}\subset\mathcal{B}{\mathrm{val}} containing B rel B{\mathrm{rel}} and all previously confirmed counterfactuals. At the population level, the ideal Proposer solves: where ε ≥ 0 \varepsilon\geq 0 controls how much of the released-benchmark gain may disappear under any archived counterfactuals. The ideal Challenger targets the resulting H t H_{t} by searching for an executable violation, and returns code for transformation Φ b t \Phi_{b_{t}} . The criterion in equation 1 is inadequate for CHASE, since a high-scoring harness may violate counterfactual constraints discovered in later rounds. After updating the archive, we therefore select the final harness from those evaluated during evolution: Before final selection, we evaluate any missing harness–benchmark pairs among H 0 , … , H T H_{0},\ldots,H_{T} and 𝒜 T \mathcal{A}{T} on D evo D{\rm evo} , and then apply the archive-feasibility rule. The objectives in equations 2 – 4 above are written at the population level; implementation and the analysis below use their empirical counterparts. For any finite task set D D , define R ^ B , D ​ ( 𝖠 , H ) := 1 | D | ​ ∑ U i ∈ D r B ​ ( τ i H , B , U i ) , G ^ b , D ​ ( H , H 0 ) := R ^ B b , D ​ ( 𝖠 , H ) − R ^ B b , D ​ ( 𝖠 , H 0 ) \widehat{R}{B,D}(\mathsf{A},H):=\frac{1}{|D|}\sum{U_{i}\in D}r_{B}(\tau_{i}^{H,B},U_{i}),\widehat{G}{b,D}(H;H{0}):=\widehat{R}{B{b},D}(\mathsf{A},H)-\widehat{R}{B{b},D}(\mathsf{A},H_{0}) , and Δ ^ b , D ​ ( H , H 0 ) := G ^ rel , D ​ ( H , H 0 ) − G ^ b , D ​ ( H , H 0 ) \widehat{\Delta}{b,D}(H;H{0}):=\widehat{G}{{\rm rel},D}(H;H{0})-\widehat{G}{b,D}(H;H{0}) . When a task is evaluated with multiple rollouts, the rollout scores are first averaged within task. To separate search, confirmation, and final certification, the evaluation tasks are assigned disjoint roles: Here D evo D_{\rm evo} supplies the feedback used by the Proposer to search H t H_{t} in equation 2 , D disc D_{\rm disc} supplies the feedback used by the Challenger to search for Φ b t \Phi_{b_{t}} in equation 3 , D conf , t D_{\rm{conf},t} is used to confirm Φ b t \Phi_{b_{t}} after it has been fixed, and D cert D_{\rm cert} remains sealed until final certification. This separation reserves fresh tasks for confirmation and final certification, extending the search–test distinction ( D s , D h ) (D_{s},D_{h}) in Section 3.1 . Since ℬ val \mathcal{B}{\mathrm{val}} is unknown and cannot be exhaustively searched, CHASE maintains the sequence of finite archives { 𝒜 t } t = 0 T {\mathcal{A}{t}}{t=0}^{T} . Initialize 𝒜 0 = { B rel } \mathcal{A}{0}={B_{\mathrm{rel}}} . Once the Challenger proposes Φ b t \Phi_{b_{t}} , its executable code is fixed and its contrast is re-estimated on D conf , t D_{\rm{conf},t} . For a confirmation threshold η conf , t > 0 \eta_{\rm{conf},t}>0 , let 𝖢𝗈𝗇𝖿 t := { Δ ^ b t , D conf , t ( H t ; H 0 ) ≥ η conf , t } \mathsf{Conf}{t}:=\mathbf{1}!\left{\widehat{\Delta}{b_{t},D_{\rm{conf},t}}(H_{t};H_{0})\geq\eta_{\rm{conf},t}\right} . The archive is then updated by: In practice, each archived B b t B_{b_{t}} is stored together with the executable code for Φ b t \Phi_{b_{t}} . Thus a Challenger proposal becomes a constraint on subsequent harness evolution only when it is valid and its effect is confirmed on D conf , t D_{\rm{conf},t} . To summarize performance over any finite counterfactual archive 𝒜 ⊂ ℬ val \mathcal{A}\subset\mathcal{B}{\mathrm{val}} , define: with empirical counterparts on a task set D D , Here, Γ 𝒜 ​ ( H , H 0 ) \Gamma{\mathcal{A}}(H;H_{0}) is the largest gain destruction, G min , 𝒜 ​ ( H , H 0 ) G_{\min,\mathcal{A}}(H;H_{0}) is the smallest surviving gain over 𝒜 \mathcal{A} , and we have G min , 𝒜 ​ ( H , H 0 ) + Γ 𝒜 ​ ( H , H 0 ) = G rel ​ ( H , H 0 ) G_{\mathrm{min},\mathcal{A}}(H;H_{0})+\Gamma_{\mathcal{A}}(H;H_{0})=G_{\mathrm{rel}}(H;H_{0}) . The set D cert D_{\rm cert} is used once to evaluate the constraints in 𝒜 T \mathcal{A}{T} . Figure 2 summarizes the complete CHASE information flow. Theoretical analyses of harness self-evolution remain limited. Recent work studies finite-data certification and safe adoption under a fixed task distribution, focusing on expected-reward improvement and retention of prior behavior ( 4 ) . It does not model benchmark-protocol dependence or counterfactual discovery. We therefore ask what a finite counterfactual archive can certify about B 0 B{0} and, when it does not yet recover B 0 B_{0} , what can be concluded from sequential Challenger search. For valid Φ b , Φ b ′ \Phi_{b},\Phi_{b^{\prime}} , define B b ′ ∘ b := Φ b ′ ​ ( Φ b ​ ( B rel ) ) B_{b^{\prime}\circ b}:=\Phi_{b^{\prime}}(\Phi_{b}(B_{\mathrm{rel}})) by applying Φ b \Phi_{b} first and Φ b ′ \Phi_{b^{\prime}} second. This composition maps ( U , V rel , Q rel , Y ) (U,V_{\rm rel},Q_{\rm rel},Y) to ( U , V b ′ ∘ b , Q b ′ ∘ b , Y ) (U,V_{b^{\prime}\circ b},Q_{b^{\prime}\circ b},Y) , and we assume ℬ val \mathcal{B}{\mathrm{val}} is closed under it. For nonempty finite 𝒜 ⊂ ℬ val \mathcal{A}\subset\mathcal{B}{\mathrm{val}} , define: Thus K ⁡ ( 𝒜 ) = ∞ K(\mathcal{A})=\infty if no composition of transformations in 𝒜 \mathcal{A} produces B 0 B_{0} . Also define: Thus K ⁡ ( 𝒜 ) K(\mathcal{A}) is the smallest number of archived transformations whose composition produces B 0 B_{0} , while ρ \rho measures the worst-case excess gain destruction under composition. Equivalently, Δ b ′ ∘ b ​ ( H , H 0 ) ≤ Δ b ​ ( H , H 0 ) + Δ b ′ ​ ( H , H 0 ) + ρ \Delta_{b^{\prime}\circ b}(H;H_{0})\leq\Delta_{b}(H;H_{0})+\Delta_{b^{\prime}}(H;H_{0})+\rho . Fix α ∈ ( 0 , 1 ) \alpha\in(0,1) . We obtain the following results. Let 𝒜 ⊂ ℬ val \mathcal{A}\subset\mathcal{B}{\mathrm{val}} be nonempty and finite. If 𝒜 \mathcal{A} and H ⋆ H^{\star} are fixed before D cert D{\rm cert} is opened and K ⁡ ( 𝒜 ) < ∞ K(\mathcal{A})<\infty , then, under the conditions in Supplementary A , with probability at least 1 − α / 2 1-\alpha/2 : Theorem 1 converts a finite archive into guarantees for B 0 B_{0} . The bounds tighten as K ⁡ ( 𝒜 ) K(\mathcal{A}) and ρ \rho decrease. If B 0 ∈ 𝒜 B_{0}\in\mathcal{A} , then K ⁡ ( 𝒜 ) = 1 K(\mathcal{A})=1 and every term containing ρ \rho disappears. For each search set D D , let ℜ D \mathfrak{R}{D} denote the expected absolute Rademacher complexity of its released-score and gain-destruction functions, defined in Supplementary A.1 . We use r = max D ∈ { D evo , D disc } ⁡ { 2 ​ ℜ D + 8 ​ log ⁡ ( 8 / α ) / | D | } r=\max{D\in{D_{\rm evo},D_{\rm disc}}}\left{2\mathfrak{R}{D}+\sqrt{8\log(8/\alpha)/|D|}\right} which bounds the empirical-to-population deviation of the released score and gain destruction over the candidates considered. For confirmation, set η conf , t = ε + γ + 8 ​ log ⁡ ( 8 ​ T / α ) / | D conf , t | , 0 < γ ≤ 1 \eta{{\rm conf},t}=\varepsilon+\gamma+\sqrt{8\log(8T/\alpha)/|D_{{\rm conf},t}|},0<\gamma\leq 1 . Supplementary A discusses ℜ D \mathfrak{R}{D} and γ \gamma . Suppose Valid ⁡ ( Φ b t ) = 1 \operatorname{Valid}(\Phi{b_{t}})=1 and Δ ^ b , D evo ​ ( H t , H 0 ) ≤ ε \widehat{\Delta}{b,D{\rm evo}}(H_{t};H_{0})\leq\varepsilon for every B b ∈ A t − 1 B_{b}\in A_{t-1} at each round. Assume that, for every 0 < q ≤ 1 0<q\leq 1 , any m m harnesses with sup B b ∈ ℬ val | Δ b ​ ( H ( j ) , H 0 ) − Δ b ​ ( H ( k ) , H 0 ) | ≥ q \sup_{B_{b}\in\mathcal{B}{\mathrm{val}}}|\Delta{b}(H^{(j)};H_{0})-\Delta_{b}(H^{(k)};H_{0})|\geq q , for all j ≠ k j\neq k , obey m ≤ ( C / q ) d m\leq(C/q)^{d} , for constants C ≥ 1 C\geq 1 and d > 0 d>0 . If r < γ r<\gamma , then, with probability at least 1 − α / 2 1-\alpha/2 , Consequently, if T > ( C γ − r ) d T>\left(\frac{C}{\gamma-r}\right)^{d} , then 𝖢𝗈𝗇𝖿 t = 0 \mathsf{Conf}{t}=0 for at least one t ≤ T t\leq T . Theorem 2 shows that each counterfactual forces a nontrivial change in the subsequent harness, so such admissions cannot continue indefinitely. In particular, for a sufficiently large T T , the process must reach a round with 𝖢𝗈𝗇𝖿 t = 0 \mathsf{Conf}{t}=0 . We next characterize what can be concluded at such a round. Suppose H t H_{t} and B b t B_{b_{t}} solve the empirical counterparts of equation 2 and equation 3 . On the common evolution event, of probability at least 1 − α / 2 1-\alpha/2 , every round with a valid tested proposal and 𝖢𝗈𝗇𝖿 t = 0 \mathsf{Conf}_{t}=0 satisfies: The comparison set is nonempty when ε ≥ r \varepsilon\geq r . Theorem 3 gives the complementary conclusion: a non-confirmed round certifies small gain destruction over the full valid family, up to estimation error, while retaining near-optimal released gain among uniformly feasible harnesses. Thus, confirmed rounds expand the archive, whereas a non-confirmed round provides an approximate stopping certificate for the current harness. OfficeQA is a new benchmark suite for end-to-end grounded reasoning over dense U.S. government financial documents ( 19 ) . Its questions require document discovery, text and table retrieval, numerical reasoning, and exact answer extraction. OfficeQA Full has 246 questions; Pro V2 adds 90 questions over a separate receipts-and-expenditures corpus ( 6 ) . Full supports harness optimization, and Pro V2 provides a challenging public evaluation. OfficeQA is sensitive to harness design because search, evidence handling, tool use, computation, and answer formatting are harness-controlled. At a fixed model, EnvHarness improved OfficeQA exact match from 54.40 54.40 to 56.20 56.20 and token-level F1 from 55.77 55.77 to 57.73 57.73 , compared with skills extracted from the original environments ( 10 ) . We retain all released questions and answers and let the agent 𝖠 \mathsf{A} search the full 697-document transformed-text corpus for OfficeQA Full. Supplementary C gives release, corpus, and scoring details. The 246 questions are divided into evolution, Challenger discovery, Challenger confirmation, and final certification sets of sizes 49, 49, 72, and 76, respectively. We use a fixed budget of three Proposer rounds for all optimized methods. The confirmation set is partitioned into three disjoint round-specific blocks, each of size 24 24 . Pro V2 is reported separately as a cross-corpus analysis. Our use of task splits follows prior OfficeQA evaluations ( 1 ; 27 ) ; we additionally report Pro V2 results, which were not reported in those studies.