How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.20474 · published 2026-09-17 · submitted 17 Sep 2026

Abstract

Agent harnesses supply planning guidance, organize execution, and check completion. We study how these components affect success, erroneous acceptance, and cost in two Retail experiments and an Airline pilot in $τ^2$-bench. The primary comparison pairs prewritten task-specific plans (Fixed) with shuffled policy text matched in word count (Sham), isolating the contribution of guidance content. Across 265 matched cells, Fixed improves oracle-verified success by 7.17 percentage points (90% task-clustered bootstrap interval, 1.15--13.36 points), with gains concentrated in higher-complexity tasks. A read-only terminal verifier rejects 61% of Retail oracle-invalid episodes while withholding 17% of correct ones, at less than one cent of additional cost per episode. Which component matters more depends on the loss assigned to erroneous acceptance: at low liability the planning gain dominates; at high liability the verifier's avoided false passes dominate---and a standalone verifier captures nearly all the false-pass benefit of the full planning-plus-verification stack at a fraction of its cost.

Authors

Yukun Zhang, Kemu Xu, Yishen Chen

Key claims (verbatim-leaning English extract)

• Agent harnesses supply planning guidance, organize execution, and check completion. We study how these components affect success, erroneous acceptance, and cost in two Retail experiments and an Airline pilot in $τ^2$-bench. The primary comparison pairs prewritten task-specific plans (Fixed) with shuffled policy text matched in word count (Sham), isolating the contribution of guidance content. Across 265 matched cells, Fixed improves oracle-verified success by 7.17 percentage points (90% task-clustered bootstrap interval, 1.15--13.36 points), with gains concentrated in higher-complexity tasks. A read-only terminal verifier rejects 61% of Retail oracle-invalid episodes while withholding 17% of correct ones, at less than one cent of additional cost per episode. Which component matters more depends on the loss assigned to erroneous acceptance: at low liability the planning gain dominates; at high liability the verifier's avoided false passes dominate---and a standalone verifier captures nearly all the false-pass benefit of the full planning-plus-verification stack at a fraction of its cost.

• Primary comparison: Fixed (prewritten task-specific plans) vs Sham (shuffled policy text matched in word count) isolates guidance content.

• Across 265 matched cells, Fixed improves oracle-verified success by 7.17 percentage points (90% task-clustered bootstrap interval 1.15–13.36), concentrated in higher-complexity tasks.

• A read-only terminal verifier rejects 61% of Retail oracle-invalid episodes while withholding 17% of correct ones, at less than one cent additional cost per episode; false-pass rate falls from 57.21% to 20.96%.

• Which component matters more depends on liability for erroneous acceptance: at low liability planning gain dominates; verification captures most avoided false-pass benefit at a twelfth of incremental cost.

• How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents # How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents

• Large language model agents retrieve information, invoke tools, and modify persistent environment states. Their outcomes depend on both the model and the surrounding mechanisms that organize its work: planning, repair, memory, tool orchestration, and completion checks. We call this execution and control structure an agent harness. Reasoning–action integration, reflective feedback, and memory management illustrate how external organization shapes the behavior of a given model (Yao et al., 2023b; Shinn et al., 2023; Packer et al., 2023).

• Interactive benchmarks and system-level comparisons measure what models and harnesses achieve together (Liu et al., 2024; Ma et al., 2024; Yao et al., 2025; Kapoor et al., 2026). Attributing that performance to a component requires a more specific comparison. Additional guidance, inference expenditure, execution constraints, and repair opportunities can all change success. A useful component evaluation states what support changes, which outcome it affects, and what resources it consumes.

• 1.2 Planning Information, Release Control, and Operating Cost ### 1.2 Planning Information, Release Control, and Operating Cost

• Planning supplies subgoals, dependencies, and execution guidance that an executor may find useful (Zhou et al., 2023; Wang et al., 2023a; Erdogan et al., 2025). Comparing a supplied plan with Minimal changes both the guidance and the amount of context. A control matched in word count and packaging helps isolate the contribution of the supplied task-specific content.

Structure (section headings from HTML)

• How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents
• 1 Introduction
• 1.1 The Attribution Problem in Agent Performance
• 1.2 Planning Information, Release Control, and Operating Cost
• 1.3 Research Questions and Experimental Approach
• 1.4 Main Findings and Contributions
• 2 Related Work
• 3 Evaluation Framework: Planning Information, Verification, and Conditional Harness Value
• 3.1 Harness Components and Their Intervention Points
• 3.2 Observed Outcomes and Resource Use
• 3.3 Planning Information and Model–Task Heterogeneity
• 3.4 Verification and the Interpretation of False Pass
• 3.5 Conditional Value under Cost and Risk
• 4 Experimental Design, Data, and Evidence Scope
• 4.1 Evaluation Environments and Task Coverage
• 4.1.1 Retail: Shared Experiment
• 4.1.2 Retail: Planner-Focused Experiment
• 4.1.3 Airline: Pilot-Scale Evaluation
• 4.2 Models and Harness Configurations
• 4.3 Fixed Plans, Sham Controls, and Self-Planning
• 4.4 Recorded Outcomes and Resource Measures
• 4.5 Analysis Samples, Matching, and Missingness
• 4.6 Statistical Procedures and Evidence Hierarchy
• 5 Planning Information: Evidence from the Fixed–Sham Comparison
• 5.1 Average Planning Effect in the Retained Matched Sample

Remainder

Full original English text: see html_url / source_url / pdf_url in frontmatter.