Note: body below is original English text extracted from https://arxiv.org/html/2609.01437. Do not treat this file as a translation.

arXiv:2609.01437v1 [cs.SE] 01 Sep 2026

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

September 1, 2026

affiliation: ByteDance Seed; Singapore University of Technology and Design; Georgia Institute of Technology; M-A-P; TokenWave.AI Project Page: https://self-developing-agents.github.io/

Abstract

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability—task success on held-out benchmarks, and efficiency—execution-token cost. The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Motivating fact (Introduction, verbatim)

Its impact is substantial: with identical weights, GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 but 49.6% inside Codex CLI (The Terminal-Bench Team, 2026).

Two stages

Weak seed H_seed: a runnable compatibility layer, not a task-solving agent. Unmodified, it produces an empty or partial artifact and scores zero on every downstream benchmark.

Creation coverage

Creators evaluated: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, Seed 2.0 Pro.

Key findings (paper conclusions, condensed from abstract + §4)

Harness Creation: Model-built harnesses match reference on short-form writing and exceed it on ML experimentation; largest gap on search/research; substantial gap remains on code. Cost varies widely (MLE-bench ~19× token spread) without reliable score correlation.

Harness Evolution: All five self-runtime creators improve on visible feedback pair, but gains shrink on held-out. Opus 4.8 largest held-out improvement (+4.44). Under fixed Gemini executor, only Opus improves held-out; others regress. Feedback and held-out move same direction only 53.1% of switches; 2/9 declared versions are held-out optimal.

Portability: A runnable harness can be used by another model, but capability transfers only when prompts, tool protocol, budgets, and stopping rules remain compatible. Example: Opus Self-Eval SWE-Pro 69.3 → 33.0 under Gemini when hard-coding a 120-step limit around the original executor.

Remainder

Full original English text: see html_url / source_url / pdf_url in frontmatter. Do not invent missing sections from this partial extract.