Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.
arXiv:2609.12742 · published 2026-09-11
Abstract
Coding agents increasingly read repository knowledge from SKILLs --- plain .md files versioned alongside the code. Recent work synthesizes these files automatically, by optimizing the document against a benchmark. A bare repository comes with no benchmark, and the synthetic tasks prior work builds are small enough that a capable agent saturates them with no document at all. We mine harder tasks --- merged pull requests of the repository, reverted at a single frozen base commit; and score a candidate document by whether the same agent does better with it than without it. On three Kotlin repositories, the documents GEPA finds raise this score by 4.9pp on average, and the ones SkillOpt finds leave it where it started, 0.1pp above the seed. The GEPA gain matches what prior work reports with the same optimizer, and at the dataset size a single repository supplies it cannot be separated from the agent's run-to-run variance; settling that would take more tasks than one repository's history yields. The documents themselves read better than the score: a maintainer of one repository found in them knowledge one only gets by working in the project.
Key claims (verbatim-leaning English extract)
- Coding agents increasingly read repository knowledge from SKILLs --- plain .md files versioned alongside the code.
- Recent work synthesizes these files automatically, by optimizing the document against a benchmark.
- A bare repository comes with no benchmark, and the synthetic tasks prior work builds are small enough that a capable agent saturates them with no document at all.
- We mine harder tasks --- merged pull requests of the repository, reverted at a single frozen base commit; and score a candidate document by whether the same agent does better with it than without it.
- On three Kotlin repositories, the documents GEPA finds raise this score by 4.9pp on average, and the ones SkillOpt finds leave it where it started, 0.1pp above the seed.
- The GEPA gain matches what prior work reports with the same optimizer, and at the dataset size a single repository supplies it cannot be separated from the agent's run-to-run variance; settling that would take more tasks than one repository's history yields.
- The documents themselves read better than the score: a maintainer of one repository found in them knowledge one only gets by working in the project.
Remainder
Full original English text: see html_url / source_url / pdf_url in frontmatter.