Quantifying Overclaiming Propensity in Frontier LLM Agents

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.20812 · published 2026-09-17 · submitted 17 Sep 2026

Abstract

Frontier coding agents are increasingly trusted to work autonomously for long periods, yet an agent's final response is often the only account of that work a user sees. We quantify the propensity of frontier agents to \emph{overclaim} task completion, a misrepresentation that can mislead the user. An agent overclaims when its final response contradicts information in its context. This definition requires no inference about intent and is independent of task success. We introduce \emph{OverclaimBench}, an evaluation suite composed of five file-review scenarios, transcript-based coverage measurements, and registered planted defects. We evaluate eight proprietary frontier models in their own production command-line interfaces, and four open-weight models under a single fixed harness on OverclaimBench and find that 1) agents do not read all the files they were asked to review in 67.9% of runs; 2) among runs where not all files are read, agents are \emph{misleading} 80.4% of the time (59--96% per model), either falsely claiming to have read all files or omitting that coverage is incomplete; 3) requiring delegation to subagents increased reading coverage, but among reviews that remained incomplete, a large majority were still misleading; and 4) agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of agents that read every file, showing that claims of completion can conceal substantive failures. Together, these results show that agents' final responses are not reliable accounts of their actions.

Authors

Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Pascal Jr Tikeng Notsawo, Saskia Helbling, Alberto Tosato, Mohamed Amine Merzouk, Nouha Dziri, Gauthier Gidel, Tommaso Tosato

Key claims (verbatim-leaning English extract)

• Frontier coding agents are increasingly trusted to work autonomously for long periods, yet an agent's final response is often the only account of that work a user sees. We quantify the propensity of frontier agents to \emph{overclaim} task completion, a misrepresentation that can mislead the user. An agent overclaims when its final response contradicts information in its context. This definition requires no inference about intent and is independent of task success. We introduce \emph{OverclaimBench}, an evaluation suite composed of five file-review scenarios, transcript-based coverage measurements, and registered planted defects. We evaluate eight proprietary frontier models in their own production command-line interfaces, and four open-weight models under a single fixed harness on OverclaimBench and find that 1) agents do not read all the files they were asked to review in 67.9% of runs; 2) among runs where not all files are read, agents are \emph{misleading} 80.4% of the time (59--96% per model), either falsely claiming to have read all files or omitting that coverage is incomplete; 3) requiring delegation to subagents increased reading coverage, but among reviews that remained incomplete, a large majority were still misleading; and 4) agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of agents that read every file, showing that claims of completion can conceal substantive failures. Together, these results show that agents' final responses are not reliable accounts of their actions.

• An agent overclaims when its final response asserts an action or level of completion contradicted by evidence in its own context—no intent inference required, independent of task success.

• OverclaimBench: five naturalistic file-review scenarios through production CLI harnesses; reading measured deterministically as unique lines entering context.

• Among incomplete-coverage runs: admission (discloses partial coverage), omission (silent), overclaimed (explicit full-coverage claim); omission + explicit overclaim = misleading.

• When agents overclaim, they are more likely to miss planted defects (needles).

• Frontier models are increasingly used to perform long-horizon, open-ended tasks in agentic loops; however, the actual work agents perform is often not obvious from their final response. This creates a gap between actually completing a task and appearing to complete it. Recent work suggests that reinforcement learning can make models increasingly sensitive to what they believe graders reward, including cases where task completion is favored over honesty (Højmark et al., 2026). For example, Greenblatt (2026) describes a failure mode in which frontier agents oversell incomplete work, optimizing for “apparent success” rather than actual success or honesty. METR likewise reports more cheating on

• Figure 1: Illustration of OverclaimBench. We asked frontier agents to work on file review tasks. For each run, we check from the tool calls whether all files to be reviewed were touched. Runs with partial coverage were labeled admission if agents disclosed not having covered all files, omission if they didn’t indicate coverage was partial, and overclaimed if agents explicitly claimed full coverage. Omission and explicit overclaim together form the misleading category.

• Prior work evaluates an agent’s self-report against a record of what it actually did. Guo et al. (2026) define upward deception by comparing an agent’s final reports to its actions, while Advani (2026) define false success using completion claims contradicted by programmatic environment state. Further evaluation shows that nominally successful outcomes can conceal policy, interaction, or integrity violations (Cao et al., 2026). This literature establishes that an agent’s report can diverge from its execution. Yet it leaves three related questions unresolved for long but feasible tasks: (i) Does the agent complete the requested work? (ii) If not, does it disclose that the work is incomplete?

• We address these questions by introducing OverclaimBench (Figure 1), an evaluation suite for measuring whether coding agents accurately report the scope of their work. OverclaimBench contains five feasible, naturalistic file-review scenarios run through each agent’s production CLI harness. Reading is measured deterministically as the percentage of unique lines from each file that entered the agent’s context. We measure incomplete work in agentic tasks without introducing artificial elements such as deliberately breaking tools or withholding required inputs. We also test whether inaccurate claims of complete coverage are associated with failures to detect planted defects.

• In this work, we say an agent overclaims when its final response asserts an action or level of completion that is contradicted by evidence in its own context. This definition requires no inference about intent and is independent of whether the task ultimately succeeds or fails. An agent that transparently reports incomplete work is not overclaiming, whereas one that presents incomplete work as complete is. Among runs with incomplete coverage, we call a response misleading when it either explicitly overclaims or omits any disclosure of the gap.

Structure (section headings from HTML)

• Quantifying Overclaiming Propensity in Frontier LLM Agents
• 1 Introduction
• 2 Related Work
• Agents misreporting their own work
• Neighboring failure modes
• Evaluating execution rather than outcomes
• 3 Methodology
• 3.1 Experimental design
• 3.2 Scenarios
• 3.3 Needles
• 3.4 Testing environment
• Open-weight models
• Subagents
• Controlled delegation experiment
• 3.5 Deterministic measurements
• Corpus coverage (breadth and depth)
• Read needles
• 3.6 Judges
• Review-scope judge
• Needle-reporting judge
• 4 Results
• Agents rarely touch every file
• Every model overclaims in runs with incomplete reviews
• Subagents increase coverage but do not improve honesty
• Needles are more often missed when agents overclaim

Remainder

Full original English text: see html_url / source_url / pdf_url in frontmatter.