second brain
source
← 首页

external-source

DeltaSelect: Affordable A/B Testing for Coding Agents

DeltaSelect: Affordable A/B Testing for Coding Agents

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.19607 · published 2026-09-17 · submitted 17 Sep 2026

Abstract

Coding-agent benchmarks are built for broad and comprehensive comparisons, not frequent development decisions. Individual runs vary, full suites are expensive, and the benchmark harness may differ from the harness used in practice. In a resampling analysis of DeepSWE's published trials, only 19.5% of tasks (22 of 113) had a fifth-percentile Pearson correlation of at least 0.50 with full-benchmark performance. The paper presents DeltaSelect, an open-source method that identifies tasks whose one-run results consistently track full-benchmark performance using Pearson correlation, maps fractional verifier results to a common score using linear regression, and selects a fixed task set within a dollar budget. DeltaSelect is intended for repeated baseline-versus-candidate comparisons during development, not model rankings. In a gpt-5.6-luna low-reasoning case study, DeltaSelect was used to revise custom skills and instructions. Across 13 evaluations, the recorded cost was USD 27.86 at rates published August 16, 2026. The adopted version cost 58.1% less than the initial version (USD 1.75 versus USD 4.18; p=0.008), while the calibrated score was higher (42.36% versus 36.46%; published-analog variance p=0.326).

Authors

Nicholas J. Conn

Key claims (verbatim-leaning English extract)

Structure (section headings from HTML)

(section headings from HTML)

Body excerpts (original English)

Coding-agent benchmarks are built for broad and comprehensive comparisons, not frequent development decisions. Individual runs vary, full suites are expensive, and the benchmark harness may differ from the harness used in practice. In a resampling analysis of DeepSWE’s published trials, only 19.5% of tasks (22 of 113) had a fifth-percentile Pearson correlation of at least 0.50 with full-benchmark performance. The paper presents DeltaSelect, an open-source method that identifies tasks whose one-run results consistently track full-benchmark performance using Pearson correlation, maps fractional verifier results to a common score using linear regression, and selects a fixed task set within a dollar budget. DeltaSelect is intended for repeated baseline-versus-candidate comparisons during development, not model rankings. In a gpt-5.6-luna low-reasoning case study, DeltaSelect was used to revise custom skills and instructions. Across 13 evaluations, the recorded cost was $27.86 at rates published August 16, 2026. The adopted version cost 58.1% less than the initial version ($1.75 versus $4.18; p =0.008), while the calibrated score was higher (42.36% versus 36.46%; published-analog variance p =0.326). Developers routinely change skills and instructions, tune reasoning levels, switch models, and make other changes intended to improve their coding agents. Improvement may mean a higher score, a lower cost, fewer failures, or a useful tradeoff between these goals. Regardless of the goal, iteration is central to research and development. Comparisons must be low cost enough to repeat throughout the development process while providing enough statistical power to distinguish meaningful changes from run-to-run variation. Full benchmarks remain the right tool for broad comparisons, but their cost prevents routine use during development. This paper presents DeltaSelect, a method that uses repeated published trials to select a small, fixed task set within a specified budget and compare a baseline with a candidate. DeltaSelect is not a leaderboard, a universal model ranking, or a replacement for a full benchmark. The research question is specific: Can a single run per selected task provide useful evidence for frequent baseline-versus-candidate comparisons without full-benchmark cost? One barrier to low-cost comparison is how benchmark results are typically scored. Binary scoring discards useful information from each run. DeepSWE executes each task four times and scores each run as pass or fail [ Huang et al., 2026 ] . Repeating each task four times reduces the effect of run-to-run variation, but does not recover discarded partial progress and increases the benchmark’s scored-run cost four-fold. DeepSWE’s released verifier records contain more information. In addition to binary resolution, the records report the fraction of new fail-to-pass tests passed (F2P) and the fraction of existing pass-to-pass tests preserved (P2P). F2P distinguishes partial progress mapped to zero by binary scoring. F2P varies across runs and is not equally informative for every task. F2P also measures verifier tests, not the percentage of a task’s requirements completed: tests can overlap, differ in difficulty, or encode several assertions. Even with those limits, F2P preserves partial progress that binary scoring discards. An all-or-nothing result is too coarse for practical one-run comparisons. Reducing cost to one run per task then creates a task-selection problem. DeltaSelect ranks tasks by how consistently one published run correlates with full-benchmark performance across the observed configurations. The ranking does not judge a task’s broader benchmark value; it measures whether one run provides useful evidence for the development decision studied here. A low one-run correlation does not invalidate a task or diminish its value in a comprehensive benchmark. It means only that a single execution provides weak evidence for this development decision. Across 10,000 deterministic one-trial resamples, the median task’s fifth-percentile Pearson correlation with the full DeepSWE configuration score was 0.321. Only 22 of 113 tasks reached 0.50, and only two exceeded 0.70. Reliable one-run signal was limited to a small part of the benchmark (Figure 1 ). Ranked plot of 113 tasks showing fifth-percentile correlation, median correlation, and the fifth-to-95th-percentile resampling band. A second challenge with many leading benchmarks is the harness gap. A coding-agent score reflects the entire system used to produce it: the model, reasoning setting, client, tools, instructions, and execution harness. A published benchmark may deliberately fix a neutral harness to isolate other differences. A developer instead cares about the harness that will be used in practice, which may have different context management, tools, prompts, and control flow. On Terminal-Bench 2.1, holding model and reasoning effort fixed while changing the harness moved GPT-5.5 xhigh by 5.1 percentage points and Opus 4.7 max by 2.8 points [ Terminal-Bench, 2026b ] . Changing the harness can therefore move a score even when the model and reasoning level do not change. Selected tasks must be validated in the intended harness before use. Using frontier models creates another constraint: a comparison needs headroom. Frontier agents are beginning to saturate established coding benchmarks, making further improvements harder to distinguish [ OpenAI, 2026c ] . A baseline near 100% on every selected task leaves little room for a better candidate to raise the score. A baseline near 0% leaves no room to detect a further regression. Selecting tasks with usable headroom can mitigate the problem, but the mitigation is not free: those tasks may be less repeatable or more expensive. The choice is a tradeoff between headroom, repeatability, and price. Published results are used to build the task set and scoring model. The baseline must then be run in the exact harness and version intended for the comparison. That step cannot be skipped: changing or updating the harness can change task behavior, score, and cost. Only that target-harness baseline can establish the selected tasks’ headroom, repeatability, and starting score. DeltaSelect addresses this decision with a deliberately simple pipeline. The pipeline ranks tasks by fifth-percentile correlation across repeated public trials, maps each task’s F2P result onto the published full-score scale, weights tasks with smaller calibration errors more heavily, and selects the highest-ranked tasks that fit the model-specific dollar budget. Figure 2 shows these steps. The resulting tasks, repetitions, calibration coefficients, weights, price assumptions, snapshot checksum, and seeds are frozen before the first baseline-versus-candidate comparison and remain fixed throughout iteration. The method is intended to compare two versions of the same coding-agent system under a fixed specification. DeltaSelect is not a leaderboard, a universal model ranking, proof of unseen-task generalization, or a replacement for comprehensive benchmarks. A small correlated task set does not inherit the full benchmark’s breadth merely by tracking the historical full score. A deterministic task ranking based on repeated trials to account for run-to-run variation instead of ranking tasks from mean results alone. A calibrated score retaining fractional verifier evidence discarded by binary scoring while keeping raw F2P, P2P, and binary outcomes visible. A model- and effort-specific budget scan based on observed task prices without allowing price to change the reliability ranking. The central claim is intentionally narrow: repeated public trials can identify a small, fixed set of reliable tasks for affordable baseline-versus-candidate comparisons. In the published example, the selected tasks are projected to distinguish two materially different model–effort configurations at far lower scored-trial cost than the full benchmark. Target-harness runs show why the selected tasks must then be validated and baselined in the intended harness. The case study shows how a frozen task set can guide development toward higher score, lower cost, or both. Prior work provides several techniques for estimating model performance with fewer evaluations. The Active Testing technique uses adaptive sampling and importance weighting to estimate performance with fewer labels [ Kossen et al., 2021 ] . Active Surrogate Estimators adds learned surrogate predictions to the selection process [ Kossen et al., 2022 ] . Efficient Benchmarking allocates evaluation resources according to decision reliability and reports large reductions in HELM evaluation cost with limited reliability loss [ Perlitz and others, 2024 ] . These methods reduce evaluation cost by evaluating a selected subset, allocating runs where they are most informative, or using surrogate predictions. If examples are sampled at different rates, importance weighting gives each observed result a compensating influence so that the estimate represents the original evaluation set rather than the biased sample. Short-form benchmark construction provides an even closer precedent. tinyBenchmarks combines item-response models with anchor examples to estimate full-benchmark scores from small subsets [ Maia Polo and others, 2024b ] . Efficient Multi-Prompt Evaluation treats prompt choice as another measurement dimension [ Maia Polo and others, 2024a ] . BenTo reduces benchmarks at the task level using transferability and facility-location selection, explicitly considering redundancy [ Zhao et al., 2025 ] . SubLIME is the closest correlation-based analogue. SubLIME learns to select 1–20% of benchmark items preserving global rankings, using item features and partial evaluations from anchor models [ Saranathan and others, 2025 ] . SubLIME aims to preserve rankings across many models. DeltaSelect instead asks whether a change improved or worsened the same coding-agent system during development. DeltaSelect directly models uncertainty from repeated task executions, incorporates different dollar costs by task, maps fractional verifier results to a common reference scale, requires a target-harness baseline, and freezes a user-visible task set for longitudinal comparison. A task can help preserve the ranking of many models yet still be too noisy to compare two system versions from one run. What is new is the combination for a specific use case: frequent A/B testing of a coding-agent system during development. DeltaSelect uses repeated trials to rank task reliability, converts partial-credit results onto one score scale, selects tasks within a dollar budget, validates them in the target harness, and reports workflow cost. The task set and scoring rules then stay fixed while the system changes. DeltaSelect measures only run-to-run randomness by repeatedly sampling one of each task’s four published outcomes; it does not measure variation caused by data, training, or implementation differences [ Bouthillier et al., 2021 ] . The method is not specific to DeepSWE. Any benchmark can serve as input if it reports a full score across configurations and repeated task-level trials for estimating uncertainty. Terminal-Bench 2.1 is one example: its leaderboard requires at least five public trials per task and reports an aggregate task-resolution score [ Terminal-Bench, 2026a ] . The calibrated score has one intended use: compare a baseline and candidate on the fixed selected tasks. It should not be read as a general measure of software-engineering ability. Even when the subset tracks DeepSWE, it does not cover every capability tested by the full benchmark [ Messick, 1995 ] . SWE-bench evaluates whether an agent can fix real issues in software repositories [ Jimenez and others, 2024 ] . SWE-agent showed that the agent-computer interface materially affects performance [ Yang and others, 2024 ] . Agentless showed that a simpler workflow could be competitive and inexpensive, reinforcing the need to evaluate cost and outcome together [ Xia et al., 2025 ] . Harness-Bench adds recent controlled evidence that harness choices affect completion, cost, safety, and process outcomes [ Yao and others, 2026 ] . The current Terminal-Bench leaderboard provides contemporary same-model comparisons under native and neutral harnesses, although it offers first-party benchmark evidence rather than peer-reviewed causal evidence. DeepSWE provides the repeated long-horizon task matrix used here. DeepSWE uses original tasks, a fixed mini-swe-agent harness, binary rewards, and four trials per task–configuration cell; the DeepSWE paper also acknowledges the lack of partial credit and the tradeoff between harness standardization and native-product realism [ Huang et al., 2026 ] . SkillsBench evaluates skills across model–harness configurations with repeated task-macro pass rates and observes harness-mediated effects [ Li and others, 2026 ] . F2P is useful, but its numeric detail can imply more certainty than the verifier tests support. A fraction of F2P tests passed can distinguish a near-complete implementation from an attempt without meaningful progress. F2P does not establish that each test represents an equal, independent unit of the requirement. Mutation-testing research shows why suite adequacy matters: passing a weak suite can leave faults undetected [ Jia and Harman, 2011 ] . DeltaSelect therefore preserves raw F2P, P2P, binary resolution, and build or environment failures beside the aggregate score. Prior work also treats score and cost together. AI Agents That Matter argues for cost-controlled comparisons and accuracy–cost Pareto frontiers, and shows that repeated sampling can confound comparisons among agent architectures [ Kapoor et al., 2025 ] . DeltaSelect does not introduce the idea that agent score and cost should be evaluated jointly. DeltaSelect addresses measurement expense and attributes realized spend to the workflow being changed. DeltaSelect reports score and recorded model cost for every completed evaluation. A lower cost can be the desired improvement; showing both values lets the caller decide whether a score change, a cost change, or both justify adopting a candidate. DeltaSelect converts repeated trials from an existing benchmark into a fixed, budget-constrained task set for measuring changes between agent configurations. The method separates four easily conflated operations: task ranking, score construction, budget allocation, and target-harness validation. Ranking asks which tasks remain informative when run once. Score construction maps heterogeneous task fractions onto a common reference scale. Allocation asks which ranked tasks fit the execution budget. Target-harness validation runs the selected tasks in the intended harness to establish their actual baseline behavior, cost, and headroom before comparing a candidate. The input is a pinned benchmark snapshot containing repeated fractional F2P outcomes, full-benchmark scores for observed model–effort configurations, and trial prices. The analysis uses the DeepSWE v1.1 trial snapshot [ Huang et al., 2026 ] . The snapshot contains 22,586 source trials, including 22,417 usable trials, spanning 18 models, 50 model–effort configurations, and 113 tasks. The 169 excluded trials are not silently filled or imputed. Luna low , Luna medium , and Luna high refer to gpt-5.6-luna at low, medium, and high reasoning, respectively. Selection and published-data feasibility examples use the snapshot trial prices as recorded. Only the target-harness and case-study costs from Section 4.2 through Section 5.3 , including the published estimates compared there, are priced using the August 16, 2026 schedule. Under that schedule, the $0.65 published Luna low budget corresponds to $0.13. Let t t index tasks, c c index model–effort configurations, and j ∈ { 1 , … , 4 } j\in{1,\ldots,4} index repeated published runs. Let x t ​ c ​ j x_{tcj} be the F2P result for run j j , and let Y c Y_{c} be the fixed full DeepSWE score for configuration c c , calculated as the mean published score across included trials. C t C_{t} denotes the set of configurations with a complete four-run cell for task t t . The number of complete configurations can differ by task. Incomplete configuration–task cells are excluded from the corresponding task’s ranking and calibration rather than reconstructed. F2P was chosen on evidence. Under the same 10,000-resample procedure, F2P produced 22 tasks with a fifth-percentile correlation of at least 0.50. All verifier tests combined produced 15, binary pass/fail produced 11, and P2P alone produced zero. F2P is used to measure implementation progress; P2P remains a separate regression check. The choice does not make F2P a literal measure of how much of the task was completed. The 0.50 threshold is a descriptive reporting cut, not a selection rule; allocation uses the complete task ranking. For every task and resample, DeltaSelect independently draws a single trial from the four complete trials for each available configuration and correlates the resulting one-run task vector with the fixed full-score vector: The Pearson correlation is recomputed for 10,000 deterministic resamples. The choice of 10,000 is not statistically special or tuned to the result. It is a practical large sample that makes Monte Carlo noise small while adding negligible computation to the task-selection stage. Each pseudorandom task stream is seeded from SHA-256 of the pinned snapshot checksum, a null byte, and the task identifier, using the first little-endian 32-bit integer. Constant sampled vectors receive correlation zero. The ranking statistic is the fifth percentile of the task-level resampled correlation distribution: Tasks are ordered from highest to lowest q t q_{t} , with ties broken by median correlation and then task identifier. The fifth percentile is conservative in a specific sense: a task ranks highly only if most one-run combinations preserve a strong relationship with full-score variation across the observed configurations. The statistic is not a 90% confidence bound on future models, tasks, or harnesses. The statistic is conditional on the four retained trials and the finite configuration matrix. The selector does not choose tasks that best separate one favorable historical pair. It asks whether tasks continue to track configuration differences after each four-run cell is reduced to one sampled outcome. Task fractions are not directly comparable. A 0.50 F2P result on one repository can represent a different level of observed system performance than 0.50 on a second repository. DeltaSelect fits a separate ordinary least-squares mapping from each task’s four-run mean to the common published full-score scale: Here x ¯ t ​ c \bar{x}{tc} is the mean of the four published F2P trials. The fitted line is later applied to the noisier one-run observation in a low-cost evaluation. Ordinary least squares was chosen for transparency and auditability, not because the relationship is known to be linear. This mapping is also the normalization step. Raw F2P fractions come from different verifiers and do not carry the same meaning across tasks. After calibration, every task estimates the same full-score quantity. The normalized weights sum to one, so the aggregate remains on that common scale rather than growing with the number of selected tasks. A task’s number of verifier checks affects the granularity of its F2P fraction, not the scale of the aggregate. Calibration does not remove dependence between related tasks. The example calibration curve (Figure 3 ) shows possible structure near high F2P values; nonlinear alternatives remain unmeasured future work. Residual error determines task influence in the final selected set S S : Tasks with undefined or nonpositive residual variance are excluded to avoid undefined infinite weights. The remaining positive inverse-error weights are normalized over the final selected set. A task with smaller linear-calibration residuals receives more influence. The weights resemble precision weighting, but the interpretation must stay limited. Task estimates are correlated, and each residual variance is estimated from the same finite matrix. The weights also omit the one-run sampling variance of x t x{t} , a source not captured by the calibration residual. The weights do not form an optimal inverse-variance meta-analysis. For an evaluated arm, the calibrated score is where x t x_{t} is the arm’s observed F2P fraction, averaged over m m repetitions when m > 1 m>1 . The calibrated score expresses a task’s F2P result on the published DeepSWE full-score scale. This puts otherwise incomparable task outcomes on one common scale so they can be combined into a single score. It is not an official DeepSWE score or a literal percentage of requirements completed. DeltaSelect does not cap the estimate at 0 or 100%. A linear fit can produce values outside that range, especially when a new result falls beyond the published observations. Capping those values would hide the magnitude of changes near the boundary and alter differences between the baseline and candidate. Values outside 0–100% should therefore be reported as model estimates and interpreted with raw F2P, P2P, binary outcomes, and task failures. Two-panel score-construction workflow showing per-task ordinary least-squares mappings and normalized inverse residual-variance weighting. Let e e denote the execution configuration used for pricing, including model and reasoning effort; let c ¯ t , e \bar{c}_{t,e} be the observed mean price of task t t under that configuration; let B B be the per-arm dollar budget; and let m m be the uniform repetition count. DeltaSelect scans the fixed reliability order once: DeltaSelect includes every task fitting the remaining budget. After an unaffordable task, the scan continues and allows a cheaper lower-ranked task to use the remaining amount. Figure 2 shows the resulting noncontiguous selection for the $0.65 published Luna low budget: ranks 1–7 fit, ranks 8–24 do not fit the remaining $0.056, and rank 25 is the next affordable task. Neither price nor budget changes the task ranking. Reliability is calculated before either is considered. Price, budget, model, reasoning effort, and repetition count affect only which tasks from that fixed ranking are selected; they do not change the order or task-specific score mapping. Weights are renormalized over the selected set. The rule is deterministic and easy to inspect, but it does not guarantee the best possible combination of tasks for the budget. The scan can skip a combination with better joint information, choose redundant tasks, or spend less than the budget. The limitations are accepted in exchange for a simple reproducible specification; matched-cost comparisons with random, cheapest-first, and median-correlation selectors have not been run. The published price schedule is a planning input, not a target-harness-cost forecast. A mini-swe-agent task price cannot predict the number of delegated sessions or continuations created by a target-harness multi-agent workflow. The target-harness baseline must be run before an operating budget is trusted. The case study in Section 5.1 shows how to plan around this gap: run the intended baseline, check headroom, and compare the intended reasoning level with the next level up before investing in workflow optimization. For displayed published baseline or comparison points, DeltaSelect resamples task outcomes and prices 1,000 times, recomputes the selected-set score and total, and reports the 5th-to-95th-percentile range: This range answers a limited question: if the same published configuration were run again on the same selected tasks, how much might its reported score or price move? It does not predict results for new model families, harnesses, environments, or provider behavior absent from the snapshot. For a one-run A/B result, DeltaSelect also estimates how far two repeats of the same unchanged configuration could differ. For each selected task, two published outcomes from the same configuration are independently sampled with replacement and transformed into a calibrated difference: The observed baseline-minus-candidate difference is reported together with this range. If the difference falls inside the range, it is consistent with the repeat-run variation in the published data. If it falls outside, it exceeds the central 90% of that variation. This is not a significance test for a new harness because the candidate was run only once and may have different variance. When both versions have repeated published runs, the test in the next subsection provides the formal comparison. When both arms have repeated published trials, the calibrated weighted difference and propagated standard error are The two-sided comparison uses the Welch–Satterthwaite approximation for the effective degrees of freedom [ Welch, 1947 , Satterthwaite, 1946 ] . The calculation assumes separately executed task cells are independent and treats the selected calibration coefficients and weights as fixed. Related tasks may share errors, which would violate that independence assumption. The resulting p -value measures how strongly the two published arms differ in this fixed dataset under those assumptions. The comparison is retrospective because it was chosen after the data were available rather than preregistered. The planner exports the ranked tasks, selected repetitions, calibration coefficients, weights, price assumptions, source checksum, and random seeds as a single experiment specification. The same specification is used for baseline and candidate. The harness records task checksums, client and model versions, reasoning effort, environment identity, raw F2P and P2P results, binary resolution, build or verifier failures, calibrated score, and cost telemetry. In the Agent Layer case study (Section 4.2 ), every revision used the same tasks, calibration, weights, and repetition count. This kept the benchmark itself from changing while the workflow changed. A change to the tasks or score calculation would define a different benchmark, so its results would not be combined with the original series. The frozen specification also defines how cost is counted. It records request-level token and billing data for model calls caused by the experiment, including failed, retried, delegated, continued, and abandoned work. A concrete published-data example tests whether DeltaSelect can recover a difference that the full DeepSWE matrix already shows clearly. Claude Opus 5 medium scores 20.3 points higher than Gemini 3.6 Flash high on the full binary score. The pair was chosen because the full benchmark shows a clear difference; if a small task set cannot distinguish these configurations, it is not ready for smaller development decisions. The released DeepSWE v1.1 trial artifact [ Huang et al., 2026 ] contains 447 scored Claude trials and 451 scored Gemini trials, or 898 total. It also records five additional Claude trials that were excluded from the published score because the agent or verifier errored. One Gemini task contains only three trials, and DeepSWE does not explain why the fourth trial is missing. A complete 113-task, four-run protocol for both configurations would contain 904 trials. The 898 scored trials cost $3,063.63. The analysis begins with the highest-ranked task and extends the set by one task at each step, following the fixed ranking. The test from the preceding subsection uses the published four-run mean difference as the effect and the published one-run task variances as the expected noise. The result is a projection of a one-run experiment, not a new experiment that was run. The first prefix with projected p < 0.05 p<0.05 contains nine tasks and costs $113.08 across both configurations ( p = 0.0473 p=0.0473 ). The nine-task design costs 27.1 times less than the recorded cost of the full protocol. All 109 tasks with complete cells for both configurations cost $740.58 and produce projected p = 0.00378 p=0.00378 . Figure 4 compares the cost and projected evidence.