second brain
source
← 首页

external-source

Chronicle: Cut-Point Replay for Regression Testing of LLM Agents

Chronicle: Cut-Point Replay for Regression Testing of LLM Agents

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.20625 · published 2026-09-17 · submitted 2026-09-17

Abstract

Large language model responses are non-deterministic, so failures in LLM agents are hard to reproduce: a failure depends on inference that is not bitwise reproducible, on tools that read changing state, and on a multi-step trajectory that a re-run rarely repeats. Record-and-replay makes a run reproducible, but existing agent tooling records runs only to trace or score them, not to test a code change against them. We present Chronicle, which records an agent run at its non-deterministic boundaries as immutable envelopes and replays it from the record. Its central operation, cut-point replay, serves a chosen subset of boundaries from the record and executes the complementary subset live with new code, turning a recorded incident into a regression test that runs in continuous integration. On a benchmark of 6 recorded failures with simulated model boundaries, recording adds 23 μs per crossing (0.008% of an assumed 300 ms model call), full replay issues zero model calls and is bit-stable across 20 repetitions, and cut-point tests fail on faulty code and pass on guarded and benign changes for all 6 incidents. In a mutation study of the guarded tools, cut-point tests catch every mutant that lets the recorded unsafe action through, while a baseline that stubs every boundary, using the same assertion, catches none. Chronicle and the benchmark are publicly available at https://github.com/theagentplane/chronicle.

Authors

Tisha Chawla, Susheem Koul

Key claims (verbatim-leaning English extract)

Structure (section headings from HTML)

Body excerpts (original English)

Chronicle: Cut-Point Replay for Regression Testing of LLM Agents Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Related Work Record and replay for agents. Testing of agents. Failure attribution. 3 Chronicle 3.1 Record 3.2 Replay 3.3 Test 4 The Incident Benchmark Why cut-point, not per-boundary mocks. Task and metrics. 5 Experiments Setup. Recording overhead. Determinism and replay cost. Fault detection. Mutation study and full-stub baseline. Workflow. References A Additional Replay Diagrams License: CC BY 4.0 arXiv:2609.20625v1 [cs.CL] 17 Sep 2026 Chronicle: Cut-Point Replay for Regression Testing of LLM Agents Tisha Chawla † † thanks: Equal contribution. Susheem Koul 1 1 footnotemark: 1 Affiliation: Microsoft Email: tisha.chawla2020@vitalum.ac.in Email: f2015347p@alumni.bits-pilani.ac.in Abstract Large language model responses are non-deterministic, so failures in LLM agents are hard to reproduce: a failure depends on inference that is not bitwise reproducible ( Atıl et al., 2025 ) , on tools that read changing state, and on a multi-step trajectory that a re-run rarely repeats. Record-and-replay makes a run reproducible, but existing agent tooling records runs only to trace or score them, not to test a code change against them. We present Chronicle, which records an agent run at its non-deterministic boundaries as immutable envelopes and replays it from the record. Its central operation, cut-point replay , serves a chosen subset of boundaries from the record and executes the complementary subset live with new code, turning a recorded incident into a regression test that runs in continuous integration. On a benchmark of 6 recorded failures with simulated model boundaries, recording adds 23 µs per crossing (0.008% of an assumed 300 ms model call), full replay issues zero model calls and is bit-stable across 20 repetitions, and cut-point tests fail on faulty code and pass on guarded and benign changes for all 6 incidents. In a mutation study of the guarded tools, cut-point tests catch every mutant that lets the recorded unsafe action through, while a baseline that stubs every boundary, using the same assertion, catches none. Chronicle and the benchmark are publicly available at https://github.com/theagentplane/chronicle . 1 Introduction Large language model (LLM) agents take consequential actions through tool calls, such as issuing refunds, submitting trades, or deleting files ( Schick et al., 2023 ; Yao et al., 2023 ) . When such an action is wrong, because the surrounding code misses an edge case or the task is underspecified, correcting it requires first reproducing the failure. This is often the hardest step: a failure is seldom a single wrong output but a trajectory of model calls, tool calls, and routing decisions ( Cemri et al., 2025 ) . Re-executing the agent rarely reproduces that trajectory. LLM inference is not bitwise reproducible even at temperature zero ( Atıl et al., 2025 ) ; tools read external state that has since changed ( Yao et al., 2024 ) ; and retries and routing change how many times a step runs ( Cemri et al., 2025 ) . The result is flakiness : identical code passes or fails depending on nondeterministic conditions and execution order ( Lam et al., 2020 ; Hashemi et al., 2025 ) , so re-execution alone cannot turn an incident into a repeatable test. Existing agent infrastructure observes runs but does not make them testable: tracing records what happened ( Arize AI, 2026 ) and evaluation frameworks score whether an output is acceptable ( promptfoo contributors, 2026 ; Zheng et al., 2023 ) , but neither lets a developer change one component of a recorded run and check whether the change fixes the failure while everything else is held fixed. We present Chronicle, which supplies exactly that operation. This paper contributes (i) cut-point replay , which runs new code at a chosen subset of boundaries while serving the rest from the record, producing a committed regression test from the incident, with each stubbed boundary guarded by a per-name call-count check; and (ii) an implementation and a benchmark of 6 recorded failures on which full replay makes no model call and reproduces each run identically across 20 repetitions, while cut-point tests flag every unguarded incident, accept its guarded fix and benign edits, and kill mutants of the fix that a stub-every-boundary baseline cannot. 2 Related Work Chronicle relates to three lines of work and differs from each in the purpose for which replay is used. Record and replay for agents. Deterministic record-replay is long established for debugging general programs, where an entire execution is captured and faithfully reproduced ( O’Callahan et al., 2017 ) . Chronicle instead replays at semantic boundaries and runs new code at chosen ones, so replay tests a fix rather than reproduces a run. Prior systems record runs to reuse successful agent behavior ( Feng et al., 2025 ) or to resume a durable workflow after a failure ( Temporal Technologies, 2026 ) ; Temporal’s replay tests also re-run changed workflow code against a recorded event history, and Chronicle adapts this selective-replay idea to agent boundaries, with indexed crossings and a freely chosen live set. Graph frameworks checkpoint state to permit re-entry at a previously executed node ( LangChain, 2026 ) . These systems act during a run, to resume or steer it in flight; a workflow resumed after a human-in-the-loop pause continues forward and does not re-issue the model calls it already completed. Chronicle instead operates after a run has finished: it replays a recorded trace, serving stubbed crossings from the record and running the live subset with new code, so a candidate fix is evaluated against the past incident rather than by steering a live run. Testing of agents. Recent work adapts software-testing techniques such as traces, mocks, and assertions to agents ( Kohl et al., 2025 ) . A documented limitation of heavy mocking is that mock-heavy tests can be less effective at validating real interactions ( Hora and Robbes, 2026 ) . Chronicle replaces hand-written mocks with envelopes drawn from a recorded run and lets any subset of boundaries run live, so a tool gate or router change is exercised against the recorded trajectory rather than against invented stubs. Failure attribution. A related line of work intervenes in a recorded run (rewinding, editing, or otherwise modifying a crossing) to attribute or localize a failure ( Shah, 2026 ; Lin et al., 2026 ; Ma et al., 2025 ) . Those methods characterize the cause of a past failure by observing a modified crossing, whereas Chronicle executes new code at the crossing and records the outcome as a regression test. Other tools detect and diagnose faults from traces without cut-point execution of new agent code ( Balusu, 2026 ; Ou et al., 2025 ; Deshpande et al., 2025 ; Kang et al., 2026 ) , and LLM-based judges assess semantic quality subject to documented limitations in agreement and bias ( Norman et al., 2026 ) ; Chronicle ships an advisory LLM-as-judge for non-structural properties but does not evaluate judge reliability here. The agents considered in this work employ standard reason-and-act loops with tool use ( Yao et al., 2023 ; Schick et al., 2023 ) . In contrast to hand-authored agent benchmarks and evaluation environments ( Yao et al., 2024 ; Debenedetti et al., 2024 ; Zhou et al., 2024 ) , the scenarios in our benchmark are derived from recorded failures rather than constructed to evaluate capability or attack surface. 3 Chronicle Chronicle records a run, replays it from the record, and tests the replay with assertions. Its unit is the boundary : a point where the agent calls the model, calls a tool, or makes a routing decision, which is exactly where a rerun can diverge. Chronicle adapts selective replay to agents: any subset of boundaries can run live while the rest are served from a real recorded run rather than from hand-written mocks, so a recorded incident becomes a deterministic test of a chosen change. 3.1 Record Developers mark boundaries with a one-line annotation (Figure 1 ); a model client or a set of LangGraph nodes can be instrumented with a single call. Each execution of a boundary, a crossing , is saved as an immutable envelope holding its input, its output, and the metadata needed to detect drift, such as the model version and sampling parameters. Crossings are addressed by boundary name and occurrence, so a boundary crossed three times in a loop yields agent[1] , agent[2] , and agent[3] (Figure 5 ). Recording is transparent: it changes no return value or exception, redacts secrets and volatile fields before storage, and emits standard OpenTelemetry spans. An envelope stores a boundary’s input and output, not the work inside it, so replaying it is faithful as long as its output depends only on that recorded input; a boundary that reads hidden state, such as a clock or a database, is the exception. 3.2 Replay In full replay , every boundary returns its recorded output, so the run is reproduced exactly with no model call (Figure 5 ). In cut-point replay , a chosen subset of crossings run live with new code while the rest are served from the record (Figure 2 ); the live subset is arbitrary (e.g., run a tool gate live while stubbing the model; Figure 5 in Appendix A ). Full replay fixes every output and so tests the deterministic glue code between boundaries against real recorded inputs; cut-point replay tests a change together with its consequences, reproducing the stubbed lead-up without re-running it and executing the live subset forward. Chronicle serves the k k th crossing of a boundary its k k th recorded envelope, so if a stubbed boundary is crossed more or fewer times than recorded (an extra loop, a dropped retry) lookup fails and replay raises. This per-name count check does not detect a reordering that preserves each name’s count. An order-sensitive digest over the stubbed crossings would close this gap cheaply; it is not part of the release evaluated here. The guarantee is conditional on coverage: an unmarked non-deterministic call runs live and escapes the check. 3.3 Test A structural assertion checks what the agent did (which tool, which arguments, whether a guarded action was refused); on full replay it is deterministic and needs no model call, so a recorded incident and one assertion become a CI regression test at no cost, and on cut-point replay the same assertion checks the fix on the live subset. For non-structural properties (faithfulness, safety) Chronicle ships an advisory LLM-as-judge ( Zheng et al., 2023 ) ; its reliability is not evaluated here ( Norman et al., 2026 ) . @boundary("place_order", kind="tool") def place_order(symbol, qty): ... plan = (ReplayPlan() .stub("agent", 1) # from record .live("place_order", 1) # cut-point .live("agent", 2)) assert session.captured_result( "place_order", 1)["blocked"] Figure 1: Recording annotation and a cut-point plan: the same @boundary records in production and serves its recorded output in replay; any subset of crossings may run live. agent[1] stubbed place_order[1] live agent[2] live from record cut-point assert Figure 2: Cut-point replay (trade-notional fixture): agent[1] stubbed; place_order[1] and agent[2] live; stubbed crossings return their recorded outputs. 4 The Incident Benchmark We release a benchmark of 6 recorded incidents, each a small agent (model, then tool, then model) in which an unguarded tool produces an unsafe result and a guarded version corrects it: a refund sized to an order identifier, a currency mismatch, a notional-versus-quantity trade error, an over-broad email audience, a payout-account injection, and a production file deletion. Each ships with the recorded trace, the unguarded and guarded tools, and a cut-point test. Each incident instantiates a documented failure mode from the multi-agent failure taxonomy of Cemri et al. (2025) , chiefly its task-verification category (missing or weak checks before acting), rather than an arbitrary bug (Table 1 ). All 6 agents use deterministic simulated boundaries, so the harness makes no provider API calls and runs in continuous integration; fixtures are versioned. Incident Unsafe result Guard Refund refund sized to order id flat cap Invoice wrong currency sent currency check Trade notional read as shares notional cap Email audience too broad recipient allowlist Payout account substituted account check Deletion production file removed delete gate Table 1: The 6 recorded incidents. Each unguarded tool produces the unsafe result; the guarded version applies the guard, and the cut-point test asserts that the guard fired. Why cut-point, not per-boundary mocks. A baseline that stubs every boundary with its recorded return never re-enters the tool, so its verdict cannot depend on the tool’s code: the mock returns the recorded unsafe result whether or not a gate exists. On all 6 incidents it fails the unguarded code, the guarded fix, and the benign edit alike (Table 2 ). Chronicle’s plan stubs the first model crossing and runs the tool live, so the gated tool executes against the recorded arguments. On the trade-notional incident the unguarded tool sells 1,000 shares ( ≈ $ 190 \approx$190 k) for a ≈ $ 1 \approx$1 k request; the gated tool blocks it; the cut-point test fails on the unguarded code and passes on the gated and benign rewordings. The same pattern runs for all 6 incidents. Task and metrics. A method receives a recorded trace and a code version and returns fail or pass. We report on the curated suite: (i) fault-detection rate (unguarded fail, gated and benign pass); (ii) determinism of full replay (0 divergences over 20 repetitions); (iii) provider calls and dollar cost under full replay; (iv) recording overhead per crossing relative to a typical model call; (v) wall-clock time for one full-stub and one cut-point pass over the suite; (vi) boundary-annotation coverage on the released agents (18 sites, three per incident, all annotated explicitly); and (vii) the share of mutants of each guarded tool that a test kills. Evaluation is scoped to these 6 incidents; we do not claim reproduction rates for traces outside the suite. 5 Experiments Setup. A single harness runs each item under recording and replay and, for each code version, the cut-point test. Determinism is measured by re-running the full-stub replay suite 20 times. Separately, we time 50 calls per arm to Qwen3.5 4B, an open-weight model served locally (4-bit, Ollama on a laptop CPU), through a model boundary with recording on versus off (interleaved pairs in random order) to place instrumentation cost against real inference latency and its variance. Recording overhead. In-memory recording adds a median 23 µs per crossing, or 0.008% of an assumed 300 ms model call, and the store grows by at most 1.44 KB per crossing (Table 2 ). Against real Qwen3.5 4B calls, mean latency is 3,136 ms (s.d. 379 ms) with recording on and 3,045 ms (s.d. 387 ms) with it off; the difference of means is + 91 +91 ms (95% CI − 59 -59 to + 241 +241 ms), indistinguishable from zero. Chronicle’s measured cost of 23 µs per crossing is four orders of magnitude below this run-to-run variation, so recording is not a detectable source of latency. Determinism and replay cost. Full replay is bit-stable: across 20 repetitions it reproduces each recorded run identically, with 0 divergences and 0 live boundary crossings. Because the benchmark’s model boundaries are simulated, this validates the replay mechanism rather than reproduction under a nondeterministic provider (see Limitations). A cut-point test inherits this stability for its stubbed crossings, while its live crossings re-execute the annotated functions. Full replay issues 0 of the 12 agent model-boundary crossings in the suite, so the provider cost of 1,000 full-suite replays is $0.00; one full-stub pass over the suite completes in 3.5 ms and one cut-point pass (tool and final model crossing live) in 3.6 ms (Table 2 ). A suite of recorded incidents thus runs on every commit at no model cost. Fault detection. On the 6 curated incidents, cut-point tests give the expected fail / pass / pass on 6 of 6 (the unguarded code is caught; the gated fix and benign rewordings pass) and tolerate 30 unrelated rewordings that leave the safety invariant unchanged (Table 2 ). Because the incidents, guards, and assertions were written together, these outcomes validate the mechanism on curated incidents rather than measure fault detection in general. Mutation study and full-stub baseline. To test whether the assertions catch faults beyond the known missing guard, we generate all first-order mutants of each guarded tool with standard relational, logical, and constant mutation operators (192 in total); a mutant is killed when a test’s verdict on it differs from its verdict on the unmutated fix. Cut-point tests kill 51; the full-stub baseline kills none, since the tool never runs under it. Of the 141 survivors, 110 cannot be killed by any test built from these recordings: they change code the recorded input never reaches, or change the guard without changing its decision on that input (e.g., > > to ≥ \geq on a threshold the input exceeds). The other 31 change output fields the assertion deliberately ignores, such as the status label, while still blocking the action, so no surviving mutant lets the recorded unsafe action through. Among mutants that change behavior on the recorded input, cut-point tests thus kill 51 of 82 (62%). The unkillable mutants reflect a coverage limit of single-incident fixtures: each recording exercises one input. Workflow. A team records a production incident once, commits the trace and a cut-point assertion to the repository, and the test then runs on every commit with no model calls. When a boundary’s contract changes so that a stubbed name is crossed a different number of times, the count check flags the fixture for re-recording rather than passing silently, which keeps the committed test honest as the agent evolves. Cut-point Full-stub Test outcomes Fails unguarded code 6/6 6/6 Passes fix + benign edit 6/6 0/6 Rewordings tolerated 30/30 – Mutants killed 51/192 0/192 Replay Live model crossings 6/12 0/12 Divergences (20 runs) – 0 Suite pass (ms) 3.6 3.5 Recording Per crossing (µs) 23 vs. 300 ms call 0.008% Store per crossing (KB) ≤ \leq 1.44 Model call, on / off (ms) 3,136 / 3,045 Table 2: Harness results on the released fixtures. Cut-point stubs the first model crossing and runs the tool and the second model crossing live; full-stub stubs every boundary. All 18 boundary sites (three per incident) are annotated explicitly with @boundary . Limitations Chronicle does not capture streaming responses (recorded as their assembled form) or concurrent parallel tool calls, and replay does not yet re-raise an exception recorded at a stubbed boundary. An envelope captures a boundary’s interface, not its internal side effects, so a live cut-point on a destructive tool should target a sandbox. Determinism here is partly by construction: the released agents replace the model with a deterministic stub, so we do not measure reproduction on a live nondeterministic provider. The benchmark is small and self-constructed: six three-step incidents without the loops, retries, or multi-agent routing that motivate this work. Count-preserving reorderings escape the per-name check, and coverage of non-deterministic call sites is the user’s responsibility. The LLM-as-judge is implemented but not evaluated for reliability ( Norman et al., 2026 ) . Ethics Statement A recording copies prompts, agent state, and tool arguments, and may contain secrets or personal data. Chronicle applies redaction at record time, before any record is written or committed, preserving the structure that tests assert on while removing sensitive values. References Arize AI (2026) Arize AI OpenInference and Arize Phoenix: open-source LLM observability . Note: https://github.com/Arize-ai/phoenix Accessed 2026 Cited by: §1 . Atıl et al. (2025) B. Atıl, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Rajagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, and B. Baldwin Non-determinism of “deterministic” LLM system settings in hosted environments . In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) , Mumbai, India , pp. 135–148 . Cited by: §1 , Abstract . Balusu (2026) K. C. Balusu AgentTelemetry: a fault detection benchmark and toolkit for LLM agent observability . In Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware), Data and Benchmark Track , External Links: Document Cited by: §2 . Cemri et al. (2025) M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica Why do multi-agent LLM systems fail? . arXiv preprint arXiv:2503.13657 . Cited by: §1 , §4 . Debenedetti et al. (2024) E. Debenedetti, J. Zhang, M. Balunović, L. Beurer-Kellner, M. Fischer, and F. Tramèr AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents . In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , Note: ArXiv:2406.13352 Cited by: §2 . Deshpande et al. (2025) D. Deshpande, V. Gangal, H. Mehta, J. Krishnan, A. Kannappan, and R. Qian TRAIL: trace reasoning and agentic issue localization . arXiv preprint arXiv:2505.08638 . Cited by: §2 . Feng et al. (2025) E. Feng, W. Zhou, Z. Liu, L. Chen, Y. Dong, C. Zhang, Y. Zhao, D. Du, Z. Hua, Y. Xia, and H. Chen Get experience from practice: LLM agents with record & replay . arXiv preprint arXiv:2505.17716 . Cited by: §2 . Hashemi et al. (2025) N. Hashemi, A. Tahir, S. Rasheed, A. Shi, and R. Blagojevic Detecting and evaluating order-dependent flaky tests in JavaScript . In IEEE International Conference on Software Testing, Verification and Validation (ICST) , Note: ArXiv:2501.12680 Cited by: §1 . Hora and Robbes (2026) A. Hora and R. Robbes Are coding agents generating over-mocked tests? an empirical study . In IEEE/ACM International Conference on Mining Software Repositories (MSR) , Note: ArXiv:2602.00409 Cited by: §2 . Kang et al. (2026) D. H. Kang, H. Cha, and D. Weon Knowledge-based zero-replay debugging of multi-agent LLM traces . arXiv preprint arXiv:2606.14805 . Cited by: §2 . Kohl et al. (2025) J. Kohl, O. Kruse, Y. Mostafa, A. Luckow, K. Schroer, T. Riedl, R. French, D. Katz, M. P. Luitz, T. Takher, K. E. Friedl, and C. Laurent-Winter Automated structural testing of LLM-based agents: methods, framework, and case studies . In IEEE International Conference on Big Data (BigData) , Note: ArXiv:2601.18827 Cited by: §2 . Lam et al. (2020) W. Lam, S. Winter, A. Wei, T. Xie, D. Marinov, and J. Bell A large-scale longitudinal study of flaky tests . Proceedings of the ACM on Programming Languages 4 ( OOPSLA ), pp. 202:1–202:29 . Cited by: §1 . LangChain (2026) LangChain LangGraph: checkpointing and time-travel for agent graphs . Note: https://langchain-ai.github.io/langgraph/ Accessed 2026 Cited by: §2 . Lin et al. (2026) X. Lin, Y. Wang, T. S. T. Kwok, D. Guo, S. A. Nale, C. Fleming, and G. Cheng REFLECT: intervention-supported error attribution for silent failures in LLM agent traces . arXiv preprint arXiv:2606.09071 . Cited by: §2 . Ma et al. (2025) M. Ma, J. Zhang, F. Yang, Y. Kang, Q. Lin, S. Rajmohan, and D. Zhang DoVer: intervention-driven auto debugging for LLM multi-agent systems . arXiv preprint arXiv:2512.06749 . Cited by: §2 . Norman et al. (2026) J. D. Norman, M. U. Rivera, and D. A. Hughes Reliability without validity: a systematic, large-scale evaluation of LLM-as-a-judge models across agreement, consistency, and bias . arXiv preprint arXiv:2606.19544 . Cited by: §2 , §3.3 , Limitations . Ou et al. (2025) T. Ou, W. Guo, A. Gandhi, G. Neubig, and X. Yue AgentDiagnose: an open toolkit for diagnosing LLM agent trajectories . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Cited by: §2 . O’Callahan et al. (2017) R. O’Callahan, C. Jones, N. Froyd, K. Huey, A. Noll, and N. Partush Engineering record and replay for deployability . In 2017 USENIX Annual Technical Conference (USENIX ATC 17) , Santa Clara, CA , pp. 377–389 . Cited by: §2 . promptfoo contributors (2026) promptfoo contributors Promptfoo: test and evaluate LLM outputs . Note: https://github.com/promptfoo/promptfoo Accessed 2026 Cited by: §1 . Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools . In Advances in Neural Information Processing Systems (NeurIPS) , Cited by: §1 , §2 . Shah (2026) J. Shah Causal agent replay: counterfactual attribution for LLM-agent failures . arXiv preprint arXiv:2606.08275 . Cited by: §2 . Temporal Technologies (2026) Temporal Technologies Temporal: durable execution and deterministic workflow replay . Note: https://temporal.io/ Accessed 2026 Cited by: §2 . Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ \tau -bench: a benchmark for tool-agent-user interaction in real-world domains . arXiv preprint arXiv:2406.12045 . Cited by: §1 , §2 . Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models . In International Conference on Learning Representations (ICLR) , Cited by: §1 , §2 . Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-Bench and Chatbot Arena . In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , Cited by: §1 , §3.3 . Zhou et al. (2024) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig WebArena: a realistic web environment for building autonomous agents . In International Conference on Learning Representations (ICLR) , Cited by: §2 . Appendix A Additional Replay Diagrams Figures 5 and 5 show the other two replay modes, and Figure 5 illustrates boundary indexing, expanding the cut-point diagram in Figure 2 . agent[1] stub ⋅ \cdot recorded place_order[1] stub ⋅ \cdot recorded agent[2] stub ⋅ \cdot recorded Figure 3: Full replay: every boundary returns its recorded envelope (0 model calls; bit-stable over 20 reps). agent[1] stub model place_order[1] live tool agent[2] stub model live tool; models stubbed Figure 4: Any subset may be live: tool live, both model crossings stubbed, so a gate is tested without re-paying for model calls. agent[1] plan ⋅ \cdot tool_call place_order[1] tool agent[2] finalize ⋅ \cdot text 1st agent 2nd agent Figure 5: Boundary indexing: the two model crossings are addressed as agent[1] and agent[2] , by boundary name and occurrence rather than by global step number. Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" ( ) button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions . We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from