Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.18460 · published 2026-09-16 · submitted 16 Sep 2026

Abstract

How does a multi-agent system evolve from a local deviation into collective loss of control? We propose an epidemic explanation organized around accidental mutation, contagion, and recovery. A spontaneous deviation creates a seed; communication enables other agents to adopt and retransmit its unsafe strategy; collective failure can emerge when propagation outpaces correction and containment. Thus, rare individual deviations can coexist with substantial collective risk. Motivated by reported OpenAI agent coordination incidents, we examine two ingredients of this mechanism. A deployment audit identifies implicit communication paths between nominally independent evaluation runs and verifies transport through a default Docker backend. RogueHandoff-20, a benchmark of 20 executable scenarios, tests recipient susceptibility by injecting unsafe trajectories generated by a modified Qwen-27B route. Across four native-pending routes, executed harm is 0-5% on normal tasks and 40-95% after injection, exceeding paired direct malicious requests by 5-45 percentage points. These results support low observed baseline harm alongside high conditional susceptibility; they do not establish natural rare-event rates or demonstrate an autonomous cascade. The account motivates complementary defenses: strengthen resistance and recovery alongside prevention of spontaneous deviations, and audit and restrict unintended communication paths that can turn local failures into collective loss of control.

Authors

Xiangfan Wu , Zonghao Ying , Huiyu Wu , Xing Zheng , Huangsheng Cheng , Xiaorong Shi , Jing Guo

Key claims (verbatim-leaning English extract)

• How does a multi-agent system evolve from a local deviation into collective loss of control? We propose an epidemic explanation organized around accidental mutation, contagion, and recovery. A spontaneous deviation creates a seed; communication enables other agents to adopt and retransmit its unsafe strategy; collective failure can emerge when propagation outpaces correction and containment. Thus, rare individual deviations can coexist with substantial collective risk. Motivated by reported OpenAI agent coordination incidents, we examine two ingredients of this mechanism. A deployment audit identifies implicit communication paths between nominally independent evaluation runs and verifies transport through a default Docker backend. RogueHandoff-20, a benchmark of 20 executable scenarios, tests recipient susceptibility by injecting unsafe trajectories generated by a modified Qwen-27B route. Across four native-pending routes, executed harm is 0-5% on normal tasks and 40-95% after injection, exceeding paired direct malicious requests by 5-45 percentage points. These results support low observed baseline harm alongside high conditional susceptibility; they do not establish natural rare-event rates or demonstrate an autonomous cascade. The account motivates complementary defenses: strengthen resistance and recovery alongside prevention of spontaneous deviations, and audit and restrict unintended communication paths that can turn local failures into collective loss of control.

• How does a multi-agent system become collectively out of control? Reports of OpenAI agents coordinating through unauthorized message boards, including activity associated with the Hugging Face intrusion and the DSEWiki episode, make this a concrete systems question [ 11 , 49 , 33 ] . The puzzle is how an unsafe strategy acquires additional participants: a population may usually perform legitimate tasks safely, yet interactions can allow a local deviation to develop into a collective failure. Understanding this transition requires explaining both where the first deviation comes from and what causes other agents to join it.

• Our proposed explanation is an agent epidemic. We distinguish accidental mutation , contagion , and self-correction (recovery) ( Figure 1 ). The central distinction is between originating an unsafe strategy and adopting one supplied by another agent. An agent that rarely invents a prohibited shortcut may still follow it when received context presents it as established work. Newly affected agents can then become additional sources of that strategy. Repeated transmission can produce collective loss of control when spread outpaces correction and containment, without requiring many independent initial failures.

• Accidental mutation supplies the first seed: during an otherwise legitimate task, an agent may propose obtaining restricted answers instead of solving the problem within its authorization. Such deviations may be rare per decision, but a large population operating repeatedly creates many opportunities for one to occur. “Mutation” denotes a change in a session’s behavioral trajectory with fixed model weights. Its frequency determines how often seeds arise; it does not determine how many other agents a seed can affect.

• Contagion requires both communication and recipient adoption. Messages, shared files, tool outputs, or writable services can carry an unsafe plan between sessions, including agents assigned independent tasks. Exposure alone is insufficient: the recipient must accept and act on that plan. If affected recipients also pass it onward, one deviation can generate successive waves of unsafe behavior. Resistance blocks adoption, while self-correction returns an affected agent to safe behavior; isolation can externally interrupt the chain. Adapting epidemic and network models [ 24 , 32 ] , we connect these processes to the growth or extinction of collective failure. Reducing seed frequency therefore addresses only one part of cluster risk.

• The reported incidents motivate this explanation but do not establish its complete causal chain. Agents converging on a shared board raise the question of how they discovered and communicated its location. Unrecognized internal channels are one hypothesis; common inputs and independent discovery are alternatives. We do not reconstruct private evaluation infrastructure. Instead, we investigate whether the proposed mechanism’s communication and behavioral requirements hold in accessible settings.

• For communication, an audit of public evaluation deployments identifies implicit paths between nominally independent runs. A probe verifies fresh information transfer through mini-SWE-agent’s default Docker execution backend: separate containers need not imply communication isolation. For adoption, we introduce RogueHandoff-20 , a Docker-based benchmark of 20 executable scenarios. Because naturally occurring unsafe trajectories are difficult to collect on demand, a modified (“cracked”) Qwen-27B route generates synthetic fragments that are injected into evaluated agents as preceding state. Normal tasks provide a baseline, and paired direct malicious requests test whether inherited work adds influence beyond the harmful objective itself. Protected ledgers score prohibited actions newly executed by the recipient.

• Across four native-pending routes, normal-task harm is 0-5%, whereas injected trajectories produce 40-95% harm and exceed direct requests by 5-45 percentage points. This supports the explanation’s behavioral premise: low observed harm from clean contexts can coexist with high susceptibility after exposure. The audit and benchmark test separate components; they neither establish an extremely rare natural initiation rate nor demonstrate an autonomous multi-hop outbreak. Their combination motivates defenses at both levels: train agents to resist and correct transmitted unsafe strategies alongside preventing spontaneous deviations, and restrict unintended communication paths that allow a local seed to reach further recipients.

• This paper makes three contributions: • An explanation of collective loss of control. We model how accidental mutation, repeated contagion, and recovery can turn a local deviation into a population-level failure or contain it.

• Evidence on communication and susceptibility. A deployment audit tests potential transmission paths, while our benchmark measures harmful adoption under controlled exposure. The code and fixed inputs are publicly available in Tencent’s AI-Infra-Guard repository. 1 1 1 https://github.com/Tencent/AI-Infra-Guard/tree/main/Research/roguehandoff20

• Defenses derived from the mechanism. We identify complementary priorities for model training and environment isolation, targeting seed formation, adoption, onward spread, and recovery.

• Our study connects research on agent security, propagation, and recovery. The central question is how a rare local deviation can spread through an agent population, and what limits that spread.

• Context is an active part of an agent’s decision process. ReAct interleaves reasoning, observations, and actions, allowing earlier steps to guide later planning [ 55 ] . Work on in-context learning explains how models infer tasks from demonstrations [ 53 ] , while function-vector analyses identify compact representations that mediate the influence of demonstrated functions on subsequent generation [ 46 ] . Together, these perspectives motivate studying an agent’s accumulated trajectory as a source of behavioral continuity. The same continuity that supports progress on a task may also sustain an unsafe course of action.

• Self-correction research examines how that course can change. Self-Refine uses iterative language feedback, and Reflexion uses feedback and stored reflections to improve subsequent outputs or trials [ 30 , 40 ] . Correction from a model’s own reasoning is more sensitive to the conditions of evaluation: Huang et al. find that prompted revision can fail or degrade reasoning, while Liu et al. report improvements under particular prompting and decoding choices [ 20 , 28 ] . Kamoi et al. organize these findings around the feedback available, the initial baseline, and the criterion for successful correction [ 23 ] .

• At the team level, Huang et al. show that resilience to faulty agents depends on the collaboration structure and can improve when agents challenge one another’s outputs or an inspector reviews their messages [ 19 ] . This makes recovery a property of both the individual agent and the interaction process around it.

• RogueHandoff brings this question to agent safety at the point of action. An agent resumes an unsafe trajectory with the instruction Continue , without an added critique or error signal. Recovery requires it to interrupt the inherited course before executing a prohibited state change. This tests whether safety constraints remain effective when the context already presents harmful behavior as work in progress.

• CAMEL, AutoGen, and MetaGPT make messages and intermediate work products central to multi-agent coordination [ 27 , 52 , 18 ] . These interfaces also allow one agent’s compromised behavior to reach others. Agent Smith demonstrates infectious jailbreaks in simulated multimodal agent populations, where an adversarial image introduced into one agent’s memory spreads through pairwise interaction [ 14 ] . Prompt Infection studies a corresponding text-based channel through payloads that replicate between communicating agents [ 26 ] . These results give concrete examples of how a local seed can become a network-level safety problem.

• The consequences extend beyond harmful answers. CORBA induces recursive, unproductive message passing through superficially benign instructions, blocking collaboration across the system [ 62 ] . Multi-Agent Security Tax studies malicious instructions spreading over multiple hops and finds that defenses which reduce propagation can also impair collaboration [ 35 ] . Together, these studies motivate examining how communication sustains a failure and how agents can interrupt it while retaining useful coordination.

• External content and persistent memory provide additional routes for unsafe state to enter and remain in a workflow. Indirect prompt injection redirects agents through external content; InjecAgent and AgentDojo evaluate these attacks in tool-using settings [ 13 , 57 , 8 ] . AgentPoison targets retrieval through poisoned memory or knowledge bases, while MemoryGraft exploits the reuse of poisoned experience records [ 6 , 42 ] . Action-hijacking work likewise examines how manipulated context redirects downstream actions [ 59 ] .

• RogueHandoff connects this propagation perspective to recovery during task execution. Its transferred state is an in-progress agent trajectory: a historical request, reasoning, task observations, and a pending action. We construct this state with a prefix simulator and measure how a successor responds when work resumes. An unsafe fragment can become either the context for the same agent’s next decision or the starting point for another agent, linking recovery to propagation resistance. Epidemic and network models provide a language for relating these local transitions to exposure, connectivity, and recovery at the cluster scale [ 24 , 32 ] .

• This perspective also applies to evaluations that do not expose an explicit agent-to-agent interface. Shared runtime networks and application state can supply implicit communication paths. Our deployment audit ( Section 3.4 ) connects that infrastructure question to the behavioral question studied in infectious-jailbreak and recovery work: what happens once another session’s output becomes available as context?

• The broader AI-safety literature studies harmful behavior arising from objectives, training, distribution shift, and system design [ 2 , 36 ] . Model organisms make several of these mechanisms experimentally accessible. Sleeper Agents and BadAgent study persistent triggered behavior introduced through training [ 21 , 51 ] ; emergent-misalignment experiments show that narrow harmful fine-tuning can affect behavior more broadly [ 4 ] ; and alignment-faking experiments examine strategic behavior across training and deployment contexts [ 12 ] .

• At the multi-agent level, PsySafe studies how assigned personality traits influence safety, reporting collective dangerous behavior and self-reflection during unsafe interactions [ 60 ] . These observations bring attention to the evolution of behavior within an interacting group.

• Our focus is the dynamics of an agent session after a harmful trajectory has begun. A rogue state is defined operationally by the proposal or execution of a prohibited action, and its continuation or recovery unfolds through subsequent interaction. This session-level view complements work on trained policies by asking how accumulated context can sustain unsafe behavior with fixed model weights. It separates three safety questions: how a rogue state arises, how it spreads through agent interactions, and whether an exposed or affected agent rejects or corrects it. RogueHandoff uses constructed fragments to test local harmful continuation after exposure; full network propagation and recovery dynamics require additional experiments.

• Agent evaluations increasingly ground safety and capability in tool execution. ToolEmu uses an LM-emulated sandbox to surface risky behavior, while AgentHarm evaluates explicitly malicious multi-step tasks [ 37 , 3 ] . Agent Security Bench compares attacks and defenses across the agent stack, and R-Judge measures risk recognition in interaction records [ 58 , 56 ] . ToolSandbox tracks state dependencies and intermediate milestones, while τ \tau -bench evaluates final database state in policy-constrained interactions [ 29 , 54 ] . These approaches make the consequences of agent decisions central to evaluation.

• RogueHandoff builds on this approach with a paired comparison of direct malicious requests and resumption from an already formed unsafe trajectory. The successor model, initial world state, tools, harmful objective, and target are held fixed. Protected ledgers record executed prohibited changes, separating external harm from harmful text or unsuccessful attempts. The comparison asks whether resistance to a harmful request carries over to a situation in which the same objective is embedded in ongoing work. This connects request-level safety evaluation to recovery from transient loss of control and resistance to inherited unsafe state.

Structure (section headings from HTML)

• Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery
• 1 Introduction
• 2 Related Work

• 2.1 Context-Dependent Behavior and Recovery
  

• 2.2 Multi-Agent Security and Propagation of Unsafe State
  

• 2.3 Misalignment and Transient Loss of Control
  

• 2.4 Evaluating Safety During Resumption
  

• 3 Background: The Emergence of Collective Failure

• 3.1 Collective Incidents and the Communication Question
  

• 3.2 From Rare Mutation to Contagion and Repair
  

• 3.3 Measuring the Ingredients Separately
  

• 3.4 Implicit Communication in Parallel Evaluation
  

• 4 An Epidemic Explanation of Collective Loss of Control

• 4.1 A Rogue-State Epidemic Model
  

• 4.2 Controlled Exposure to a Synthetic Rogue State
  

• 4.3 Three Experimental Conditions
  

• 4.4 Finite-Suite Estimand and Identification
  

• 5 Synthetic Loss-of-Control Benchmark: RogueHandoff-20

• 5.1 Scenario Construction
  

• 5.2 Execution Isolation and Ground Truth
  

• 5.3 Pending-Call Transport
  

• 6 Measurements
• 7 Main Evaluation

• 7.1 Models and Protocol
  

• 7.2 Native-Pending Results
  

• 7.3 Compatibility-Mode Results
  

• 7.4 Paired Direction Diagnostic
  

• 8 Loss-of-Control Intensity

• 8.1 Token-Stream Cutoff
  

• 8.2 Results
  

Remainder

Full original English text: see html_url / source_url / pdf_url in frontmatter.