Cognitive Extensions for Dual-Process Language Agents: Memory and Self-Reflection in Interactive Environments

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.19128 · published 2026-09-16 · submitted 16 Sep 2026

Abstract

Language agents remain brittle in interactive environments, where success requires long-horizon state tracking, valid action execution, and recovery from failed steps. We extend SwiftSage, a dual-process agent that combines a fast action proposer with a slower planner, using two modular cognitive extensions: an Adaptive Memory Module (AMM) for salience-gated episodic storage and trigger-driven retrieval, and a Self-Reflection Module (SRM) for bounded execution-time validation and corrective intervention. Both modules are implemented as feature-flagged extensions over the same execution substrate, enabling controlled ablations on ScienceWorld. Across four configurations---baseline, baseline+AMM, baseline+SRM, and the full system---the full system achieves the best mean final score (64.62), success rate (43.17%), and successful-step efficiency (19.33 steps), while SRM is the strongest standalone contributor. The results suggest that execution-time control is the dominant bottleneck in this setting, while episodic memory becomes most useful once the runtime loop is stabilized.

Authors

João Meneses dos Santos , Arlindo L. Oliveira

Key claims (verbatim-leaning English extract)

• Language agents remain brittle in interactive environments, where success requires long-horizon state tracking, valid action execution, and recovery from failed steps. We extend SwiftSage, a dual-process agent that combines a fast action proposer with a slower planner, using two modular cognitive extensions: an Adaptive Memory Module (AMM) for salience-gated episodic storage and trigger-driven retrieval, and a Self-Reflection Module (SRM) for bounded execution-time validation and corrective intervention. Both modules are implemented as feature-flagged extensions over the same execution substrate, enabling controlled ablations on ScienceWorld. Across four configurations—baseline, baseline+AMM, baseline+SRM, and the full system—the full system achieves the best mean final score (64.62), success rate (43.17%), and successful-step efficiency (19.33 steps), while SRM is the strongest standalone contributor. The results suggest that execution-time control is the dominant bottleneck in this setting, while episodic memory becomes most useful once the runtime loop is stabilized.

• Large language models can generate fluent text and solve many short-horizon reasoning problems, but these abilities do not automatically yield reliable agentic behavior. Interactive environments require an agent to maintain state over time, choose actions that are valid in the current world, decompose goals into executable subgoals, recover from unexpected observations, and avoid locally plausible but unproductive loops. The central problem is therefore not only what an agent knows, but how it coordinates fast action proposal, slower deliberation, memory, and execution-time monitoring. A common failure pattern in such environments is that a reasonable high-level plan degrades at execution time: the agent repeats a stale action, proposes an action unavailable in the current state, or continues navigating despite no measurable progress. These failures are difficult to solve with larger prompts alone because they occur at the interface between language generation and environment transition.

• This work studies this problem through a dual-process language-agent architecture. SwiftSage Lin et al. (2024) provides a natural baseline because it separates Swift , a fast System 1-style action proposer, from Sage , a slower System 2-style planner whose outputs are executed through an action buffer. This design improves efficiency over methods that query a large model at every timestep, but it leaves two important limitations in long-horizon tasks. First, the agent lacks persistent episodic memory: it cannot selectively reuse salient prior experience across episodes. Second, it does not systematically validate actions immediately before execution or intervene in a bounded way when behavior stagnates.

• We address these limitations with two modular extensions. The Adaptive Memory Module (AMM) adds salience-gated episodic writing and trigger-driven retrieval. It records compact episodes after informative transitions, such as success, positive progress, near misses, or invalid-action failures, and retrieves relevant memories only at selected recovery and planning points. The Self-Reflection Module (SRM) adds bounded execution-time control. It validates actions immediately before they reach the environment, monitors post-step trajectory signals for stagnation, and invokes a constrained Critic only when corrective intervention is justified.

• The modules occupy different causal interfaces. AMM is informational: it changes what the controller can remember and reuse, but it never chooses the final action. SRM is control-oriented: it changes what is allowed to reach the environment and when corrective actions are inserted. This separation makes the resulting ablation study interpretable as cooperation between evidence and control rather than as an opaque redesign of the baseline agent.

• We evaluate the system on ScienceWorld Wang et al. (2022) , an interactive text benchmark requiring agents to perform elementary science tasks through grounded sequential actions. The evaluation compares four configurations under a shared runtime substrate: baseline, baseline+AMM, baseline+SRM, and the full system. The results show that the full system achieves the best aggregate performance, while SRM is the strongest standalone extension. AMM alone yields smaller gains, but its contribution is more coherent once SRM stabilizes execution. Overall, the findings support a precise claim: in this setting, execution-time control is the dominant lever for improving interactive language agents, while episodic memory is most useful when inserted as bounded evidence into an already controlled runtime loop.

• Dual-process theory distinguishes fast, automatic cognition from slower deliberative reasoning Kahneman (2011) . In language-agent design, this distinction is useful as an engineering abstraction rather than a claim of cognitive equivalence: fast pathways support cheap local action proposal, while slower pathways support planning, verification, and recovery. Chain-of-thought prompting and related methods make deliberation more explicit in static reasoning tasks Wei et al. (2022) ; Press et al. (2023) ; Khot et al. (2023) ; Zhou et al. (2023) ; Wang et al. (2023) ; Zhang et al. (2023) , but interactive agents additionally need to decide when to deliberate, how to ground plans in current state, and how to prevent invalid actions from consuming environment steps.

• A second relevant line of work augments LLMs with actions, tools, and feedback, including affordance-grounded action selection in SayCan Ahn et al. (2022) . ReAct interleaves reasoning traces with environment-facing actions Yao et al. (2023) , Reflexion uses verbal feedback from prior attempts Shinn et al. (2024) , Self-Refine iteratively revises model outputs Madaan et al. (2024) , and CRITIC verifies and corrects outputs using external tools Gou et al. (2024) . Toolformer and ART further show that tool calls can be learned or orchestrated in multi-step reasoning pipelines Schick et al. (2023) ; Paranjape et al. (2023) . These approaches show that feedback can improve LLM behavior, but they also motivate boundedness: unconstrained self-correction can increase cost or degrade performance when feedback is unreliable Huang et al. (2023) . SRM follows this lesson by making reflection trigger-gated and execution-facing rather than always on.

• AMM is motivated by Complementary Learning Systems, where rapid episodic learning and slower consolidation play distinct roles Marr (1971) ; Mcclelland et al. (1995) , as well as by memory-augmented generation and cognitive-agent memory systems. Retrieval-augmented generation conditions model outputs on external information Lewis et al. (2020) , while MemGPT treats the context window as a scarce resource and explicitly moves information between transient context and persistent storage Packer et al. (2023) . CoALA similarly frames language agents in terms of memory, actions, and decision procedures Sumers et al. (2023) , and long-horizon language-agent systems such as Generative Agents show how memory, reflection, and planning can support coherent behavior over extended interactions Park et al. (2023) . AMM adapts these ideas to ScienceWorld by storing compact episodic records and retrieving them only under operational triggers, instead of using memory as continuous prompt expansion or as an alternative planner.

• The closest architectural predecessor is SwiftSage Lin et al. (2024) . It already instantiates fast and slow thinking in ScienceWorld: Swift proposes local actions using an efficient model, while Sage performs higher-level planning and grounding through an action buffer. ScienceWorld is especially suitable for this comparison because it evaluates whether science knowledge can be transformed into valid procedures, not merely whether a model can state the right answer. Our contribution is to preserve the SwiftSage substrate while adding two missing mechanisms: persistent episodic reuse and just-in-time execution control. This framing also separates our work from approaches that simply add more reasoning calls. We ask whether memory and reflection improve an already dual-process controller when inserted at bounded, causally interpretable interfaces.

• The proposed system preserves the SwiftSage control loop and inserts AMM and SRM at precise runtime interfaces. At each timestep, the controller executes the next buffered action when available; otherwise it queries Swift, escalating to Sage under baseline conditions such as invalid actions, no-progress behavior, or the need for deliberate planning. AMM can augment selected Swift, Sage, and Critic prompts with retrieved memories. SRM validates actions before execution and can inject bounded corrective actions into the same buffer used by Sage.

• Figure 1: Full-system architecture. SwiftSage remains the action-selection substrate; AMM adds salience-gated episodic writing and trigger-driven retrieval; SRM adds Gate–1 validation, stagnation detection, and bounded Critic correction before actions reach ScienceWorld.

• AMM addresses the absence of persistent experience reuse. It has two hook families: a post-step write hook and pre-decision retrieval hooks. The write hook runs after the environment returns an observation and score transition. It receives the task, previous state, executed action, resulting observation, score change, and recent history, then builds a candidate episodic record. This record is stored only if a salience gate detects an informative transition, such as terminal success, positive score change, near-miss progress, explicit invalid-action feedback, or an avoidance-worthy failure.

• Stored memories are compact semi-structured records, not raw transcripts. Each memory includes fields such as task, local state, recent context, executed action, resulting observation, score transition, and a type tag. The tag distinguishes success, near-miss, and avoidance-oriented memories. This representation improves retrieval targeting and makes prompt injection safer, because retrieved evidence is already concise, typed, and easy to filter or truncate.

• Retrieval is trigger-driven. T1 corresponds to Swift failure: when Swift fails to produce a valid action, AMM retrieves related episodes and retries Swift with memory-conditioned context. T4 corresponds to System 2 planning: when the baseline invokes Sage and memory planning is enabled, AMM retrieves success and near-miss episodes for deliberative planning. T2 and T3 are implemented as stagnation and repeated-invalid-action retrieval/caching hooks, but in the evaluated AMM-only configuration they do not directly modify model inputs. This conservative design avoids injecting weakly grounded negative evidence into the fast pathway.

• Prompt augmentation is bounded and explicitly delimited. Swift receives only a small memory block in recovery mode, while Sage may receive a slightly larger block because it is already the deliberative pathway. Prompts instruct the model to treat memories as hints rather than authority: current observations, inventory, admissible actions, and runtime constraints always dominate. Retrieved memories are deduplicated, filtered by operational type, and truncated using a fixed compression order before injection. If retrieval, formatting, or prompt-structure checks fail, execution falls back to the unmodified baseline prompt. AMM therefore changes the evidence available to the controller without changing the final execution channel.

• SRM targets execution-time failures: actions may be plausible in language but invalid, stale, redundant, or ineffective in the current environment. It consists of Gate–1 validation, post-step stagnation detection, and bounded Critic intervention.

• Gate–1 is applied immediately before an action reaches the environment. It receives the proposed action, current valid-action set, recent state descriptors, and runtime constraints. It normalizes the action, checks admissibility, applies deterministic repair when the mismatch is minor and safe, and drops the action if it remains invalid or violates constraints. Gate–1 is source-agnostic: Swift actions, Sage-buffered actions, and Critic-generated actions all pass through the same pre-execution control surface.

Structure (section headings from HTML)

• Cognitive Extensions for Dual-Process Language Agents: Memory and Self-Reflection in Interactive Environments
• 1 Introduction
• 2 Motivation and Related Work
• 3 Method

• 3.1 Adaptive Memory Module
  

• 3.2 Self-Reflection Module
  

• 3.3 Full-System Composition
  

• 4 Experimental Setup
• 5 Results and Analysis
• 6 Discussion
• 7 Conclusion
• 8 Limitations
• 9 Ethical Considerations
• References
• Appendix A Reproducibility Notes
• Appendix B Score Results by Task
• Appendix C Success and Successful-Step Efficiency
• Appendix D Mechanism-Level Activity
• Appendix E System Usage and Cost Proxies
• Appendix F Sensitivity Analyses

Remainder

Full original English text: see html_url / source_url / pdf_url in frontmatter.