On-Demand Attention: Language Models Know When to Recall

Note: body below is original English text extracted from arXiv abs / HTML. Do not treat this file as a translation.

arXiv:2609.20734 · published 2026-09-17 · submitted 2026-09-17

Abstract

Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, regardless of its benefit to the next prediction. We show that a pretrained model's decoding states already contain information predictive of this benefit, before the global read. Building on this finding, we introduce On-Demand Attention (ODA), a local-first decoding method that uses a lightweight recall head to selectively invoke global attention as its predicted benefit changes during generation. ODA trains only the recall head, leaving pretrained weights unchanged and the complete historical KV cache available for future recall. We further implement GPU-side conditional execution in vLLM, translating reduced global reads into practical decoding speedups over full attention at long context lengths. Experiments across Qwen and Gemma models, including hybrid-attention backbones, show that selective recall recovers most of the performance lost under local attention while substantially reducing global reads. These findings support long-context inference in which pretrained models guide their own access to the information they retain.

Authors

Haibo Feng, Ruiqi Liang, Hanyang Peng, Shiqi Yu

Key claims (verbatim-leaning English extract)

• Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, regardless of its benefit to the next prediction. We show that a pretrained model's decoding states already contain information predictive of this benefit, before the global read. Building on this finding, we introduce On-Demand Attention (ODA), a local-first decoding method that uses a lightweight recall head to selectively invoke global attention as its predicted benefit changes during generation. ODA trains only the recall head, leaving pretrained weights unchanged and the complete historical KV cache available for future recall. We further implement GPU-side conditional execution in vLLM, translating reduced global reads into practical decoding speedups over full attention at long context lengths. Experiments across Qwen and Gemma models, including hybrid-attention backbones, show that selective recall recovers most of the performance lost under local attention while substantially reducing global reads. These findings support long-context inference in which pretrained models guide their own access to the information they retain.
• We show that a pretrained model’s decoding states already contain information predictive of this benefit, before the global read.
• Building on this finding, we introduce On-Demand Attention (ODA), a local-first decoding method that uses a lightweight recall head to selectively invoke global attention as its predicted benefit changes during generation.
• Building on these findings, we introduce On-Demand Attention (ODA), which uses a lightweight recall head to turn the model’s existing decoding states into attention decisions.
• 5% global-call rate, ODA reduces FLOPs for the main decoding operations by approximately 76% and achieves 1.
• We introduce ODA, which makes global attention conditional on predicted benefit through a lightweight recall head, while keeping pretrained weights unchanged and preserving access to the complete historical KV cache.
• To account for this cost when learning the decision, we introduce a penalty λ ≥ 0 \lambda\geq 0 , expressed on the prediction-loss scale.
• This behavior extends to the larger Qwen3-8B, which achieves 91.

Structure (section headings from HTML)

• On-Demand Attention: Language Models Know When to Recall
  • 1 Introduction
  • 2 On-Demand Attention
    • 2.1 Problem setup and recall objective
    • 2.2 Learning the recall head
    • 2.3 Attention allocation during decoding
  • 3 Experiments
    • 3.1 Experimental setup
    • 3.2 Generation quality and selective global access
      • Choosing when to recall matters.
    • 3.3 Computation and decoding efficiency
    • 3.4 Understanding recall decisions
      • The inputs shape the learned access policy.
      • Head scores identify beneficial global access.
      • Recall predicts relative benefit, not Local errors.
  • 4 Related Work
    • Dynamic local–global attention.
    • Selective context access.
    • Hybrid architectures and efficient attention execution.
    • Model-guided information acquisition.
    • Adaptive execution and speculative decoding.
  • 5 Conclusion
  • AI Use Statement
  • Reproducibility Statement
  • References
  • Appendix A Additional Method Details
    • A.1 Access gain and the cost of routing errors
    • A.2 Recall-head architecture
    • A.3 Paired supervision and historical inputs
      • Branch computation.
      • Previous hidden state during training.
      • Prefill handoff.
    • A.4 Regression target and reduction
    • A.5 Selected-state execution
    • A.6 Cost of conditional recomputation
  • Appendix B Additional Experiments and Evaluation Protocols
    • B.1 Training and evaluation settings
      • Data and optimization.
      • Generation protocol.
    • B.2 Per-task generation results
    • B.3 Random and learned global access
      • Common evaluation.
      • Random control.
    • B.4 Input composition and training behavior
      • Input subsets.
      • Primary checkpoints and alternative supervision.
    • B.5 Fixed-reference Qwen3 diagnostics
      • Collection.
      • Benefit ranking.
      • Benefit is distinct from correctness.
    • B.6 Independent-reference Qwen3.5 pilot
    • B.7 Access benefit and distribution disagreement
    • B.8 Runtime protocol and complete results
      • Environment and workload.
      • Timing.
      • Major-operation FLOPs.
      • Numerical checks and scope.
    • B.9 Training resources
  • Appendix C Discussion and Limitations
  • Instructions for reporting errors

Body excerpts (original English)

On-Demand Attention:Language Models Know When to Recall Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 On-Demand Attention 2.1 Problem setup and recall objective 2.2 Learning the recall head 2.3 Attention allocation during decoding 3 Experiments 3.1 Experimental setup 3.2 Generation quality and selective global access Choosing when to recall matters. 3.3 Computation and decoding efficiency 3.4 Understanding recall decisions The inputs shape the learned access policy. Head scores identify beneficial global access. Recall predicts relative benefit, not Local errors. 4 Related Work Dynamic local–global attention. Selective context access. Hybrid architectures and efficient attention execution. Model-guided information acquisition. Adaptive execution and speculative decoding. 5 Conclusion References A Additional Method Details A.1 Access gain and the cost of routing errors A.2 Recall-head architecture A.3 Paired supervision and historical inputs Branch computation. Previous hidden state during training. Prefill handoff. A.4 Regression target and reduction A.5 Selected-state execution A.6 Cost of conditional recomputation B Additional Experiments and Evaluation Protocols B.1 Training and evaluation settings Data and optimization. Generation protocol. B.2 Per-task generation results B.3 Random and learned global access Common evaluation. Random control. B.4 Input composition and training behavior Input subsets. Primary checkpoints and alternative supervision. B.5 Fixed-reference Qwen3 diagnostics Collection. Benefit ranking. Benefit is distinct from correctness. B.6 Independent-reference Qwen3.5 pilot B.7 Access benefit and distribution disagreement B.8 Runtime protocol and complete results Environment and workload. Timing. Major-operation FLOPs. Numerical checks and scope. B.9 Training resources C Discussion and Limitations License: arXiv.org perpetual non-exclusive license arXiv:2609.20734v1 [cs.CL] 17 Sep 2026 On-Demand Attention: Language Models Know When to Recall Haibo Feng Affiliation: Southern University of Science and Technology Ruiqi Liang Affiliation: Peking University Hanyang Peng Affiliation: Peng Cheng Laboratory Shiqi Yu Affiliation: Southern University of Science and Technology Abstract Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, regardless of its benefit to the next prediction. We show that a pretrained model’s decoding states already contain information predictive of this benefit, before the global read. Building on this finding, we introduce On-Demand Attention (ODA), a local-first decoding method that uses a lightweight recall head to selectively invoke global attention as its predicted benefit changes during generation. ODA trains only the recall head, leaving pretrained weights unchanged and the complete historical KV cache available for future recall. We further implement GPU-side conditional execution in vLLM, translating reduced global reads into practical decoding speedups over full attention at long context lengths. Experiments across Qwen and Gemma models, including hybrid-attention backbones, show that selective recall recovers most of the performance lost under local attention while substantially reducing global reads. These findings support long-context inference in which pretrained models guide their own access to the information they retain. 1 Introduction Large language models are increasingly used for reasoning and agentic tasks that involve extended generation, making decoding efficiency a growing concern ( Anthropic, 2026 ; Yang et al., 2026 ) . As context lengths grow, attention becomes a major source of decoding cost ( Tang et al., 2024 ) . However, the need for distant information varies across generation steps ( Vakilian et al., 2026 ; Luo et al., 2025 ) , while full-attention decoding performs global computation at every step. Matching the use of global attention to the needs of each prediction is therefore a key challenge for efficient long-context inference. Such an allocation requires predicting the value of global attention before performing it. One class of dynamic attention methods incorporates this decision into training, jointly learning language modeling and local–global attention allocation ( Luo et al., 2025 ; Choudhary et al., 2026 ; Zhao et al., 2026 ; Pan et al., 2026 ) . Our starting point is that the states a pretrained model forms during generation may already contain information about the benefit of extending attention beyond the local context. If this information can be read out, the same generation process can support both next-token prediction and attention allocation, enabling the model to invoke global attention on demand without changing its pretrained weights. Our analysis shows that the predictive benefit of global attention varies across generation steps: expanding the attention span does not always improve the current prediction. Readout experiments with a frozen backbone further show that the available decoding states contain signals predictive of this benefit relative to local attention, before global attention is computed for the current step (Figure 4 ). Together, these observations provide an empirical basis for using information already available during generation to assess the value of additional attention computation. Building on these findings, we introduce On-Demand Attention (ODA), which uses a lightweight recall head to turn the model’s existing decoding states into attention decisions. Figure 1 illustrates the resulting allocation: ODA first computes locally, then selectively recomputes the current step with global attention according to its predicted benefit. Only the recall head is trained; pretrained weights remain unchanged, and the complete historical KV cache remains available for future recall. Figure 1: On-Demand Attention. Top: ODA performs Local at every step and selectively adds Full. Recall can change the top-ranked token (MAPLE to ORCHID) or change its probability while retaining it (Friday). Words, distributions, and the call schedule are schematic. Bottom: The recall head combines the preceding selected hidden state, current token embedding, and current Local hidden state. When q t > θ q_{t}>\theta , Full recomputes the step from the same pre-step history. The selected h t h_{t} feeds the frozen LM head; the current-position K ​ V t KV_{t} from the same branch is committed to the retained history. Both access views share this complete history. Only the recall head is trained. The O ⁡ ( n ) O(n) and O ⁡ ( n 2 ) O(n^{2}) labels denote cumulative Local and Full attention costs across n n causal positions, with fixed model dimensions and a fixed Local window. Our RULER ( Hsieh et al., 2024 ) and LongBench ( Bai et al., 2024 ) evaluations on Qwen3-1.7B ( Qwen Team, 2025 ) show that ODA recovers most of the performance lost under local attention while substantially reducing global-attention call frequency. For example, on RULER16K, ODA uses Full on 41.6% of routed steps while improving the score from 19.23 under local attention to 81.17, close to the full-attention score of 81.94 (Table 1 ). Complementary cost analysis and vLLM ( Kwon et al., 2023 ) measurements further establish the efficiency gains: in a controlled experiment with 128K input tokens, 1K output tokens, and a 12.5% global-call rate, ODA reduces FLOPs for the main decoding operations by approximately 76% and achieves 1.98 × 1.98\times the single-request decoding throughput of native full attention. Moreover, on the hybrid Qwen3.5-2B backbone ( Qwen Team, 2026 ) , ODA reduces global-call frequency with little loss in task performance, showing that allocation across generation steps can complement architectural efficiency improvements. The Qwen3-8B and Gemma-4-12B-it ( Gemma Team, 2026 ) results further extend this quality–access trade-off to a larger Qwen backbone and a different model family, using separately trained recall heads. Our contributions are threefold: • Predicting the value of global attention. We identify signals in a frozen model’s decoding states that predict the relative benefit of global attention before it is computed, providing an empirical basis for allocating attention according to predicted benefit. • On-demand decoding for pretrained models. We introduce ODA, which makes global attention conditional on predicted benefit through a lightweight recall head, while keeping pretrained weights unchanged and preserving access to the complete historical KV cache. • Establishing quality and efficiency gains. We demonstrate selective recall across Qwen and Gemma models, spanning model scales and hybrid-attention backbones, and quantify computational savings and decoding acceleration on Qwen3-1.7B through FLOPs analysis and controlled vLLM measurements. 2 On-Demand Attention On-Demand Attention (ODA) first performs a Local computation, then uses the model’s available decoding states to decide whether to recompute the current step with global attention (Figure 1 , bottom). We first define the relative benefit underlying this decision (Section 2.1 ), then describe how the recall head learns to predict it before the current Full computation (Section 2.2 ). Finally, we explain how these predictions guide attention allocation during decoding, analyze the computational savings, and describe an efficient implementation (Section 2.3 ). 2.1 Problem setup and recall objective During autoregressive decoding, attention aggregates contextual values according to their compatibility with the current query ( Vaswani et al., 2017 ) . Historical keys and values are stored in a KV cache and reused across steps ( Kwon et al., 2023 ) . For one attention head at position t t , let ( K < t , V < t ) (K_{<t},V_{<t}) denote this cache, and let 𝐪 t , 𝐤 t , 𝐯 t \mathbf{q}{t},\mathbf{k}{t},\mathbf{v}{t} be the query, key, and value produced for the current input. Combining the cached history with the current key and value gives K ~ t = [ K < t ; 𝐤 t ] , V ~ t = [ V < t ; 𝐯 t ] . \widetilde{K}{t}=[K_{<t};\mathbf{k}{t}],\qquad\widetilde{V}{t}=[V_{<t};\mathbf{v}{t}]. (1) For an access set 𝒮 t ⊆ { 1 , … , t } \mathcal{S}{t}\subseteq{1,\ldots,t} , the attention output is 𝐨 t = softmax ⁡ ( 𝐪 t ​ K ~ t ​ [ 𝒮 t ] ⊤ d k ) ​ V ~ t ​ [ 𝒮 t ] , \mathbf{o}{t}=\operatorname{softmax}!\left(\frac{\mathbf{q}{t}\widetilde{K}{t}[\mathcal{S}{t}]^{\top}}{\sqrt{d{k}}}\right)\widetilde{V}{t}[\mathcal{S}{t}], (2) where d k d_{k} is the key dimension and set indexing selects the corresponding positions. Layer and head indices are omitted for clarity. Thus, historical KV can remain fully retained while 𝒮 t \mathcal{S}{t} determines which positions participate in the current attention computation. ODA uses this distinction to vary the attention span within the same pretrained model. Full attends to every available position, whereas Local attends to s s initial positions and a recent window of width w w , following the initial-token-plus-window pattern of StreamingLLM ( Xiao et al., 2024b ) . Our window includes the current token: 𝒮 t F \displaystyle\mathcal{S}{t}^{F} = { 1 , … , t } , \displaystyle={1,\ldots,t}, (3) 𝒮 t L \displaystyle\mathcal{S}{t}^{L} = { 1 , … , min ⁡ ( s , t ) } ∪ { max ⁡ ( 1 , t − w + 1 ) , … , t } . \displaystyle={1,\ldots,\min(s,t)}\cup{\max(1,t-w+1),\ldots,t}. At decoding position t t , the model processes the known input token x t x{t} to predict the next token. Let C t − 1 C_{t-1} denote the historical state committed before this step. Starting from the same C t − 1 C_{t-1} , Local and Full process x t x_{t} with the same pretrained parameters ϕ \phi and produce next-token distributions p t L p_{t}^{L} and p t F p_{t}^{F} , respectively. Their current states are computed separately under the corresponding access sets. To quantify the value of adding global attention at this step, we measure the reduction in negative log-likelihood (NLL) on the same reference next token x t + 1 x_{t+1} : g t \displaystyle g_{t} = NLL L ⁡ ( x t + 1 ) − NLL F ⁡ ( x t + 1 ) \displaystyle=\operatorname{NLL}{L}(x{t+1})-\operatorname{NLL}{F}(x{t+1}) (4) = log ⁡ p t F ​ ( x t + 1 ) − log ⁡ p t L ​ ( x t + 1 ) . \displaystyle=\log p_{t}^{F}(x_{t+1})-\log p_{t}^{L}(x_{t+1}). Positive gain means that Full assigns greater probability to the reference token; negative gain favors Local. The gain therefore captures both the direction and magnitude of the change in predictive quality. It can be nonzero even when both branches make the same top-ranked prediction, as illustrated by the schematic Friday example in Figure 1 . At a fixed history and reference token, choosing the worse branch incurs an additional | g t | |g_{t}| in NLL (Appendix A.1 ). Paired computations provide this gain during training, but online decoding has access to neither the reference next token nor the current Full prediction when the recall decision is made. The decision must therefore rely on states already available after the Local computation. We next describe how the recall head uses these states to predict a score associated with the relative gain. 2.2 Learning the recall head To predict the benefit of Full before invoking it, the recall head produces a scalar score from three inputs available at the current step: q t = R ψ ​ ( h t − 1 , E ϕ ​ ( x t ) , h t L ) , q_{t}=R_{\psi}\bigl(h_{t-1},E_{\phi}(x_{t}),h_{t}^{L}\bigr), (5) where E ϕ ​ ( x t ) E_{\phi}(x_{t}) is the embedding of the known input token, h t L h_{t}^{L} is the final normalized hidden state from the current Local computation, and h t − 1 h_{t-1} is the hidden state passed from the preceding position. During deployment, h t − 1 h_{t-1} comes from the Local or Full branch actually selected at the previous step and is updated after every step. Together, these inputs describe the preceding computation, the current input, and the current Local result. Only the head parameters ψ \psi are trained; all pretrained parameters ϕ \phi remain unchanged. The head architecture is specified in Appendix A.2 . We obtain supervision from paired Local and Full computations over a shared Full history, with the backbone frozen. During training, the preceding hidden-state input is selected using the preceding position’s reference gain; during deployment, both this input and the history follow the actual branch decisions. Algorithm 1 (a) summarizes how paired supervision and input states are constructed, alongside the decoding procedure in Algorithm 1 (b). The complete construction and sequence-boundary handling are given in Appendices A.3 and A.3 . Invoking Full adds computation beyond the Local attempt. To account for this cost when learning the decision, we introduce a penalty λ ≥ 0 \lambda\geq 0 , expressed on the prediction-loss scale. Relative to accepting Local, the cost-adjusted gain is d t \displaystyle d_{t} = NLL L ⁡ ( x t + 1 ) − [ NLL F ⁡ ( x t + 1 ) + λ ] \displaystyle=\operatorname{NLL}{L}(x{t+1})-\bigl[\operatorname{NLL}{F}(x{t+1})+\lambda\bigr] (6) = g t − λ . \displaystyle=g_{t}-\lambda. Thus, d t > 0 d{t}>0 when the predictive improvement from Full exceeds its assigned cost. We treat λ \lambda as a cost hyperparameter, with configuration-specific values reported in Appendix A.4 . To compress large gains while preserving their sign and ordering, we apply a signed logarithmic transform to the cost-adjusted target: y t = T ⁡ ( d t ) , T ⁡ ( u ) = sign ⁡ ( u ) ​ log ⁡ ( 1 + | u | ) . y_{t}=T(d_{t}),\qquad T(u)=\operatorname{sign}(u)\log(1+|u|). (7) We then train the recall head with Huber regression ( Huber, 1964 ) : ℒ ⁡ ( ψ ) = 1 | ℰ | ​ ∑ t ∈ ℰ ℓ Huber , 1 ​ ( q t − y t ) , \mathcal{L}(\psi)=\frac{1}{|\mathcal{E}|}\sum_{t\in\mathcal{E}}\ell_{\mathrm{Huber},1}(q_{t}-y_{t}), (8) where ℰ \mathcal{E} contains eligible reference positions in a training batch. Eligibility requires a valid reference next token and different Local and Full access sets. We retain near-zero gains and use no gain-dependent token weights. The reduction across workers and accumulated batches is specified for each training configuration in Appendix A.4 . The learned score is then used to decide whether to invoke Full. Algorithm 1 Training and inference of On-Demand Attention Pretrained parameters ϕ \phi are frozen; only R ψ R_{\psi} is trained. (a) One training update 1: Token batch X X , λ \lambda , τ hist \tau_{\mathrm{hist}} 2: ℰ ← Eligible ​ ( X ) \mathcal{E}\leftarrow\textsc{Eligible}(X) 3: ( C F , H tr ) ← FullTrunk ​ ( X ) (C^{F},H^{\mathrm{tr}})\leftarrow\textsc{FullTrunk}(X) 4: ( H L , H F ) ← Pair ​ ( X , C F ) (H^{L},H^{F})\leftarrow\textsc{Pair}(X,C^{F}) 5: g ← AccessGain ​ ( H L , H F , X ) g\leftarrow\textsc{AccessGain}(H^{L},H^{F},X) 6: H train ← Select ​ ( H L , H tr , g , τ hist ) H^{\mathrm{train}}\leftarrow\textsc{Select}(H^{L},H^{\mathrm{tr}},g,\tau_{\mathrm{hist}}) 7: H − ← Shift ​ ( H train ) H^{-}\leftarrow\textsc{Shift}(H^{\mathrm{train}}) 8: for t ∈ ℰ t\in\mathcal{E} do 9: q t ← R ψ ​ ( H t − , E ϕ ​ ( x t ) , h t L ) q_{t}\leftarrow R_{\psi}(H^{-}{t},E{\phi}(x_{t}),h_{t}^{L}) 10: y t ← T ⁡ ( g t − λ ) y_{t}\leftarrow T(g_{t}-\lambda) 11: end for 12: ℒ ← HuberReduce ​ ( q , y , ℰ ) \mathcal{L}\leftarrow\textsc{HuberReduce}(q,y,\mathcal{E}) 13: Update ψ \psi using ∇ ψ ℒ \nabla_{\psi}\mathcal{L} (b) One decoding step 1: x t x_{t} , C t − 1 C_{t-1} , h t − 1 h_{t-1} , θ \theta 2: ( h t L , Δ ​ C t L ) ← F ϕ L ​ ( x t , C t − 1 ) (h_{t}^{L},\Delta C_{t}^{L})\leftarrow F_{\phi}^{L}(x_{t};C_{t-1}) 3: q t ← R ψ ​ ( h t − 1 , E ϕ ​ ( x t ) , h t L ) q_{t}\leftarrow R_{\psi}(h_{t-1},E_{\phi}(x_{t}),h_{t}^{L}) 4: if q t > θ q_{t}>\theta or q t q_{t} is nonfinite then 5: ( h t , Δ ​ C t ) ← F ϕ F ​ ( x t , C t − 1 ) (h_{t},\Delta C_{t})\leftarrow F_{\phi}^{F}(x_{t};C_{t-1}) 6: else 7: ( h t , Δ ​ C t ) ← ( h t L , Δ ​ C t L ) (h_{t},\Delta C_{t})\leftarrow(h_{t}^{L},\Delta C_{t}^{L}) 8: end if 9: C t ← C t − 1 ∥ Δ ​ C t C_{t}\leftarrow C_{t-1}\mathbin{|}\Delta C_{t} 10: p t ← softmax ⁡ ( W LM ​ h t ) p_{t}\leftarrow\operatorname{softmax}(W_{\mathrm{LM}}h_{t}) 11: x t + 1 ← Decode ​ ( p t ) x_{t+1}\leftarrow\textsc{Decode}(p_{t}) 12: return x t + 1 , C t , h t x_{t+1},C_{t},h_{t} Pair computes both candidates from the Full-trunk history. Select uses H L H^{L} at eligible positions with g t < τ hist g_{t}<\tau_{\mathrm{hist}} , and H tr H^{\mathrm{tr}} otherwise. Shift moves the selected states one position forward within each sequence, using zero at its start. HuberReduce applies Equation 8 , using the configuration-specific reduction in Appendix A.4 . Decoding begins after Full prefill; W LM W_{\mathrm{LM}} is the frozen vocabulary projection and Δ ​ C t \Delta C_{t} is the selected current-step state update. Details are in Appendix A . 2.3 Attention allocation during decoding At inference time, ODA uses the recall score to decide whether to add global computation. Each step first runs Local. If q t > θ q_{t}>\theta , the model recomputes the current step with Full from the same pre-step history; otherwise, it accepts the Local result. The decision threshold θ \theta is set to zero by default. The selected hidden state h t h_{t} passes through the shared LM head and is carried to the next step; its corresponding current-position KV is appended to the retained history. The Local and Full views in Figure 1 access the same committed history, and recomputation changes only the current-step candidate. Every earlier position therefore remains accessible even after consecutive Local steps. Algorithm 1 (b) gives the complete decoding procedure. For hybrid backbones, the same decision changes the access range of ordinary attention layers while preserving the model’s native recurrent computation. This execution scheme pays for a Local attempt at every step in exchange for opportunities to skip Full computation. To characterize this trade-off, at a given history length let F L F_{L} and F F F_{F} denote the backbone FLOPs of Local and Full, and let F R F_{R} denote the recall-head FLOPs. If ρ \rho is the fraction of steps invoking Full, counting the Local attempt and Full recomputation as separate forward passes gives the average per-step savings relative to always using Full: Δ ​ F = ( 1 − ρ ) ​ F F ⏟ skipped Full computation − ( F L + F R ) ⏟ Local attempt and decision . \Delta F=\underbrace{(1-\rho)F_{F}}{\text{skipped Full computation}}-\underbrace{(F{L}+F_{R})}{\text{Local attempt and decision}}. (9) The final vocabulary projection is performed once in either method and therefore cancels from the difference. Savings arise when the avoided Full computation exceeds the cost of the Local attempt and recall decision. With fixed model dimensions and Local access parameters s s and w w , the per-step attention costs are O ⁡ ( 1 ) O(1) for Local and O ⁡ ( n ) O(n) for Full, where n n is the history length. Accumulating these costs across a causal sequence of n n positions gives O ⁡ ( n ) O(n) for Local and O ⁡ ( n 2 ) O(n^{2}) for Full, as labeled in Figure 1 . These are branch attention costs; ODA’s savings also account for the Local attempt, head, and conditional Full recomputation through Equation 9 . Consequently, at a fixed recall rate ρ < 1 \rho<1 , longer contexts offer greater computational savings. To translate these savings into decoding acceleration, we implement GPU-side conditional execution in vLLM ( Kwon et al., 2023 ) , keeping the Local computation, recall decision, and conditional Full recomputation within a common execution flow to reduce routing-related synchronization overhead. Both branches share the subsequent output computation. Complete FLOPs and latency analyses, together with state-management and implementation details, are provided in Appendices A.6 and A.5 . 3 Experiments We evaluate whether selective global attention can preserve generation quality while reducing the cost of long-context decoding. We first compare quality and global access across models and tasks, and test whether learned access timing matters (Section 3.2 ). We then measure computation and decoding speed (Section 3.3 ), before examining the states and benefits underlying recall decisions (Section 3.4 ). 3.1 Experimental setup Our primary model is Qwen3-1.7B ( Qwen Team, 2025 ) , evaluated on RULER16K and LongBench v1 ( Hsieh et al., 2024 ; Bai et al., 2024 ) . RULER16K contains 13 tasks with 100 examples each; our LongBench v1 evaluation covers 13 tasks and 2,550 examples. Qwen3-8B tests a larger scale, Qwen3.5-2B ( Qwen Team, 2026 ) hybrid compatibility, and Gemma-4-12B-it ( Gemma Team, 2026 ) applicability beyond the Qwen family. We compare ODA with each backbone’s native attention configuration (Full) and a fixed Local policy. Local and ODA use four initial positions and a 2,048-token recent window, including the current token. In hybrid backbones, this restriction applies to attention layers that read the full history; the remaining layers retain their native computation. All quality evaluations use Full prefill followed by each policy’s own generated continuation. Each backbone has a separately trained recall head and unchanged pretrained weights. We report task-macro scores and actual decode-time Full-call rates; the latter are token-weighted unless specified otherwise. Efficiency measurements include the Local attempt, head, and conditional Full recomputation, and exclude prefill. Checkpoints, decoding settings, and metric definitions appear in Appendix B.1 . 3.2 Generation quality and selective global access Table 1: Long-context generation across model scales, families, and architectures. Scores are task macros ( ↑ \uparrow ); parentheses report ODA’s actual Full-call rates ( ↓ \downarrow ). Full uses each backbone’s native attention configuration. Each backbone has its own recall head. LongBench v1 denotes the 13-task subset specified in Appendix B.2 . Dashes indicate measurements not yet available; checkpoint and backend details are in Appendix B.1 . RULER16K LongBench v1 Model Full Local ODA (Full%) Full Local ODA (Full%) Qwen3-1.7B 81.94 19.23 81.17 (41.6%) 37.94 23.66 36.82 (70.6%) Qwen3-8B 92.59 25.43 91.07 (47.3%) 44.70 29.83 — Qwen3.5-2B 94.35 28.21 94.08 (44.3%) 39.47 25.31 — Gemma-4-12B-it 96.61 30.07 95.27 (57.8%) 41.88 35.71 — Selective recall recovers most of the performance lost under local attention while substantially reducing global-call frequency (Table 1 ). On Qwen3-1.7B, ODA reaches 81.17 on RULER16K with 41.6% Full calls, compared with 81.94 for Full and 19.23 for Local. On LongBench v1, it reaches 36.82 against Full’s 37.94, using Full on 70.6% of routed steps. Thus, the same head recovers 98.8% and 92.2% of the respective Full–Local score gaps, while adapting its access frequency to the encountered tasks. Recovery varies by task: English passage retrieval retains a 10.33-point deficit, whereas Chinese passage retrieval matches Full. Appendix B.2 provides the complete task breakdown. This behavior extends to the larger Qwen3-8B, which achieves 91.07 with 47.3% Full calls, compared with Full’s 92.59 and Local’s 25.43 in the HF evaluations. ODA also complements the native hybrid architecture of Qwen3.5-2B: limiting access in its six full-attention layers yields 94.08 against Full’s 94.35, with a Full-call rate of 44.3%. The other 18 layers retain their Gated DeltaNet computation ( Yang et al., 2025 ) . Gemma-4-12B-it further extends this result across model families: its step-1,024 head scores 95.27 with 57.8% Full calls, versus 96.61 for Full. Each model uses a separately trained head; these results do not test head transfer. Choosing when to recall matters. To separate learned timing from access frequency, we compare ODA with random Full calls on five RULER16K tasks. For each task and ODA threshold, Random uses that task’s observed ODA call rate as its Bernoulli probability, and then follows its own generated trajectory. Figure 2 (a) plots the realized rates. At threshold zero, ODA scores 88.96 with a task-averaged Full rate of 41.33%; Random scores 32.79 ± 0.95 32.79\pm 0.95 at 40.99%, while the freshly evaluated Full baseline scores 88.63. The large quality gap persists across all eight tested budgets. Global-call frequency alone therefore does not explain ODA’s performance: allocating those calls according to the encountered states is consequential for generation. This five-task comparison uses its own common evaluation setup and baselines (Appendix B.3 ). Figure 2: Selecting when to use global attention. (a) Qwen3-1.7B generation on five RULER16K tasks, with 100 examples per task. Both axes average tasks equally. Points represent eight ODA thresholds and their Random controls at actual call rates; Random error bars show the sample standard deviation across three routing seeds on both axes. The ring marks ODA’s zero-threshold configuration. Dashed lines mark Full and Local score references. The horizontal range is 23–60%; the broken vertical axis displays 26–38 and 84–90 on separate linear scales. (b) Offline selection on 2,048 reference positions from 16 documents. Positions are ranked by head score (ODA), uniformly selected (Random expectation), or ranked by realized gain (Oracle). Signed gains are summed and divided by all 2,048 positions. Shading is a pointwise 95% document-bootstrap interval for ODA. This panel holds recorded histories fixed; it does not measure a generated Random or Oracle trajectory. 3.3 Computation and decoding efficiency Fewer global calls reduce computation, but every recalled step also pays for its preceding Local attempt. We therefore account for both branches, the recall head, and the vocabulary projection, following Section 2.3 . Figure 3 (a) shows total major decoding FLOPs as a percentage of Full at each context length. For the 128K input and approximately 12.5% Full calls, ODA uses 24.26% of Full’s FLOPs, a 75.74% reduction and a 4.12 × 4.12\times computation ratio. The relative cost decreases with context length, while larger recall budgets leave more repeated computation. These savings translate into faster decoding in vLLM ( Kwon et al., 2023 ) (Figure 3 (b)). We run the complete Qwen3-1.7B model and generation loop on one A100-SXM4-80GB, with batch size one and CUDA Graphs enabled for both ODA and native Full. At the same 128K/12.5% operating point, median throughput rises from 75.54 to 149.52 tokens/s ( 1.98 × 1.98\times ). At 4K, however, ODA remains 15.8% slower: avoiding short global reads does not offset the Local and conditional-execution overhead. Thus, the speed benefit depends on both context length and recall frequency. These measurements use a fixed continuation and prescribed recall schedule to compare execution costs under identical work; the real head still runs at every step. They are separate from the learned-policy quality evaluations above. Appendix B.8 reports all 30 operating points, first-request costs, and the exact timing scope. For training, the Qwen3-1.7B head contains 28.3M parameters and uses a 196,608-example training pool. Complete resource accounting must also include frozen-backbone feature and label computation; the outstanding GPU-hour measurements are identified in Appendix B.9 . Figure 3: Selective global attention reduces computation and accelerates long-context decoding. (a) Total major decoding FLOPs normalized to Full at each input length (100%), including the Local attempt, additional Full computation, head, and output projection. (b) Measured vLLM decode throughput relative to native Full on the same A100, using three-round warm medians. Each length uses one complete benchmark prompt and a fixed 1,024-token continuation; the actual input lengths are 4,139–128,852, shown as equally spaced categories in both panels. Legend rates are nominal: the three schedules execute 128, 256, or 512 Full calls in 1,023 routed steps. Prefill is excluded. These controlled schedules measure execution efficiency, not task quality. 3.4 Understanding recall decisions The inputs shape the learned access policy. We train seven heads with different subsets of the preceding committed hidden state ( H H ), current token embedding ( E E ), and current Local hidden state ( L L ). All share the same initialization, capacity, data, and 1,024-update training budget; all perform Local computation before deciding. This ablation uses the cost-adjusted training target with λ = 0.001 \lambda=0.001 and evaluates each head at threshold zero (Appendix B.4 ). With H + E H+E , the policy nearly always invokes Full (99.73%). Adding L L yields 81.57 with 76.22% Full calls, compared with 81.94 for H + E H+E (Table 2 ). Conversely, the embedding-only head uses 87.32% Full yet scores just 49.88. These differences show that input composition substantially changes the learned policy. Because the resulting budgets differ, they do not by themselves isolate input value at equal access frequency. The appendix details the evaluation protocol and the H + E + L H+E+L training trajectory. Table 2: Input composition changes the quality–access operating point. Qwen3-1.7B heads trained for 1,024 updates with a shared seed and the same 28,325,889-parameter architecture. Checkmarks identify distinct available inputs; missing streams are filled by a specified available input to preserve capacity. Every policy first computes Local. These are natural zero-threshold results, not budget-matched comparisons; this training configuration is distinct from the primary head in Table 1 . H H E E L L RULER16K Full (%) ✓ — — 76.84 83.20 — ✓ — 49.88 87.32 — — ✓ 81.60 89.25 ✓ ✓ — 81.94 99.73 ✓ — ✓ 67.96 21.86 — ✓ ✓ 81.53 97.51 ✓ ✓ ✓ 81.57 76.22 Head scores identify beneficial global access. We next evaluate the primary head on 2,048 external-reference positions, comparing Local and Full from the same recorded history. At 40% offline selection, ranking by head score captures 0.150 nats of net gain per position, compared with 0.053 for uniform selection (Figure 2 (b)). The head thus prioritizes positions with greater measured benefit, while accounting for the harm from selected negative gains. This complements the generation comparison by testing the decision target directly, without changing the saved histories. The signal also differs from simple uncertainty about the next token. Within the 675 positions where Local’s top-1 prediction is incorrect, the head achieves a benefit-sign AUROC of 0.643, versus 0.486 for Local entropy; the paired 95% interval for their difference is [ 0.069 , 0.230 ] [0.069,0.230] . Across all positions, the AUROC difference is unresolved, although head ranking yields greater net gain at 40% selection (Figure 4 ). These are diagnostics of the complete head, not a confidence-controlled attribution to L L alone. The smaller Qwen3.5 independent test does not establish the same failure-subset discrimination advantage (Appendix B.6 ). Figure 4: Access benefit and prediction uncertainty provide different rankings. The primary Qwen3-1.7B head and Local entropy are evaluated on the same 2,048 fixed-reference positions: 1,373 Local-correct and 675 Local-incorrect. (a) AUROC for positive signed gain, g t > 0 g{t}>0 . (b) Net signed gain when selecting 40% of positions within each population, divided by that population’s size. Error bars are 95% paired document-bootstrap intervals. Correctness defines diagnostic subsets after evaluation and is not an input to the policy. Recall predicts relative benefit, not Local errors. Both modes predict the correct top-1 token at 1,342 reference positions, yet Full increases its probability at 718 and decreases it at 437. Conversely, Full provides positive NLL gain at 345 Local-incorrect positions but corrects only 60 of them. Correctness and access benefit are therefore distinct even at a single decoding step. A separate paired analysis likewise finds that D KL ( p F ∥ p L ) D_{\mathrm{KL}}(p^{F}|p^{L}) correlates much more strongly with absolute than signed reference gain (Spearman 0.937 versus 0.138; Appendix B.7 ). These observations motivate evaluating recall by its relative benefit rather than identifying every prediction change, or every Local error, with a useful global read. 4 Related Work Dynamic local–global attention. Conditional computation allocates expensive attention and feed-forward branches to selected tokens, as in CoLT5 ( Ainslie et al., 2023 ) . For autoregressive decoding, task-specific fine-tuning can teach models to select attention spans ( Jin et al., 2024 ) , while AHA learns per-token choices between global and sliding-window attention for each head ( Luo et al., 2025 ) . L2A uses local attention outputs to gate a global branch within each layer, adapting attention and normalization parameters while freezing feed-forward layers ( Choudhary et al., 2026 ) . Switch Attention learns token- and layer-dependent choices between full and sliding-window attention through continued pretraining ( Zhao et al., 2026 ) , while LoGo learns coupled local–global branches with a controller for the global-attention budget ( Pan et al., 2026 ) . ODA makes a single recall decision after a complete Local forward pass. It learns this decision from paired prediction gains, using the existing states of a pretrained model whose weights remain unchanged. Concurrent Declarative Attention also explores model-directed control of context access ( Ho et al., 2026 ) . Selective context access. Another approach reduces the history participating in each attention operation. StreamingLLM retains initial and recent tokens, while H 2 O and SnapKV use attention statistics to select compact KV caches ( Xiao et al., 2024b ; Zhang et al., 2023 ; Li et al., 2024b ) . RefreshKV alternates full-context and subset attention, refreshing its compact cache after full-attention steps ( Xu et al., 2025 ) . QUEST, InfLLM, and RetrievalAttention select relevant pages, memory units, or indexed KV vectors from a larger retained history ( Tang et al., 2024 ; Xiao et al., 2024a ; Liu et al., 2025 ) . Learned selectors such as SeerAttention-R and SpotAttention also preserve pretrained weights, using attention supervision to select context blocks ( Gao et al., 2025 ; Ahmad and Yun, 2026 ) . ODA addresses the value of extending attention at a given decoding step: the recall head predicts whether Full improves on the current Local computation. Complete KV retention allows information skipped at one step to become accessible again at a later recall. Hybrid architectures and efficient attention execution. Attention costs can also be reduced through head specialization and architectural choices. DuoAttention assigns full-history access to retrieval heads and bounded caches to streaming heads ( Xiao et al., 2025 ) . At the layer level, NLL-guided selection uses calibration-time changes in answer-token loss to choose which layers retain full attention ( Tang et al., 2026 ) . Hybrid models combine recurrent computation with attention, as in Gated DeltaNet hybrids and Qwen3.5 ( Yang et al., 2025 ; Qwen Team, 2026 ) . ODA adds decisions across decoding steps to an existing architecture; the Qwen3.5 experiments demonstrate compatibility with its native hybrid design. At the implementation level, FlashAttention reduces memory traffic for exact attention ( Dao et al., 2022 ) . ODA’s conditional execution addresses how often global computation is performed, making the cost of both attention branches relevant to its realized speedup. Model-guided information acquisition. The model’s own predictions and representations can guide requests for additional information. FLARE uses low-confidence tokens in a draft sentence to trigger retrieval and regeneration ( Jiang et al., 2023 ) ; Self-RAG learns reflection tokens for retrieval and critique ( Asai et al., 2024 ) ; and Probing-RAG reads hidden states to decide whether further document retrieval is needed ( Baek et al., 2025 ) . Latent sufficiency probes predict when input processing can stop ( Xie et al., 2025 ) , and retrieval-benefit predictors assess whether external evidence improves an answer ( Dado et al., 2026 ) . LongPPL uses long–short context comparisons to identify tokens that are informative for evaluating long-context ability ( Fang et al., 2025 ) . ODA brings benefit prediction to access within the retained sequence: its supervision measures the signed improvement of Full over Local at a shared history, and its readout predicts that benefit before executing the current global computation. Adaptive execution and speculative decoding. CALM uses confidence to adapt decoding depth ( Schuster et al., 2022 ) , while language model cascades learn when to defer to a more expensive model ( Gupta et al., 2024 ) . Speculative decoding verifies inexpensive proposals with a target model, preserving its sampling distribution through an acceptance correction ( Leviathan et al., 2023 ) . Speculative cascades combine deferral with speculative execution ( Narasimhan et al., 2025 ) . ODA applies adaptive execution to two attention ranges of the same pretrained model. Its learned benefit estimate determines whether to accept the Local result or recompute the current step with Full, so global computation is incurred only when recall is selected. This mechanism yields an empirical quality–cost trade-off rather than a distribution-preserving verification rule. 5 Conclusion On-Demand Attention makes global attention conditional on its predicted benefit to the next prediction. A lightweight recall head reads the model’s available decoding states to guide this decision, leaving pretrained weights unchanged and the complete historical KV cache accessible. Across the evaluated Qwen and Gemma models, including hybrid backbones, selective recall recovers most of the quality lost under local attention while substantially reducing global-call frequency. Random-routing controls establish the importance of choosing when to recall, while FLOPs analysis and controlled vLLM measurements demonstrate computational savings and faster long-context decoding. Together, these findings show that pretrained decoding states can support both generation and attention allocation, enabling models to consult the information they retain according to its predicted value. AI Use Statement Generative AI assistance (OpenAI Codex) was used to refine the research framing and hypotheses, discuss experimental controls and cost models, interpret supplied results, review related literature, translate and edit the manuscript, and write plotting and record-checking scripts. AI-assisted local checks recomputed selected statistics from archived records and compiled and inspected the manuscript. They did not rerun the reported model training or benchmark evaluations. The authors are responsible for verifying the claims, citations, and artifacts in the submitted manuscript. Reproducibility Statement Algorithm 1 specifies the training update and decoding order; Appendix A details supervision, state handling, and cost assumptions. Appendix B records evaluation protocols, per-task scores, reference diagnostics, and the controlled runtime matrix. The manuscript repository contains archived diagnostic records, 170 task-level Random/ODA control runs, 16 input-ablation checkpoint summaries, and 260 runtime trials. The script scripts/build_experiment_assets.py checks their aggregates and rebuilds the experiment figures and tables from the archived inputs; source hashes and verification results accompany it. Additional aggregate results are stored separately and reproduced at their reported precision until task-level records are available. These checks reproduce statistics from recorded experiments, rather than rerunning model training or generation. Training-resource and runtime limitations are stated in Appendices B.9 and B.8 . A complete anonymous implementation release remains pending. References Ahmad and Yun (2026) H. Ahmad and S. Yun SpotAttention: plug-in block-sparse routing for pretrained long-context transformers . arXiv preprint arXiv:2606.22874 . External Links: Link Cited by: §4 . Ainslie et al. (2023) J. Ainslie, T. Lei, M. de Jong, S. Ontañón, S. Brahma, Y. Zemlyanskiy, D. Uthus, M. Guo, J. Lee-Thorp, Y. Tay, Y. Sung, and S. Sanghai CoLT5: faster long-range transformers with conditional computation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 5085–5100 . External Links: Document , Link Cited by: §4 . Anthropic (2026) Anthropic Anthropic Economic Index report: cadences . Note: June 26, 2026 External Links: Link Cited by: §1 . Asai et al. (2024) A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi Self-RAG: learning to retrieve, generate, and critique through self-reflection . In International Conference on Learning Representations , pp. 9112–9141 . External Links: Link Cited by: §4 . Baek et al. (2025) I. Baek, H. Chang, B. Kim, J. Lee, and H. Lee Probing-RAG: self-probing to guide language models in selective document retrieval . In Findings of the Association for Computational Linguistics: NAACL 2025 , pp. 3287–3304 . External Links: Document , Link Cited by: §4 . Bai et al. (2024) Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li LongBench: a bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 3119–3137 . External Links: Document , Link Cited by: §B.8 , §1 , §3.1 . Bai et al. (2025) Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, J. Tang, and J. Li LongBench v2: towards deeper understanding and reasoning on realistic long-context multitasks . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 3639–3664 . External Links: Document , Link Cited by: §B.8 . Choudhary et al. (2026) S. Choudhary, A. Chattopadhyay, L. Zancato, E. Nunez, M. Trager, W. Xia, and S. Soatto Learning when to attend: conditional memory access for long-context LLMs . arXiv preprint arXiv:2603.17484 . External Links: Link Cited by: §1 , §4 . Dado et al. (2026) O. Dado, D. Carmel, and O. Kurland Predicting the benefit of retrieval augmentation in open-domain question answering . In Proceedings of the 35th ACM International Conference on Information and Knowledge Management , External Links: Document , Link Cited by: §4 . Dao et al. (2022) T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré FlashAttention: fast and memory-efficient exact attention with IO-awareness . In Advances in Neural Information Processing Systems , Vol. 35 . External Links: Link Cited by: §4 . Fang et al. (2025) L. Fang, Y. Wang, Z. Liu, C. Zhang, S. Jegelka, J. Gao, B. Ding, and Y. Wang What is wrong with perplexity for long-context language modeling? . In International Conference on Learning Representations , pp. 94541–94563 . External Links: Link Cited by: §4 . Gao et al. (2025) Y. Gao, S. Guo, S. Cao, Y. Xia, Y. Cheng, L. Wang, L. Ma, Y. Sun, T. Ye, L. Dong, H. K. So, Y. Hua, T. Cao, F. Yang, and M. Yang SeerAttention-R: sparse attention adaptation for long reasoning . arXiv preprint arXiv:2506.08889 . External Links: Link Cited by: §4 . Gemma Team (2026) Gemma Team Gemma 4 technical report . arXiv preprint arXiv:2607.02770 . External Links: Link Cited by: §1 , §3.1 . Gupta et al. (2024) N. Gupta, H. Narasimhan, W. Jitkrittum, A. S. Rawat, A. K. Menon, and S. Kumar Language model cascades: token-level uncertainty and beyond . In International Conference on Learning Representations , pp. 4147–4180 . External Links: Link Cited by: §4 . Ho et al. (2026) N. Ho, H. Ahmad, W. Koh, S. Yun, T. Schuster, and C. Nogueira dos Santos Language models can control their own attention . arXiv preprint arXiv:2609.02737 . External Links: Link Cited by: §4 . Hsieh et al. (2024) C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg RULER: what’s the real context size of your long-context language models? . In First Conference on Language Modeling , External Links: Link Cited by: §1 , §3.1 . Huber (1964) P. J. Huber Robust estimation of a location parameter . The Annals of Mathematical Statistics 35 ( 1 ), pp. 73–101 . External Links: Document , Link Cited by: §2.2 . Jiang et al. (2023) Z. Jiang, F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig Active retrieval augmented generation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 7969–7992 . External Links: Document , Link Cited by: §4 . Jin et al. (2024) T. Jin, W. Yazar, Z. Xu, S. Sharify, and X. Wang Self-selected attention span for accelerating large language model inference . arXiv preprint arXiv:2404.09336 . External Links: Link Cited by: §4 . Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th Symposium on Operating Systems Principles , pp. 611–626 . External Links: Document , Link Cited by: §1 , §2.1 , §2.3 , §3.3 . Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from transformers via speculative decoding . In Proceedings of the 40th International Conference on Machine Learning , Proceedings of Machine Learning Research , Vol. 202 , pp. 19274–19286 . External Links: Link Cited by: §4 . Li et al. (2024a) J. Li, M. Wang, Z. Zheng, and M. Zhang LooGLE: can long-context language models understand long contexts? . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 16304–16333 . External Links: Document , Link Cited by: §B.8 . Li et al. (2024b) Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen SnapKV: LLM knows what you are looking for before generation . In Advances in Neural Information Processing Systems , Vol. 37 , pp. 22947–22970 . External Links: Link Cited by: §4 . Liu et al. (2025) D. Liu, M. Chen, B. Lu, H. Jiang, Z. Han, Q. Zhang, Q. Chen, C. Zhang, B. Ding, K. Zhang, C. Chen, F. Yang, Y. Yang, and L. Qiu RetrievalAttention: accelerating long-context LLM inference via vector retrieval . In Advances in Neural Information Processing Systems , Vol. 38 , pp. 54358–54385 . External Links: Link Cited by: §4 . Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization . In International Conference on Learning Representations , External Links: Link Cited by: §B.1 . Luo et al. (2025) X. Luo, K. Zhang, and X. Yan Learning when not to attend globally . arXiv preprint arXiv:2512.22562 . External Links: Link Cited by: §1 , §1 , §4 . Narasimhan et al. (2025) H. Narasimhan, W. Jitkrittum, A. S. Rawat, S. Kim, N. Gupta, A. K. Menon, and S. Kumar Faster cascades via speculative decoding . In International Conference on Learning Representations , pp. 44949–44987 . External Links: Link Cited by: §4 . Pan et al. (2026) Y. Pan, Z. Li, B. Tang, Z. Qin, and G. Li LoGo: token-level dynamic local-global attention . arXiv preprint arXiv:2608.29539 . External Links: Link Cited by: §1 , §4 . Peng et al. (2024) B. Peng, J. Quesnelle, H. Fan, and E. Shippole YaRN: efficient context window extension of large language models . In International Conference on Learning Representations , External Links: Link Cited by: §B.8 . Qwen Team (2025) Qwen Team Qwen3 technical report . arXiv preprint arXiv:2505.09388 . External Links: Link Cited by: §1 , §3.1 . Qwen Team (2026) Qwen Team Qwen3.5-2B . Note: Hugging Face model card External Links: Link Cited by: §1 , §3.1 , §4 . Schuster et al. (2022) T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, and D. Metzler Confident adaptive language modeling . In Advances in Neural Information Processing Systems , Vol. 35 , pp. 17456–17472 . External Links: Link Cited by: §4 . Shaham et al. (2023) U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy ZeroSCROLLS: a zero-shot benchmark for long text understanding . In Findings of the Association for Computational Linguistics: EMNLP 2023 , pp. 7977–7989 . External Links: Document , Link Cited by: §B.8 . Shazeer (2020) N. Shazeer GLU variants improve transformer . arXiv preprint arXiv:2002.05202 . External Links: Link Cited by: §A.2 . StepFun (2026) StepFun Step-3.5-Flash-SFT . Note: Hugging Face dataset External Links: Link Cited by: §B.1 . Tang et al. (2024) J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han QUEST: query-aware sparsity for efficient long-context LLM inference . In Proceedings of the 41st International Conference on Machine Learning , Proceedings of Machine Learning Research , Vol. 235 , pp. 47901–47911 . External Links: Link Cited by: §1 , §4 . Tang et al. (2026) Q. Tang, X. Hu, X. Liu, Y. Chen, and Y. Shao NLL-guided full-attention layer selection for training-free sliding-window adaptation . arXiv preprint arXiv:2606.27791 . External Links: Link Cited by: §4 . Vakilian et al. (2026) V. Vakilian, Z. Wang, A. S. Rawat, and C. Thrampoulidis Short-context dominance: how much local context natural language actually needs? . arXiv preprint arXiv:2512.08082 . External Links: Link Cited by: §1 . Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need . In Advances in Neural Information Processing Systems , Vol. 30 . External Links: Link Cited by: §2.1 . Wolf et al. (2020) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush Transformers: state-of-the-art natural language processing . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 38–45 . External Links: Document , Link Cited by: §B.1 . Xiao et al. (2024a) C. Xiao, P. Zhang, X. Han, G. Xiao, Y. Lin, Z. Zhang, Z. Liu, and M. Sun InfLLM: training-free long-context extrapolation for LLMs with an efficient context memory . In Advances in Neural Information Processing Systems , Vol. 37 , pp. 119638–119661 . External Links: Link Cited by: §4 . Xiao et al. (2025) G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han DuoAttention: efficient long-context LLM inference with retrieval and streaming heads . In International Conference on Learning Representations , External Links: Link Cited by: §4 . Xiao et al. (2024b) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks . In International Conference on Learning Representations , pp. 21875–21895 . External Links: Link Cited by: §2.1 , §4 . Xie et al. (2025) R. Xie, J. Wang, P. Rosu, C. Deng, B. Sun, Z. Lin, and B. Dhingra Knowing when to stop: efficient context processing via latent sufficiency signals . In Advances in Neural Information Processing Systems , Vol. 38 , pp. 41213–41240 . External Links: Document , Link Cited by: §4 . Xu et al. (2025) F. Xu, T. Goyal, and E. Choi RefreshKV: updating small KV cache during long-form generation . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 24878–24893 . External Links: Document , Link Cited by: §4 . Yang et al. (2026) P. Yang, C. Du, F. Zhang, H. Wang, T. Pang, C. Du, and B. An LongSpec: long-context lossless speculative decoding with efficient drafting and verification . In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1826–1844 . External Links: Document , Link Cited by: §1 . Yang et al. (2025) S. Yang, J. Kautz, and A. Hatamizadeh Gated delta networks: improving Mamba2 with delta rule . In International Conference on Learning Representations , External Links: Link Cited by: §3.2 , §4 . Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root mean square layer normalization . In Advances in Neural Information Processing Systems , Vol. 32 . External Links: Link Cited by: §A.2 . Zhang et al. (2023) Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen H 2 {}{2} O: heavy-hitter oracle for efficient generative inference of large language models . In Advances in Neural Information Processing Systems , Vol. 36 . External Links: Link Cited by: §4 . Zhao et al. (2026) Y. Zhao, H. Li, B. Wu, Y. Yin, L. Shang, J. Yuan, M. Zhang, and M. Zhang Switch attention: towards dynamic and fine-grained hybrid transformers . arXiv preprint arXiv:2603.26380 . External Links: Link Cited by: §1 , §4 . Appendix A Additional Method Details A.1 Access gain and the cost of routing errors At a fixed history and reference token, write ℓ t b = − log ⁡ p t b ​ ( x t + 1 ) \ell{t}^{b}=-\log p_{t}^{b}(x_{t+1}) for b ∈ { L , F } b\in{L,F} and g t = ℓ t L − ℓ t F g_{t}=\ell_{t}^{L}-\ell_{t}^{F} . Let a t ∈ { 0 , 1 } a_{t}\in{0,1} indicate whether Full is selected. The excess loss relative to the better current branch is ( 1 − a t ) ​ ℓ t L + a t ​ ℓ t F − min ⁡ ( ℓ t L , ℓ t F ) = ( 1 − a t ) ​ [ g t ] + + a t ​ [ − g t ] + , (1-a_{t})\ell_{t}^{L}+a_{t}\ell_{t}^{F}-\min(\ell_{t}^{L},\ell_{t}^{F})=(1-a_{t})[g_{t}]{+}+a{t}[-g_{t}]{+}, (10) where [ u ] + = max ⁡ ( u , 0 ) [u]{+}=\max(u,0) . Selecting the worse branch costs | g t | |g_{t}| , so routing-error counts alone do not measure the resulting NLL penalty. With the Full-call penalty λ \lambda , the same identity holds for penalized losses after replacing g t g_{t} by g t − λ g_{t}-\lambda . This is a comparison on fixed states: summing it does not give the loss of an alternative generated trajectory. Huber regression is a surrogate and does not directly minimize this excess loss. A.2 Recall-head architecture For Qwen3-1.7B, each of the three 2,048-dimensional inputs passes through an independent RMSNorm ( Zhang and Sennrich, 2019 ) , a biased → 1024 2048!\to!1024 projection, and SiLU. For tower outputs a , b , c ∈ ℝ 1024 a,b,c\in\mathbb{R}^{1024} , the interaction vector is u = [ a ; b ; c ; a ⊙ b ; | a − b | ; a ⊙ c ; | a − c | ; b ⊙ c ; | b − c | ] ∈ ℝ 9216 . u=[a;b;c;a\odot b;|a-b|;a\odot c;|a-c|;b\odot c;|b-c|]\in\mathbb{R}^{9216}. (11) A biased fusion projection gives z 0 = 1 2 ​ [ a + b + c + SiLU ⁡ ( W f ​ u + b f ) ] , z_{0}=\tfrac{1}{2}\bigl[a+b+c+\operatorname{SiLU}(W_{f}u+b_{f})\bigr], (12) with W f ∈ ℝ 1024 × 9216 W_{f}\in\mathbb{R}^{1024\times 9216} . Two pre-RMSNorm residual SwiGLU blocks ( Shazeer, 2020 ) follow, each with bias-free → 2048 1024!\to!2048 gate and up projections and a → 1024 2048!\to!1024 down projection. A final RMSNorm and biased scalar projection produce q t q_{t} . The head has 28,325,889 parameters and uses FP32 weights and arithmetic. The backbone, embedding, and vocabulary projection remain frozen. Head training uses neither dropout nor weight decay. A.3 Paired supervision and historical inputs Branch computation. The access sets are defined in Equation 3 , with s = 4 s=4 and w = 2048 w=2048 . They coincide through position 2,052; eligible supervision begins at position 2,053. Padding and packed-sequence boundaries prevent cross-sequence attention. For either branch, ( h t b , Δ ​ C t b ) = F ϕ b ​ ( x t , C t − 1 ) , b ∈ { L , F } , (h_{t}^{b},\Delta C_{t}^{b})=F_{\phi}^{b}(x_{t};C_{t-1}),\qquad b\in{L,F}, (13) where h t b h_{t}^{b} is the final normalized hidden state. The candidate update Δ ​ C t b \Delta C_{t}^{b} contains current-position KV at every attention layer and, for hybrid backbones, the native recurrent-state updates. Training computes a Full trunk and two current-position counterfactuals. Both counterfactuals read the trunk’s historical state, while propagating their own current-position activations through the layers. Past attention uses j < t j<t ; each candidate’s current KV contribution is merged separately by online softmax. Neither counterfactual changes the trunk history used at later positions. The Full counterfactual and trunk are mathematically equivalent in exact arithmetic, but their different BF16 reduction paths can change small gain signs. The reported implementation therefore uses the separate Full counterfactual for the Full NLL target. The frozen vocabulary projection supplies both reference-token NLLs. Vocabulary log-sum-exp is evaluated in chunks of 16 positions to avoid storing a sequence-by-vocabulary tensor. Eligibility requires valid current and next tokens, an unmasked reference label, consecutive position IDs within the same segment, and context length greater than 2,052. Backbone features and labels are computed without gradients. Previous hidden state during training. With eligibility indicator m t m_{t} , the preceding-state input is constructed from the selected training states h t train = { h t L , cf , m t = 1 ​ and ​ g t < τ hist , h t F , trunk , otherwise . h_{t}^{\mathrm{train}}=\begin{cases}h_{t}^{L,\mathrm{cf}},&m_{t}=1\ \text{and}\ g_{t}<\tau_{\mathrm{hist}},\ h_{t}^{F,\mathrm{trunk}},&\text{otherwise}.\end{cases} (14) These states are detached and shifted within each sequence, so the head at t t receives h t − 1 train h_{t-1}^{\mathrm{train}} , corresponding to H t − H^{-}{t} in Algorithm 1 (a). The input is zero at each segment start. The Full branch of this feature uses the trunk hidden state; the Full counterfactual is used for the gain target. This selection does not alter training KV. Because g t − 1 g_{t-1} is measured on the already observed x t x_{t} , it does not depend on the current reference target x t + 1 x_{t+1} . It nevertheless requires both preceding predictions, whereas deployment uses the branch actually selected by the head. Training and deployment therefore differ in both historical-state construction and preceding-state selection. Prefill handoff. For prompt x 1 : m x_{1:m} , Full prefill commits C m C_{m} , sets h m = h m F h_{m}=h_{m}^{F} , and produces the first output x m + 1 x_{m+1} . The first routed step consumes this token with ( C m , h m ) (C_{m},h_{m}) and produces x m + 2 x_{m+2} . Consequently, N N generated outputs contain N − 1 N-1 routed steps. All reported routing counts exclude the output supplied directly by prefill. A.4 Regression target and reduction For residual e = q t − y t e=q_{t}-y_{t} , the loss is ℓ Huber , 1 ​ ( e ) = { 1 2 ​ e 2 , | e | ≤ 1 , | e | − 1 2 , | e | > 1 . \ell{\mathrm{Huber},1}(e)=\begin{cases}\tfrac{1}{2}e^{2},&|e|\leq 1,\ |e|-\tfrac{1}{2},&|e|>1.\end{cases} (15) There is no gain-dependent weighting or auxiliary sign-classification loss. The primary Qwen3 configuration averages the distributed eligible-token means of its accumulated microbatches. The later input ablations and the primary Qwen3.5 configuration instead divide the summed losses by the total eligible-token count over the entire optimizer update. For accumulated microbatch sets ℰ j \mathcal{E}{j} pooled across data-parallel workers, these reductions are, respectively, ℒ micro \displaystyle\mathcal{L}{\mathrm{micro}} = 1 M ​ ∑ j = 1 M ∑ t ∈ ℰ j ℓ t | ℰ j | , \displaystyle=\frac{1}{M}\sum_{j=1}^{M}\frac{\sum_{t\in\mathcal{E}{j}}\ell{t}}{|\mathcal{E}{j}|}, (16) ℒ step \displaystyle\mathcal{L}{\mathrm{step}} = ∑ j = 1 M ∑ t ∈ ℰ j ℓ t ∑ j = 1 M | ℰ j | , \displaystyle=\frac{\sum_{j=1}^{M}\sum_{t\in\mathcal{E}{j}}\ell{t}}{\sum_{j=1}^{M}|\mathcal{E}{j}|}, shown for nonempty microbatches. They need not agree when eligible-token counts differ. In the step-normalized implementation, empty microbatches contribute zero and an entirely empty update is skipped. The training configurations are recorded in Appendix B.1 . The primary Qwen3 head uses λ = 0 \lambda=0 and τ hist = 0.001 \tau{\mathrm{hist}}=0.001 ; the primary Qwen3.5 head and input ablations use λ = 0.001 \lambda=0.001 and τ hist = 0 \tau_{\mathrm{hist}}=0 . The Full-call penalty is expressed on the NLL scale and is not a measured hardware cost. The history threshold and deployment score threshold are separate parameters. Although the signed-log transform is invertible, Huber regression on a transformed target does not make its inverse an estimate of the conditional mean raw gain. We use q t q_{t} as a regression score, without interpreting it as a probability or calibrated NLL benefit. A.5 Selected-state execution Let b t = F b_{t}=F when q t > θ q_{t}>\theta and b t = L b_{t}=L otherwise; nonfinite scores fall back to Full. The selected branch determines all committed outputs: h t \displaystyle h_{t} = h t b t , \displaystyle=h_{t}^{b_{t}}, p t \displaystyle p_{t} = softmax ⁡ ( W LM ​ h t ) , \displaystyle=\operatorname{softmax}(W_{\mathrm{LM}}h_{t}), (17) C t \displaystyle C_{t} = C t − 1 ∥ Δ ​ C t b t . \displaystyle=C_{t-1}\mathbin{|}\Delta C_{t}^{b_{t}}. The operator ∥ | appends the selected current KV and applies any recurrent-state updates. The same h t h_{t} becomes the preceding hidden input at the next step; it is updated after Local as well as Full selections. Candidate state remains uncommitted until the action is final. Only the selected hidden passes through the vocabulary projection at deployment. Full recall reads the history produced by ODA’s actual actions, without reconstructing an Always-Full trajectory. The optimized Qwen3 runtime maintains a paged complete KV cache and a compact Local view of the same selected history. With the window including the current token, Local holds at most 2,051 historical positions plus a current-candidate slot. Selection commits the same candidate to both views. A persistent workspace and fused all-layer commitment avoid copying the complete history. Local, the FP32 head, and conditional Full execute through CUDA Graphs with the decision kept on the GPU; the common engine output path projects and samples once. When Local and Full have identical access sets, the Transformers implementation can reuse the Local result for a logical Full action. The distinct-range cost model assumes n > s + w n>s+w . For Qwen3.5-2B, Local restricts only the six ordinary attention layers, at zero-based indices 3, 7, 11, 15, 19, and 23. The remaining 18 Gated DeltaNet layers retain their native computation. Before Local, the runtime saves convolution and recurrent states; before a Full retry, it restores the pre-step state. Only the selected updates are committed. Native precision uses BF16 backbone, KV, and convolution state, with FP32 recurrent state. These state transactions are necessary for recomputation from the same history; the Qwen3 timing results do not establish a speedup for this hybrid implementation. A.6 Cost of conditional recomputation Let B B be history-independent backbone cost, A ⁡ ( n ) A(n) attention cost at history length n n , R R recall-head cost, and O O the shared output cost. Let D F D_{F} and D ODA D_{\mathrm{ODA}} denote execution overheads. For a fixed Local range ℓ = s + w \ell=s+w and recall rate ρ \rho , an additive timing model is T F ​ ( n ) \displaystyle T_{F}(n) = B + A ⁡ ( n ) + O + D F , \displaystyle=B+A(n)+O+D_{F}, (18) T ODA ​ ( n , ρ ) \displaystyle T_{\mathrm{ODA}}(n,\rho) = B + A ⁡ ( ℓ ) + R + ρ ⁡ [ B + A ⁡ ( n ) ] + O + D ODA . \displaystyle=B+A(\ell)+R+\rho[B+A(n)]+O+D_{\mathrm{ODA}}. (19) The Local attempt is paid at every step, including recalled steps, while the output projection is paid only once. This model assumes no reused backbone prefix and comparable branch kernels; measured branch times can replace the shared B , A B,A approximation. Writing H = R + D ODA − D F H=R+D_{\mathrm{ODA}}-D_{F} , acceleration requires ρ < A ⁡ ( n ) − A ⁡ ( ℓ ) − H B + A ⁡ ( n ) . \rho<\frac{A(n)-A(\ell)-H}{B+A(n)}. (20) For A ⁡ ( n ) = a ​ n A(n)=an , a > 0 a>0 , and ρ < 1 \rho<1 , this becomes n > [ ρ ​ B + a ​ ℓ + H ] / [ ( 1 − ρ ) ​ a ] n>[\rho B+a\ell+H]/[(1-\rho)a] . The ratio approaches 1 / ρ 1/\rho only when global-attention cost dominates all other terms. Thus, a 12.5% recall rate does not imply an 8 × 8\times finite-context speedup, and a fixed positive recall rate retains O ⁡ ( n ) O(n) expected attention cost per decoding step. For an actual continuation, costs must be summed over its history lengths n t n_{t} and recall indicators a t a_{t} : T ODA = ∑ t [ B t + A ⁡ ( ℓ t ) + R t + a t ​ ( B t + A ⁡ ( n t ) ) + O t + D ODA , t ] . T_{\mathrm{ODA}}=\sum_{t}\bigl[B_{t}+A(\ell_{t})+R_{t}+a_{t}(B_{t}+A(n_{t}))+O_{t}+D_{\mathrm{ODA},t}\bigr]. (21) The timing of recall matters through the lengths n t n_{t} , not only its frequency. Similarly, total attended positions count both Local reads and any additional Full reads. Neither total traffic nor throughput can be inferred from 1 − ρ 1-\rho alone. The FLOPs measurements in Appendix B.8 use the actual model shapes and call schedules. Appendix B Additional Experiments and Evaluation Protocols B.1 Training and evaluation settings Data and optimization. Qwen recall heads are trained on model-tokenized versions of a content-deduplicated long-context SFT corpus derived from Step-3.5-Flash-SFT ( StepFun, 2026 ) . The training pool contains 196,608 examples with maximum length 16,384; the corresponding development and test splits each contain 8,192 examples. Pool size is not the number of distinct examples consumed at a particular checkpoint. The runs in Table 3 use a frozen BF16 backbone, an FP32 head, eight-way data parallelism, global batch size 64, AdamW ( Loshchilov and Hutter, 2019 ) , peak learning rate 3 × 10 − 4 3\times 10^{-4} , and gradient clipping at 1.0. Head dropout and weight decay are zero. Table 3 records differences between the primary runs and the input ablation. Table 3: Training configurations used in the main results and input ablation. The primary Qwen3 schedule has 102 warm-up updates followed by a plateau through update 3,072; the later schedules decay immediately after warm-up. The history threshold is separate from the Full-call penalty. Qwen3-1.7B main Qwen3.5-2B main Qwen3 input ablation Reported update 1,024 2,048 1,024 Microbatch / device 2 8 1 Accumulation steps 4 1 8 Seed 20260827 20260827 42 Full-call penalty λ \lambda 0 0.001 0.001 History threshold τ hist \tau_{\mathrm{hist}} 0.001 0 0 Loss reduction Microbatch mean Step-token mean Step-token mean LR schedule Warm-up / plateau Warm-up / cosine Warm-up / cosine The primary Qwen3 run uses a 6,144-update warm-up/plateau/cosine schedule and stops at update 3,072. The Qwen3.5 and input-ablation schedules warm up for 102 updates and decay to 3 × 10 − 5 3\times 10^{-5} at update 3,072. The Qwen3-8B and Gemma-4-12B-it results each use a separately trained step-1,024 head. These reported checkpoints are not claimed to have been chosen by an independent validation-based selection rule; later checkpoints are included below. Generation protocol. RULER16K uses 13 tasks with 100 examples each. LongBench v1 denotes the 13-task, 2,550-example subset listed in Table 5 . It uses the archived 40,960-token-budget protocol, rather than the full LongBench task collection. Scores follow each task’s official metric and are macro-averaged within a benchmark. The primary Qwen3-1.7B and Qwen3.5-2B configurations use greedy decoding with thinking disabled, Full prefill, a 512-token output cap, and four sinks plus a 2,048-token Local window including the current position. Each policy advances its own generated continuation and selected cache. Qwen3.5 applies Local only to its six ordinary attention layers and commits the sele

[... middle truncated for ingest size ...]

R16K, using Transformers SDPA, the same 13 × 100 13\times 100 population, and score threshold zero. The cost-adjusted target subtracts 0.001 from the gain. The globally normalized run also changes micro-batch size. Its merge averages head parameters from steps 1,024, 2,048, and 3,072. Rows represent separately trained checkpoints, not matched-budget comparisons. Δ \Delta is the score difference from Full. Method Score ↑ \uparrow ρ \rho ↓ \downarrow Δ \Delta vs. Full Original Full 94.35 100% 0.00 Always-Local 28.21 0% − 66.14 -66.14 ODA, v4-like, step 1,024 93.41 54.6455% − 0.94 -0.94 Cost-adjusted, step 1,024 92.08 25.6016% − 2.27 -2.27 Cost-adjusted, step 2,048 90.91 19.1125% − 3.44 -3.44 Cost-adjusted, step 3,072 93.87 39.9988% − 0.48 -0.48 Global normalization, step 1,024 92.15 21.7714% − 2.20 -2.20 Global normalization, step 2,048 94.08 44.3265% − 0.27 -0.27 Global normalization, step 3,072 93.48 34.5649% − 0.87 -0.87 Global normalization, uniform merge 93.47 30.2275% − 0.88 -0.88 The Qwen3 supervision controls in Table 13 keep the three-stream features and history-selection rule but change the training loss. Weighted BCE uses more Full calls; class balancing reduces calls and quality. Because the objectives have different score semantics, their natural thresholds do not define an equal-budget comparison. Table 13: Qwen3-1.7B supervision controls on RULER16K at step 1,024. The table reports each objective’s natural operating point. The operating points are not matched in access frequency. Objective Score ↑ \uparrow ρ \rho ↓ \downarrow Signed-log Huber 81.17 41.6415% | g | |g| -weighted BCE 81.44 88.6223% Class-balanced weighted BCE 74.61 45.4603% B.5 Fixed-reference Qwen3 diagnostics Collection. The primary Qwen3-1.7B step-1,024 head is evaluated with CPU FP32/SDPA. Sixteen test sequences are chosen by fixed content-hash order, four in each length bucket with edges 4,096, 6,144, 8,192, 10,240, and 16,385. Each contributes one contiguous block of 128 supervised reference positions beyond the Local window. Selection does not use predictions or gains; the supplied content audit finds none of these sequences in train or dev. After Full prefill, reference tokens advance ODA’s selected history. Both current candidates start from the same saved state, and only the branch chosen by the head is committed. The reference token is fixed independently of that action. Observer-disabled replay matches the actions, selected top-1 tokens, and final KV digests on all 16 sequences. Benefit ranking. Positive benefit is g t > 0 g_{t}>0 . Local-correct and Local-incorrect subsets are determined after evaluation from the Local top-1 prediction. They are not available to the policy. Table 14 compares the head with Local entropy. On the 675 Local-incorrect positions, the other scalar confidence rankings—negative maximum probability, negative probability margin, and negative top-two log-probability ratio— also have AUROCs between 0.486 and 0.491. Directions are fixed to rank lower-confidence positions first. Table 14: Primary Qwen3-1.7B router on fixed reference tokens, with positive benefit defined by g t > 0 g_{t}>0 . Intervals use 2,000 paired sequence-bootstrap replicates. Entropy ranks higher-uncertainty positions first. These diagnostics evaluate the complete existing router, not the incremental contribution of its Local hidden input. Subset Positions g t > 0 g_{t}>0 Router AUROC [95% CI] Entropy AUROC [95% CI] All 2,048 52.0% 0.556 [0.502, 0.610] 0.534 [0.455, 0.618] Local correct 1,373 52.4% 0.503 [0.445, 0.564] 0.593 [0.483, 0.689] Local incorrect 675 51.1% 0.643 [0.577, 0.707] 0.486 [0.442, 0.526] For offline budget b b , each method selects its top K = ⌊ b ​ | 𝒮 | ⌋ K=\lfloor b|\mathcal{S}|\rfloor positions within population 𝒮 \mathcal{S} . Net gain is | 𝒮 | − 1 ​ ∑ t ∈ 𝒜 g t |\mathcal{S}|^{-1}\sum_{t\in\mathcal{A}}g_{t} , including negative gains. Random’s expectation is ( K / | 𝒮 | ) (K/|\mathcal{S}|) times the population mean gain; Oracle ranks by realized gain and uses information unavailable online. At 40% of all 2,048 positions, K = 819 K=819 : ODA, Random, and Oracle capture 0.149978, 0.053385, and 0.227453 nats per position. The ODA–Random interval is [ 0.045747 , 0.160134 ] [0.045747,0.160134] . Intervals use 2,000 paired document-bootstrap resamples with seed 20260910, repeating selection within each resample. Table 15 separates positive gains from negative costs. Across all positions, the head exceeds entropy in 40%-budget net gain, with difference interval [ 0.0315 , 0.1388 ] [0.0315,0.1388] , although their AUROC and AP differences remain unresolved. Within Local-incorrect positions, the AUROC difference interval is [ 0.069 , 0.230 ] [0.069,0.230] and the net-gain difference interval is [ 0.1516 , 0.5010 ] [0.1516,0.5010] . Within Local-correct positions, both rankings have negative net gain. These are complete-head comparisons on fixed histories, rather than a confidence-controlled attribution to input L L . Table 15: Primary Qwen3 checkpoint at a 40% offline access budget. Positive gain and negative cost are sums in nats; net gain is per eligible reference position. Each subset is ranked independently, so its selected positions are not obtained by partitioning the all-position selection. Subset Ranking Calls Positive gain Negative cost Mean net gain [95% CI] All Router 819 378.486 71.331 0.1500 [0.0679, 0.2497] All Entropy 819 283.170 149.730 0.0652 [0.0114, 0.1265] Local correct Router 549 10.253 23.822 − 0.0099 -0.0099 [ − 0.0186 -0.0186 , − 0.0026 -0.0026 ] Local correct Entropy 549 19.336 40.529 − 0.0154 -0.0154 [ − 0.0284 -0.0284 , − 0.0046 -0.0046 ] Local incorrect Router 270 378.419 50.544 0.4857 [0.2503, 0.7118] Local incorrect Entropy 270 163.380 52.520 0.1642 [0.0663, 0.2647] Benefit is distinct from correctness. Of 2,048 positions, 1,342 are top-1 correct under both branches, 31 only under Local, 60 only under Full, and 615 under neither. Among the 1,342 both-correct positions, Full increases the reference probability at 718 and decreases it at 437; the remaining 187 gains are zero. Among the 675 Local-incorrect positions, 345 have positive gain but only 60 become top-1 correct under Full. Local can therefore have a better immediate prediction even though Full has lower aggregate NLL. Table 16 reports sign counts at several magnitude cutoffs. The cutoffs are descriptive, not validated numerical-noise bounds. Changing the positive label from g t > 0 g_{t}>0 to g t > 0.001 g_{t}>0.001 leaves the Local-incorrect AUROC and AP unchanged because that subset contains no gains in ( 0 , 0.001 ] (0,0.001] . The CPU precision and fixed-reference protocol are separate from the GPU free-generation evaluations. Table 16: Signed-gain sensitivity on the same 2,048 Qwen3 reference positions. Each row partitions the entire sample at the stated cutoff in nats. These counts describe magnitude and sign; they are not top-1 corrections or a rerun under another precision. ϵ \epsilon g t < − ϵ g_{t}<-\epsilon | g t | ≤ ϵ |g_{t}|\leq\epsilon g t > ϵ g_{t}>\epsilon 0 797 187 1,064 0.0001 649 632 767 0.001 594 788 666 0.01 503 1,012 533 0.1 313 1,419 316 B.6 Independent-reference Qwen3.5 pilot This CPU FP32 pilot uses the earlier Qwen3.5-2B micro8 step-3,072 head, which is distinct from the primary globally normalized step-2,048 head. Eight test sequences, two per length bucket, are chosen by a fixed content hash and contribute 64 reference positions each. The report records train/dev disjointness. Following Full prefill, the head advances the selected KV, convolution, and recurrent states under the reference tokens. Both candidates start from the same pre-step state. Observer-disabled replay matches actions, selected top-1 tokens, and final state digests. The available archive contains aggregate statistics; the original per-position tensors are not included. Positive labels, confidence directions, budget selection, and normalization follow the Qwen3 definitions above. Intervals use 2,000 paired sequence resamples, with only eight independent sequence clusters. Table 17 reports the independent test, rather than the older development panel collected under Full-trunk history. Table 17: Independent-reference Qwen3.5 pilot. Positive benefit is g t > 0 g_{t}>0 . Intervals are 95% paired sequence-bootstrap intervals. Subset Positions g t > 0 g_{t}>0 Head AUROC [95% CI] Entropy AUROC [95% CI] All 512 48.2% 0.612 [0.560, 0.652] 0.548 [0.479, 0.610] Local incorrect 123 58.5% 0.503 [0.380, 0.632] 0.411 [0.318, 0.503] On Local-incorrect positions, the head–entropy AUROC difference interval is [ − 0.0597 , 0.2662 ] [-0.0597,0.2662] : the pilot does not establish a discrimination advantage. At the prespecified 40% budget (49 of 123 positions), head and entropy net gains are 0.2788 and 0.0730 nats per position, with difference interval [ 0.0156 , 0.4392 ] [0.0156,0.4392] . Across all 512 positions, the corresponding gains are 0.0596 and 0.0584, with a difference interval spanning zero. Within Local-correct positions, both are negative ( − 0.0148 -0.0148 and − 0.0154 -0.0154 ). Thus, the favorable budgeted result on Local failures does not imply an overall or cross-model discrimination advantage. B.7 Access benefit and distribution disagreement A separate teacher-forced analysis compares Full and Local on Full-trunk histories over 2,048 Qwen3-1.7B sequences and 9,246,287 eligible assistant tokens. Targets are reference next tokens, independent of either branch’s prediction. Using natural logarithms over the full vocabulary, D KL ( p t F ∥ p t L ) D_{\mathrm{KL}}(p_{t}^{F}|p_{t}^{L}) has pooled token-level Spearman correlations 0.937029 with | g t | |g_{t}| and 0.137651 with g t g_{t} . This population is distinct from the 2,048-position head diagnostic. The observation concerns prediction-change magnitude versus signed reference benefit; weak signed rank correlation neither excludes nonlinear dependence nor measures gain-sign classification accuracy. For a fixed position, write g t ​ ( y ) = log ⁡ p t F ​ ( y ) − log ⁡ p t L ​ ( y ) g_{t}(y)=\log p_{t}^{F}(y)-\log p_{t}^{L}(y) . Then 𝔼 Y ∼ p t F ​ [ g t ​ ( Y ) ] \displaystyle\mathbb{E}{Y\sim p{t}^{F}}[g_{t}(Y)] = D KL ( p t F ∥ p t L ) , \displaystyle=D_{\mathrm{KL}}(p_{t}^{F}|p_{t}^{L}), (22) 𝔼 Y ∼ p t L ​ [ g t ​ ( Y ) ] \displaystyle\mathbb{E}{Y\sim p{t}^{L}}[g_{t}(Y)] = − D KL ( p t L ∥ p t F ) . \displaystyle=-D_{\mathrm{KL}}(p_{t}^{L}|p_{t}^{F}). (23) The target distribution therefore matters: KL can equal expected benefit under model-generated targets even when its relationship to reference-token gain differs. Exact current-step KL also requires executing Full, so it is an offline statistic, rather than a free online input when measuring avoided Full calls. B.8 Runtime protocol and complete results Environment and workload. The principal matrix uses vLLM 0.18, one A100-SXM4-80GB, CPU affinity 8–11, batch size one, and tensor/pipeline parallelism one. Backbone and KV are BF16; the recall head is FP32. All policies run on the same physical GPU. YaRN ( Peng et al., 2024 ) with factor five extends the capacity to 139,264 tokens for timing; these measurements do not establish model quality at those lengths. Each length uses one complete benchmark prompt, without repetition or concatenation: 4,139 tokens from ZeroSCROLLS Qasper ( Shaham et al., 2023 ) , 15,918 from LongBench v2 ( Bai et al., 2025 ) , 31,400 from LongBench v1 NarrativeQA ( Bai et al., 2024 ) , 63,852 from ZeroSCROLLS NarrativeQA, and 128,852 from LooGLE long-dependency summarization ( Li et al., 2024a ) . Within each bucket, all policies share the prompt and fixed 1,024-token continuation. The real head runs after every Local attempt, but a prescribed schedule controls 0, 64, 128, 256, 512, or 1,023 Full calls over 1,023 routed steps. ODA at 100% still pays for Local and the head; native Full performs neither and retains its own compilation and CUDA Graphs. Timing. Throughput is 1,023 / ( t 1024 − t 1 ) 1,023/(t_{1024}-t_{1}) , with device synchronization at the first and last output boundaries. It includes all layers, the head, conditional Full, state commitment, output projection, sampling, and the engine loop. It excludes loading, tokenization, prefill and its first output, detokenization, and HTTP transport. Numerical observers are off. Warm values are medians of three requests after warm-up and graph capture. First-request values use one request per arm in a fresh process and include graph captures encountered during decoding. The reported comparison uses the optimized device-execution arm; earlier implementation arms are retained in the source archive but not mixed into these results. Table 18: Warm and first-request decoding at 128/1,023 Full calls. TPS denotes tokens/s. Warm values are three-round medians; first-request values are single measurements including decode graph capture. Prefill is excluded in both cases. Warm decode First-request decode Input tokens Full TPS ODA TPS Speedup Full TPS ODA TPS Speedup 4,139 211.50 178.13 0.842 × 0.842\times 210.68 130.26 0.618 × 0.618\times 15,918 181.87 176.07 0.968 × 0.968\times 180.66 129.18 0.715 × 0.715\times 31,400 152.21 172.84 1.136 × 1.136\times 152.16 127.98 0.841 × 0.841\times 63,852 113.47 165.32 1.457 × 1.457\times 113.47 121.80 1.073 × 1.073\times 128,852 75.54 149.52 1.979 × 1.979\times 75.57 108.58 1.437 × 1.437\times Major-operation FLOPs. Using the notation of Appendix A.6 , with costs now counted in FLOPs, the ledger is F Full \displaystyle F_{\mathrm{Full}} = ∑ t [ B + A ⁡ ( n t ) + O ] , \displaystyle=\sum_{t}[B+A(n_{t})+O], (24) F ODA \displaystyle F_{\mathrm{ODA}} = ∑ t [ ( 1 + a t ) ​ B + A ⁡ ( ℓ t ) + a t ​ A ​ ( n t ) + R + O ] . \displaystyle=\sum_{t}[(1+a_{t})B+A(\ell_{t})+a_{t}A(n_{t})+R+O]. (25) The count includes MLP and QKV/output projections, both attention branches, the recall head, and one vocabulary projection. It uses actual model shapes, history lengths, and schedules. Attention FLOPs scale with query heads; KV bytes scale with KV heads. This is an analytical count of major operations, not measured hardware FLOPs or a latency prediction. At 128,852 input tokens and 128 Full calls, the computation ratio is 4.122 and the measured warm throughput ratio is 1.979. The full matrix retains slowdowns at short contexts and large recall budgets. Table 19: Complete 30-cell vLLM runtime and major-FLOPs matrix. Native Full is shared across the six schedules for each prompt. Warm TPS values are three-round medians; displayed Full rates are realized percentages. The 0% and 100% ODA schedules still execute Local and the recall head. Input tokens Full (%) Full TPS ODA TPS Speedup FLOPs saved (%) 4,139 0.00 211.50 194.43 0.919 × 0.919\times 11.97 4,139 6.26 211.50 185.75 0.878 × 0.878\times 6.58 4,139 12.51 211.50 178.13 0.842 × 0.842\times 1.19 4,139 25.02 211.50 164.28 0.777 × 0.777\times -9.59 4,139 50.05 211.50 142.05 0.672 × 0.672\times -31.16 4,139 100.00 211.50 111.90 0.529 × 0.529\times -74.23 15,918 0.00 181.87 196.01 1.078 × 1.078\times 44.96 15,918 6.26 181.87 185.32 1.019 × 1.019\times 39.24 15,918 12.51 181.87 176.07 0.968 × 0.968\times 33.53 15,918 25.02 181.87 159.71 0.878 × 0.878\times 22.10 15,918 50.05 181.87 135.17 0.743 × 0.743\times -0.76 15,918 100.00 181.87 102.70 0.565 × 0.565\times -46.41 31,400 0.00 152.21 197.18 1.295 × 1.295\times 63.12 31,400 6.26 152.21 184.49 1.212 × 1.212\times 57.23 31,400 12.51 152.21 172.84 1.136 × 1.136\times 51.34 31,400 25.02 152.21 153.93 1.011 × 1.011\times 39.55 31,400 50.05 152.21 126.62 0.832 × 0.832\times 15.97 31,400 100.00 152.21 93.11 0.612 × 0.612\times -31.09 63,852 0.00 113.47 196.86 1.735 × 1.735\times 78.20 63,852 6.26 113.47 180.25 1.589 × 1.589\times 72.16 63,852 12.51 113.47 165.32 1.457 × 1.457\times 66.12 63,852 25.02 113.47 141.78 1.249 × 1.249\times 54.03 63,852 50.05 113.47 110.64 0.975 × 0.975\times 29.87 63,852 100.00 113.47 77.06 0.679 × 0.679\times -18.38 128,852 0.00 75.54 193.85 2.566 × 2.566\times 88.02 128,852 6.26 75.54 168.59 2.232 × 2.232\times 81.88 128,852 12.51 75.54 149.52 1.979 × 1.979\times 75.74 128,852 25.02 75.54 121.61 1.610 × 1.610\times 63.46 128,852 50.05 75.54 88.38 1.170 × 1.170\times 38.91 128,852 100.00 75.54 57.29 0.758 × 0.758\times -10.10 Numerical checks and scope. Implementation checks compare the optimized path against a reference executing the same routing policy, including selected logits, actions, and state commitment. They do not test equivalence to the Always-Full output distribution. The measurements here isolate execution cost at controlled call schedules. Joint task quality and speed for the same learned policy, prefill-inclusive latency, peak memory, and multi-request batch scaling have not been measured. B.9 Training resources The Qwen3-1.7B recall head has 28,325,889 parameters, approximately 1.7% of the backbone’s nominal parameter count. This measures trainable capacity, not total training cost: the frozen Full trunk and both counterfactuals must still compute features and labels. Their online computation belongs in the checkpoint’s total training time and must not be counted again as a separate offline cost. Complete GPU-hours, peak memory, and the exact number of unique consumed examples and eligible targets are not yet available. We therefore do not claim a measured total-training-cost advantage over other methods. Appendix C Discussion and Limitations ODA predicts an immediate reference-token benefit, whereas each selected state also affects future predictions. Its training histories come from a Full trunk, while deployment follows the policy’s own choices. Evaluation under actual cache evolution is therefore essential to assessing this supervision. Complete KV retention preserves access to those states, but requires linear storage and does not reconstruct an Always-Full trajectory. Moreover, fewer global calls improve latency only when avoided computation offsets local computation, repeated backbone work, and control overhead. A broader direction is to learn a compact working set containing useful recent and distant information, and request additional history when its expected benefit justifies the cost. Stronger working sets could reduce recall frequency; lower-precision attention and tiered KV storage could reduce the cost of recall. Their value depends on selection, maintenance, transfer, and dequantization overheads as well as prediction quality. Joint training of representations and access policies may further improve these decisions. These extensions would move from deciding when to recall toward learning which information to keep readily accessible. Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" ( ) button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions . We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from