AgentFlareScholar · long context language models
Long-Context Language Models: Research Digest
10 foundational papers with 2,659 combined citations, plus the latest arXiv preprints.
AISA Scholar API · arXiv
You're reading the full report — free for humans and search engines.
The same content ships as a structured JSON dataset (normalized facts, sources and
citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open
HTTP 402 payment protocol.
crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify
What’s new on arXiv
Multi-State Linear Attention with Generalized Second-Order Routing
Hee Seung Hwang, Sanghyuk Chun, Xindi Wu, Olga Russakovsky, Zhiwei Deng
Self-attention plays a key role in long-sequence modeling, but its quadratic time complexity fundamentally limits the scaling of Large Language Models. Recently, Multi-State Linear Attention (MSLA) mechanisms have been proposed to make sequence modeling more efficient by splitting the context into segments and compressing each in its own memory state. MSLA aggregates the information extracted from each memory state using only the mean alignment between a query and each segment's keys, ignoring how the alignment is distributed within the segment. In order to model the relevance of a segment to a query more comprehensively and incorporate the variance of the alignment in the aggregation, we propose MERLIN (MomEnt Routing for multi-state LINear attention), a method that tracks the first and second moments of each segment's keys. Using the moments of keys, a query can recover both the mean and the variance of its alignment with each segment to perform a more content-aware aggregation of per-state readouts. Experiments on multi-state variants of Gated DeltaNet-1.3B and Mamba-2-780M show that MERLIN consistently outperforms existing aggregation methods on various long-context tasks. In particular, MERLIN outperforms baselines on RULER needle-in-a-haystack and long-context QA (+8.3% for Gated DeltaNet-1.3B and +7.0% for Mamba-2-780M) and diverse subtasks in LongBench (+13.5% and +12.0%). It even outperforms baselines that consume 4x more memory. MERLIN provides a simple and general routing mechanism for improving long-context capabilities of MSLA models.
DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models
Egor Maximov, Aleksei Merkelov, Vasiliy Alekseev, Ivan Krivorotov, Valentin Beliaev, Nikita Arsenin, Dmitry Bocharov, Aleksei Goncharov
Hybrid language models keep a key-value (KV) cache only in their few full-attention layers, but this cache is still the part of a prompt's state that grows with the prompt and that must be stored or moved to reuse it. We present DCKV, a differentiable codec for the KV cache and the first KV-cache codec that is trained end to end. Its transform, its quantizer and its entropy models, which are conditioned on the prompt tokens and on already decoded neighbouring positions, are optimized together against the frozen language model, both for the code length that the entropy models assign and for the change of the model's output. On Qwen3.5-2B, DCKV stores a compressed token in more than 55 times fewer bytes than bf16 at a loss of at most 1.8 points on three long-context benchmarks, and in more than 100 times fewer bytes at a loss of at most 3.6 points.
Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training
Yue Wu, Yang Zhang, Weinan Gan, Chongming Gao, Xinsen Guo, Xiaoyuan Li, Fuli Feng
Test-time training (TTT) enables a deployed language model to internalize incoming context on-the-fly into a small set of fast-weight parameters, rather than merely treating it as transient context for attention. In-Place TTT recently made this idea practical for large language models by repurposing the down-projection of standard MLP blocks as fast weights. However, we find that its benefit largely disappears on realistic web long-context question answering. We attribute this to In-Place TTT's uniform write schedule: every chunk is written with equal strength, so irrelevant content dilutes query-relevant information in fast-weight memory. Building on this insight, we propose Select-TTT, a model-agnostic, query-conditioned rule for In-Place TTT that scales each chunk's write strength by its relevance to the user query. This relevance signal requires no supervision and no new parameters: it is read directly from the model's frozen output head, as the probability mass that a chunk's hidden states assign to the query tokens. A temperature-scaled softmax turns these scores into learning-rate multipliers, leaving the chunk-wise, context-parallel computation of In-Place TTT intact. Select-TTT consistently improves long-context question answering on two benchmarks and two backbones. These findings highlight the value of query-conditioned fast weight writes for practical test-time training on long-context question answering, which is also potentially promising for long-horizon agent tasks involving multiple user queries over evolving contexts.
Hypergraph-Enhanced Evidence Selection for Long-Context Multimodal Document QA
Wenxin Zhao, Jiachuan Wang, Xinyi Zhu, Jiake Ge, Haoyang Li, Yongqi Zhang, Shuangyin Li, Lei Chen
Long-context multimodal document question answering (QA) has become an important yet challenging task for modern vision-language models (VLMs), as it requires reasoning over evidence dispersed across pages and modalities within limited input budgets. Current pipelines adopt retrieval-augmented generation (RAG) to select multimodal evidence for downstream VLMs. However, existing systems mainly rely on static selection rules over fixed-granularity retrieval units such as pages or blocks, resulting in information loss during retrieval and suboptimal performance in downstream reasoning. To alleviate these issues, we formulate the core task as budget-aware evidence selection over structured document evidence and propose HyperDoc, a document RAG framework that constructs compact yet sufficient evidence while preserving document relations. Specifically, HyperDoc constructs a document hypergraph to explicitly capture structural, logical, and cross-modal relationships among multimodal document elements. Over this structured evidence space, HyperDoc performs adaptive cost-aware hyperedge selection based on query intent and input budget. Extensive experiments on three representative benchmarks demonstrate that HyperDoc outperforms various types of state-of-the-art baselines.
Learning to Remember: Advances in Memory Mechanisms for Large Language Models
Chenye Han, Hongche Yu, Mengyue Wu, Dell Zhang, Jixiang Luo
Memory has become a central mechanism for extending large language models (LLMs) beyond fixed context windows, stateless interaction, and static parametric knowledge. This survey organizes the literature around three complementary dimensions: the life cycle of memory, the storage substrate, and the system role that memory plays in LLM-based agents, while using cognitive-science-inspired memory concepts as a lightweight conceptual motivation rather than as a separate technical contribution or implementation family. On this basis, we distinguish three mechanism families: nonparametric memory, where information is stored outside model parameters in text, graph, multimodal, or database-like structures; parametric memory, where information is stored or accessed through activations, key-value caches, learned modules, adapter-based parameterizations, or model weights; and memory for agent evolution, where stored experiences support reinforcement learning, continual learning, procedural reuse, context engineering, and agent harness design. Evaluation is treated as a separate cross-cutting component rather than as a memory mechanism. Unlike surveys that focus primarily on retrieval-augmented generation or long-context modeling, this paper emphasizes the full memory life cycle: construction, maintenance, retrieval, consolidation, forgetting, governance, and evaluation. By aligning conceptual motivations, computational memory mechanisms, benchmarks, and deployment concerns, the survey provides a structured roadmap for designing memory-augmented LLM agents that are persistent, adaptive, auditable, and safe to update over time.
Large Language Models: A Plain-Language Overview for Engineers
Bahaeddin Türkoğlu
Large language models (LLMs) are neural networks trained on huge text collections to generate text by predicting what comes next. Engineers using ChatGPT and other generative artificial intelligence (AI) tools need a clear picture of what training changes and prompts add. The article starts with the Transformer, which connects words in context, and pre-training, where models learn to predict text. It then explains how fine-tuning adapts models to requests, with reinforcement learning from human feedback using human answer rankings. Next, retrieval-augmented generation (RAG) supplies documents, while chain-of-thought prompting encourages intermediate steps and agents use tools. FlashAttention lowers training and running costs by reducing data movement during attention; quantized low-rank adaptation (QLoRA) saves fine-tuning memory. Readers get a life cycle figure and tables comparing methods and their limits. A short checklist for judging LLM evaluation claims flags test material in training (data contamination) and model-based grading (LLM-as-a-judge). It also covers very long inputs (long context) and hallucination, meaning false or unsupported claims. Check factual accuracy separately from whether an answer sounds helpful.
Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction
Zhe Xu, Tianhao Liao, Zhenqing Ling, Ying Shen, Yaliang Li, Daoyuan Chen
Long-context language models can increasingly process entire documents, making full-context prompting a practical alternative to passage retrieval. However, greater capacity determines how much information a model can receive, but not which information best supports a query. This distinction motivates rethinking retrieval: when the full document already fits, can selecting and organizing its evidence improve answering? The challenge is to retain supporting facts and relations while keeping the context focused on the question. To reconcile these goals, we introduce ours, a training-free framework that uses semantic abstraction to guide selection while retaining original text for answering. ours constructs a reusable semantic hierarchy from the full document in a single generation, linking node descriptions to source spans. Given a query, structural--semantic scoring and fine-grained lexical attribution nominate shared source nodes; their union with ancestor context preserves structural relations and complete local text. Across four frontier answer models, ours improves average QA-F1 by 1.72 points over full-context prompting on three LongBench tasks, surpassing the evaluated baselines on the four-model average while reducing answer-context length by 58.55\%. Our code is available at the hrefhttps://anonymous.4open.science/r/treefuse-ECFErepository.
Every Edge Needs a Quote: Building a Quote-Grounded Knowledge Graph of Healthcare Software Adoption with Language-Model Agents
Juan Felipe Duran, Arthur Tobler, Martin Vallières
Hospitals regularly buy software that clinicians and patients depend on. Because these purchases are costly, hard to reverse and can affect patient care, buyers look for evidence of which peer institutions already use a product and which systems it connects to. This evidence is public but scattered across vendor websites, press releases and third-party pages. Language-model agents can now collect it at low cost, but the correctness of the extracted claims is difficult to establish, and a wrongly listed customer can mislead buyers and misrepresent an organisation. We present a pipeline that builds a knowledge graph of customer and integration relations from the open web under one rule: every edge must be supported by a verbatim quote from its source page. Pages come from a deterministic crawl of each vendor's site and a budgeted ReAct agent that searches the rest of the web. A permissive extractor proposes relations with quotes, and a layered grounding stage decides which to keep, using a referent step that reads the whole page when a quote names only one side, deterministic checks, and a panel of five model calls voting on nine criteria. On 61 vendors, the pipeline reaches 0.94 precision and 0.71 recall against a page-level key covering the 6,428 candidate relations proposed by its extractor, whose verdicts were drafted by one model and re-checked with a second. Against the golden reference, the marketplace's own curated list, it scores lower, but 89% of the relations it keeps that the reference lacks are stated on a page. A confidence score built from the panel's votes keeps more correct relations than the fixed rule at the same precision, and reaches 0.90 precision at 0.90 recall. We also discuss how narrow, bounded model calls, string-matched evidence and voting address long-context degradation and run-to-run variance.
Foundational papers
- 2024
- 2024
- 2024
- 2024
- 2024
- 2025
- 2025
- 2025
- 2026
- 2026
- 2026-10
How to train long-context language models (effectively)
T Gao, A Wettig, H Yen, D Chen
… In this section, we study how to best enable longcontext language models to follow instructions. We focus on supervised fine-tuning on instruction datasets (Ouyang et al.…
Longalign: A recipe for long context alignment of large language models
Y Bai, X Lv, J Zhang, Y He, J Qi, L Hou et al.
… LongAlign-7B-64k: This paper presents LongAlign, a recipe for aligning large language models to handle long contexts. The key challenges in long context scaling include lack of a …
Inftythink: Breaking the length limits of long-context reasoning in large language models
Y Yan, Y Shen, Y Liu, J Jiang, M Zhang et al.
… language models has achieved remarkable performance on challenging tasks, but the prevailing long-context … context boundaries, and performance degradation beyond pre-training …
Training-free long-context scaling of large language models
C An, F Huang, J Zhang, S Gong, X Qiu, C Zhou et al.
… models with longer sequences, we propose Dual Chunk Attention (DCA), which enables LLAMA2 70B to support context … By decomposing the attention computation for long sequences …
Spell: Self-play reinforcement learning for evolving long-context language models
Z Yang, W Shen, C Li, R Chen, F Wan et al.
… Evaluation Benchmarks We evaluate our models on six long-context benchmarks, spanning multiple-choice QA on LongBench-V2 (Bai et al., 2025) and multi-hop QA across Frames (…
RULER: What's the real context size of your long-context language models?
CP Hsieh, S Sun, S Kriman, S Acharya et al.
… of long-context understanding. In this work, we propose RULER, a new benchmark to evaluate long-context modeling capabilities for language models. … simple retrieval from context: …
Longrecipe: Recipe for efficient long context generalization in large language models
Z Hu, Y Liu, J Zhao, S Wang, Y Wang et al.
… large language model H with a context window size L. This model is further trained to extend its context … Ruler: What’s the real context size of your long-context language models? arXiv …
Nolima: Long-context evaluation beyond literal matching
A Modarressi, H Deilamsalehy, F Dernoncourt et al.
… This paper presents work aimed at advancing the field of long-context language modeling by evaluating and analyzing the most commonly used LLMs. There are many potential …
L-eval: Instituting standardized evaluation for long context language models
C An, S Gong, M Zhong, X Zhao, M Li et al.
… In L-Eval, we argue that long context language models should further focus on more accurate content rather than accurate length. Practically, issues about undesired generation length …
M4LE: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models
WC Kwan, X Zeng, Y Wang, Y Sun, L Li et al.
… Managing long sequences has become an important and necessary feature for large language models (LLMs). However, assessing their ability to handle long contexts remains a …
How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.
Sources & citations
- How to train long-context language models (effectively) 165 citations
- Longalign: A recipe for long context alignment of large language models 188 citations
- Inftythink: Breaking the length limits of long-context reasoning in large language models 74 citations
- Training-free long-context scaling of large language models 103 citations
- Spell: Self-play reinforcement learning for evolving long-context language models 34 citations
- RULER: What's the real context size of your long-context language models? 1,552 citations
- Longrecipe: Recipe for efficient long context generalization in large language models 32 citations
- Nolima: Long-context evaluation beyond literal matching 155 citations
- L-eval: Instituting standardized evaluation for long context language models 315 citations
- M4LE: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models 41 citations
- Multi-State Linear Attention with Generalized Second-Order Routing arXiv · 2026-10-27
- DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models arXiv · 2026-10-27
- Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training arXiv · 2026-10-27
- Hypergraph-Enhanced Evidence Selection for Long-Context Multimodal Document QA arXiv · 2026-10-26
- Learning to Remember: Advances in Memory Mechanisms for Large Language Models arXiv · 2026-10-26
- Large Language Models: A Plain-Language Overview for Engineers arXiv · 2026-10-25
- Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction arXiv · 2026-10-25
- Every Edge Needs a Quote: Building a Quote-Grounded Knowledge Graph of Healthcare Software Adoption with Language-Model Agents arXiv · 2026-10-23