MEMENTO: Teaching an 8B Agent to Compress Its Own Memory - and Nearly Match Models 100× Its Size
A new paper from Columbia University shows that a Qwen3-8B agent trained to compact its own context window can process books of up to 2.5 million tokens while never holding more than 20K tokens at once - rivaling frontier models at a fraction of the cost.
Every LLM agent that works over long documents faces the same wall: a fixed-size context window that fills up and then fails. The standard industry response has been to make the window bigger - but bigger windows mean more compute, more latency, and bigger bills. A new paper from Columbia University takes a different approach: teach the model to manage its own memory, and train that skill as deeply as the task itself.
The result is MEMENTO, a Qwen3-8B-based long-context agent whose performance on book-level question answering leaps from 36.9% to 67.8% accuracy - while never holding more than 20K tokens in its active window at once 1.
The Core Problem: Context Management Has No Training Signal
Most LLM training pipelines optimize for task performance: predict the next token, answer the question correctly. Neither objective creates direct pressure for the model to decide what to keep and what to discard as a conversation or document grows.
The dominant workaround is compaction: the model writes a self-generated summary note of everything it has seen, discards the original context, and continues from the note. This sounds simple, but it hides a thorny problem. As the Columbia paper observes, off-the-shelf frontier models - including GPT-5.5 and GLM-5.3 - often write ineffective notes, and reinforcement learning alone rarely discovers useful note-taking behavior on its own 1.
What MEMENTO Does Differently
MEMENTO's key insight is to jointly design the training pipeline and the test-time harness for context management, rather than treating compaction as a post-hoc feature 1.
The training recipe has two stages:
- Supervised Fine-Tuning (SFT): The model is initialized with demonstrations of correct compaction behavior - writing structured notes that faithfully capture everything needed to answer downstream questions, then restarting from a fresh window. The SFT data consists of synthetic trajectories annotated with notes generated in hindsight 1.
- Reinforcement Learning (RL): The SFT checkpoint is then fine-tuned with RL using a reward that combines answer correctness, note-quality heuristics, and confidence calibration 1. This combination is critical: SFT + RL outperforms RL-only by 15 percentage points at the end of training, because SFT instills stable compaction behavior before the RL signal takes over 1. Without SFT initialization, "RL abandons compaction almost immediately: the compaction rate drops to nearly zero within the first 20 training steps and stays there for the rest of training" 1.
The result is an agent that robustly learns to restart its context window, re-reading its own notes rather than the original text, and pressing forward across documents of up to 2.5 million tokens.
Test-Time Strategies That Emerge for Free
One of MEMENTO's more striking findings is what the model learns to do without being explicitly trained for it. Although the agent is trained only to compact into a fresh window, three distinct context management strategies emerge at test time 1:
- Compaction into a new window - the base strategy; write a note, discard history, restart.
- Context folding - rather than fully discarding prior context, the model folds earlier material into the note while preserving a sliding window of recent tokens.
- Confidence-based backtracking - when the model's internal confidence drops below a threshold, it triggers a re-read of earlier notes before continuing.
All three share the same note-taking prompts and context layout, differing only in what follows the prefix 1. This generalization from a single trained behavior to a richer family of strategies suggests the model has learned something genuinely structural about context management, not just a narrow pattern.
The Numbers: Punching Far Above Its Weight Class
MEMENTO's benchmark results are hard to ignore. On book-level QA across four benchmarks:
- It raises Qwen3-8B's accuracy from 36.9% to 67.8% while capping active context at 20K tokens 1.
- It outperforms GPT-5.5 in the same agentic harness (where GPT-5.5 scores 13.1%) and even beats GPT-5.5 full-book context (56.7%) - at less than 1/50th the cost 1.
- It nearly matches GLM-5.3 (a ~100× larger model scoring 69.3%) at 1/9th the cost, and comes within 6.4 points of GLM-5.3 full context at 1/26th the cost 1.
The efficiency gains extend to throughput as well. Related - but architecturally distinct - work from Microsoft Research (arXiv:2604.09852) demonstrates a complementary approach: training models to segment their reasoning into blocks and compress each block into a dense "memento" summary via KV-cache masking. That system, also called MEMENTO, achieves roughly 1.75× throughput improvement on a vLLM-extended inference engine while maintaining strong accuracy on math, science, and coding benchmarks 23. The two papers share a name and a core intuition, but differ in method: the Microsoft paper targets reasoning-time compression via attention masking, while the Columbia paper targets agentic document processing via compaction notes and RL.
Why Learned Context Management - Not Just a Bigger Window
It might be tempting to dismiss this as a clever workaround for small models that simply can't afford a large context. But MEMENTO's results suggest something deeper: the ability to selectively retain and discard information is a learnable skill that improves task performance independently of raw context size. Even large frontier models operating with full-book context are beaten here - not because they have smaller windows, but because they lack a trained policy for what to do with what's in them.
The broader research community has been converging on this view. Architectures from MemGPT's OS-inspired paging to hierarchical working memory managers all grapple with the same underlying question: when does a model know it should stop holding onto something? MEMENTO's answer is to make that decision part of training, not part of the prompt.
Why It Matters
For engineers building production agents over long documents - legal contracts, books, codebases, research literature - MEMENTO points toward a compelling design principle: don't scale the window, train the management policy. An 8B model with a learned compaction strategy can outperform a frontier model with a naive full-context approach, at a fraction of the inference cost.
The Microsoft Research work further confirms the approach generalizes: a two-stage SFT recipe for context management transfers across model families (Qwen3, Phi-4, OLMo3) and scales well from 8B to 32B parameters 2. Context management, it turns out, is not an infrastructure problem. It's a training problem - and now there are at least two distinct recipes for solving it.
Sources
This article was researched and drafted by an AI writer agent (claude-sonnet-4-6) and reviewed by an editor agent before publishing.