Paper TL;DR · Avinash Gosavi

Paper TL;DR: RAG — The Paper That Named Retrieval-Augmented Generation

2026-10-20T15:45:00+00:00

Paper: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel & Douwe Kiela (Facebook AI Research, University College London, New York University), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, 2020; revised 2021)

One-line TL;DR: Instead of hoping a language model memorized the right fact, give it a search step: look up a few relevant Wikipedia passages, generate the answer from them, and train the search and the writer together.

The question

By 2020, large language models clearly stored a lot of facts in their weights. But the paper points out the downsides: that knowledge is hard to expand or revise, the model can’t show where an answer came from, and it can hallucinate. Earlier hybrid models that paired a language model with a retriever (REALM and ORQA) had only been tried on extractive question answering — pulling answer spans out of passages rather than writing new text. This paper asks whether one general recipe can combine the two: a model that retrieves documents and generates free-form answers, fine-tuned end to end for many different tasks.

How it works

The paper splits knowledge into two kinds of memory:

Parametric memory — what’s baked into the model’s weights. Here that’s BART-large, a pre-trained sequence-to-sequence generator.
Non-parametric memory — an external index you can search. Here that’s a December 2018 Wikipedia dump cut into 100-word chunks (about 21 million of them), each turned into an embedding by a BERT document encoder from DPR (Dense Passage Retrieval). The vectors live in a FAISS index using an HNSW approximation, so lookups stay fast.

At question time, a second BERT encoder turns the input into a query vector, the index returns the top-k passages, and BART generates an answer conditioned on the input plus each passage. Nobody labels which passage is “right”: the retriever learns only from whether the final answer comes out well.

Two variants decide how the passages get combined:

RAG-Sequence uses the same passage for the whole answer, then weighs the answers it would give for each passage.
RAG-Token can lean on a different passage for each word, so one answer can stitch together facts from several documents.

One practical shortcut: only the query encoder and the generator are trained. The document encoder — and so the whole 21-million-vector index — stays frozen, because re-embedding Wikipedia during training is expensive. The authors report they didn’t need that step for strong results.

Only the query encoder and the generator are trained. The index stays frozen, so swapping it for an older Wikipedia dump changes what the model knows.

What they found

Strong open-domain question answering. On Natural Questions, WebQuestions and CuratedTrec, RAG beat the previous best systems in the paper’s comparison, while writing answers instead of copying spans. On TriviaQA the picture is mixed: it led on one test split but not the standard one.
It can answer even when the passage doesn’t spell it out. On Natural Questions, RAG still got about 11.8% of answers right in cases where the exact answer wasn’t in any retrieved passage — something a pure extractor can’t do.
More factual, more specific text. In a human evaluation of generated Jeopardy-style questions, raters judged RAG more factual than plain BART far more often than the other way round (42.7% vs. 7.1% of cases), and more specific too.
Small next to giant models. RAG has 626M trainable parameters; the largest closed-book model it’s compared to, T5-11B, has 11 billion — yet RAG scored higher on open-domain QA.
Swap the index, update the knowledge. The authors built a second index from a 2016 Wikipedia dump and asked “Who is {position}?” about 82 world leaders who changed between 2016 and 2018. With the matching index RAG got roughly 70% right; with the mismatched one, 4–12%. Changing what the model knows meant replacing the index, not retraining.

Why it matters for builders

It’s the template most “chat with your docs” systems still follow. Embed chunks, search by meaning, paste the top results into the prompt, generate. Agent memory systems do the same thing with memories instead of Wikipedia pages — the glossary’s “Storing and finding memories” section walks through each piece.
Updating knowledge is a data problem, not a training problem. The hot-swap experiment is the cleanest argument for keeping facts outside the weights: if something changes, you change the index. The same idea is why memory systems store facts with timestamps and versions rather than hoping the model “knows” the latest.
Provenance comes for free. Because the answer is conditioned on specific passages, you can show where it came from — useful for debugging and for trust.
Retrieval quality still caps everything. The retriever here is learned only through the final answer, and the paper reports it can fail badly (see caveats). Later work in this series picks up from there: Lost in the Middle shows that even good passages help less when they land mid-prompt, and MemGPT treats the context window as something to manage rather than fill.

Honest caveats

This is 2020-era plumbing. “RAG” today usually means an instruction-tuned LLM plus an off-the-shelf retriever, with no joint training. The paper’s specific recipe — BART, DPR, a frozen Wikipedia index — is mostly of historical interest now; the idea is what stuck.
Retrieval can collapse. In early story-generation experiments, the retriever learned to return the same documents no matter the input, after which the generator ignored them and RAG behaved just like plain BART. The authors suggest tasks with long outputs or less need for facts may give the retriever too weak a signal.
Not state of the art everywhere. On FEVER fact verification, RAG came within about 4.3% of the best pipeline (3-way) without being given gold evidence — close, but not ahead.
The index is only as good as its source. The authors note that Wikipedia will likely never be fully factual or free of bias, and that the same misuse concerns as other text generators apply.
Big infrastructure for its time. The paper notes the Wikipedia index initially needed about 100 GB of CPU memory, cut to 36 GB with FAISS compression.

Earlier in the series: Sentence-BERT (the kind of embeddings that make the search step work), Lost in the Middle (where retrieved passages land in the prompt) and MemGPT (an agent that pages its own memory in and out).

Terms like retrieval-augmented generation, embedding, semantic search, top-k retrieval and HNSW are defined in the agent memory glossary.

Paper TL;DR: Lost in the Middle — Where You Put Context Matters

2026-10-20T10:45:00+00:00

Paper: Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni & Percy Liang (Stanford, UC Berkeley, Samaya AI), Lost in the Middle: How Language Models Use Long Contexts (arXiv, 2023)

One-line TL;DR: Give a model a long prompt with one useful passage in it, and how well it answers depends on where that passage sits: best at the very start or very end, noticeably worse in the middle — even for models built for long contexts.

The question

Retrieval-augmented generation and agent memory both work the same way at the last step: fetch some passages, paste them into the prompt, and hope the model reads them. Bigger context windows make it tempting to paste in more. This paper asks a simple question first: does the model actually use everything in a long prompt equally well?

How they tested it

Two tasks, each designed so the authors could move the one relevant piece of information around while keeping everything else the same.

Multi-document question answering. Questions from NaturalQuestions-Open, each paired with one Wikipedia passage that contains the answer plus distractor passages that look relevant but don’t. Prompts held 10, 20 or 30 passages, and the answer passage was moved from first to last.
Key-value retrieval. A synthetic stress test: a JSON object of random UUID keys and values (75, 140 or 300 pairs), and the model has to return the value for one key. No language cues, just “find this and copy it.”

They ran open models (MPT-30B-Instruct, LongChat-13B) and API models (GPT-3.5-Turbo, Claude-1.3, and their longer-context versions, plus GPT-4 on a subset).

What they found

A U-shaped curve. Accuracy was highest when the answer passage came at the very start or the very end, and dropped when it sat in the middle — by more than 20% in some settings. The authors describe this as a primacy bias plus a recency bias.
The middle can be worse than nothing. In one 20-document setting, GPT-3.5-Turbo with the answer buried mid-prompt scored lower than the same model answering with no documents at all.
A longer window didn’t fix it. When a prompt fit in both a model and its extended-context version, the two scored almost the same. A bigger window lets you paste more; it doesn’t make the model read it better.
More retrieved documents stop helping early. In an open-domain QA case study, the retriever kept finding the answer more often as it returned more passages, but the model’s accuracy flattened out well before that. Going from 20 to 50 passages bought only a point or two, at much higher cost and latency.
Asking the question twice helps for lookups. Putting the query both before and after the data made key-value retrieval near-perfect, but it barely changed the multi-document QA results.

A sketch of the shape the paper reports, not its numbers: the same passage helps most at the edges of the prompt and least in the middle.

Why it matters for builders

Order is part of retrieval. Ranking isn’t just about which passages make the cut but where they land. Putting the best matches first (and not burying them under weaker ones) is cheap and directly addresses this. A re-ranking pass earns its keep here.
Fewer, better passages beat more passages. The case study suggests a tighter top-k can be nearly as accurate and a lot cheaper. It also connects to the idea of sufficient evidence: the goal is enough context, not all of it.
Test position, not just recall. A memory system can retrieve the right fact and still fail if the fact lands mid-prompt. Benchmarks like LoCoMo measure the end answer; this paper’s trick of moving the gold passage around is an easy extra check to run on your own pipeline.
Keep the important stuff at the edges. Designs like MemGPT keep a small, always-visible working context near the top of the prompt. This paper is one reason that kind of layout makes sense.

Honest caveats

These are 2023 models. The results cover GPT-3.5, Claude-1.3, MPT and LongChat, with only partial GPT-4 runs because of cost. Newer models may show a weaker (or different) position effect, so it’s worth measuring on whatever you actually use.
The “why” is still open. The authors test a few explanations — architecture, instruction tuning, model size — and call those analyses preliminary. Encoder-decoder models were fairly robust within their training length but showed the U-shape on longer inputs; base models without instruction tuning showed it too.
One answer per prompt. Each test has a single relevant passage. Real questions often need several facts combined, which this setup doesn’t measure.
Greedy decoding only, and the key-value task is deliberately artificial, so it shows a capability gap rather than typical real-world behavior.

Earlier in the series: MemGPT (paging what goes into the context window), Generative Agents (ranking memories by recency, importance and relevance) and Sentence-BERT (the embeddings behind most of the retrieval step). Next in the series: RAG, the 2020 paper that named the retrieve-then-generate setup this paper stress-tests.

Terms like context window, position bias, re-ranking and top-k retrieval are defined in the agent memory glossary.

Paper TL;DR: Generative Agents — Memory Streams, Reflection and Plans

2026-10-19T21:40:00+00:00

Paper: Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang & Michael S. Bernstein (Stanford, Google Research, Google DeepMind), Generative Agents: Interactive Simulacra of Human Behavior (UIST 2023)

One-line TL;DR: Give an LLM a running log of everything it observes, a way to score which memories matter right now, and a habit of periodically writing down higher-level conclusions — and simulated characters start to behave believably over days, not just single replies.

The setup

The authors built Smallville, a small sandbox town populated by 25 characters. Each one is driven by an LLM (gpt-3.5-turbo in the paper) and starts from a single paragraph describing who they are: job, relationships, habits. From there the characters wake up, go to work, chat, form opinions of each other and make plans — all generated, step by step, by the model.

The interesting part isn’t the town. It’s the memory design that keeps each character consistent over time, which the paper splits into three pieces: a memory stream, reflection and planning.

1. The memory stream

Every character keeps a memory stream: a plain list of memories, each written in natural language with a creation time and a “last accessed” time. The basic memories are observations — things the character perceived, such as (to give an illustrative example) “a neighbor is watering the garden.”

The stream grows far too large to fit into a prompt, so each time a character needs to act, it retrieves only the most useful memories. Each memory gets three scores:

Recency: recently used memories score higher, fading exponentially with each in-game hour (the paper uses a decay factor of 0.995).
Importance: when a memory is created, the model rates it from 1 (mundane, like cleaning a room) to 10 (poignant, like asking someone out).
Relevance: cosine similarity between the memory’s embedding and the current situation’s embedding.

Each score is scaled to 0–1 and the three are added up (the paper weights them equally). The top-scoring memories that fit into the context window go into the prompt.

2. Reflection

Raw observations are too low-level to generalize from. Fifty separate observations of a neighbor reading in the library don’t, by themselves, tell a character “she cares a lot about her research.”

So characters reflect. When the summed importance of recent memories passes a threshold (150 in the paper — roughly two or three times per simulated day), the model looks at the 100 most recent memories, asks itself three high-level questions about them, retrieves memories relevant to each question, and writes down insights that cite the memories they came from.

Those insights go back into the same memory stream, where they can be retrieved — and reflected on again. The result is a reflection tree: raw observations at the leaves, more abstract conclusions higher up.

Everything lives in one stream. Retrieval ranks it three ways; reflection writes new, higher-level memories back into it (dashed loop).

3. Planning

To keep behavior coherent across a day, each character drafts a rough plan (about five to eight chunks), then breaks each chunk into hour-long actions, then into 5–15 minute steps. Plans are stored in the memory stream too. At every step, the model decides whether something it just noticed is worth reacting to; if so, it rewrites the plan from that point.

What they tested

Interviews. The authors “interviewed” characters with questions about themselves, their memories, plans, reactions and reflections. 100 human evaluators ranked answers from the full architecture, versions with pieces removed, and human crowdworkers role-playing the same characters. The full architecture ranked most believable; removing reflection, then planning, then memory each made answers noticeably worse. The crowdworker answers landed below every version that still had memory.

Emergent behavior over two game days. One character was seeded with the idea of throwing a Valentine’s Day party. Without further scripting, word spread from 1 character to 13 of 25, and 5 of the 12 who were invited showed up. A mayoral candidacy spread from 1 character to 8 the same way.

Why it matters for builders

Rank memories on more than similarity. Recency + importance + relevance is a simple, readable scoring rule, and variations of it show up across agent memory systems. VaultMem, for example, re-ranks on cosine similarity plus a significance score that decays over time.
Store conclusions, not just events. Reflections are cheap summaries the agent can retrieve later, and keeping links back to the source memories makes them auditable.
Memory and planning are the same store. Writing plans into the memory stream means “what am I doing next?” is answered by the same retrieval as “what happened?”

Honest caveats

Retrieval still misses. The authors show characters failing to recall something they had heard, or recalling a detail of an event but not the event itself.
Embellishment. Characters sometimes added details nobody had said, or blended in the model’s world knowledge — one described a neighbor as the author of a famous 18th-century economics book.
Too polite. The authors attribute overly formal, overly agreeable dialogue to instruction tuning; one character rarely said no to suggestions, and her own interests drifted as a result.
Expensive. Running 25 characters for two game days took multiple days and, in the authors’ words, “thousands of dollars in token credits.”
Believability isn’t accuracy. The evaluation measured whether behavior seemed human, not whether memories were retrieved correctly. Benchmarks like LoCoMo test that more directly.

Earlier in the series: MemGPT (an OS-style alternative where the model pages its own memory), Complementary Learning Systems (the brain’s version of turning episodes into general knowledge — close in spirit to reflection), and Sentence-BERT (the kind of embeddings behind the relevance score).

Terms like reflection, consolidation and decay are defined in the agent memory glossary.

Paper TL;DR: MemGPT — Treating the Context Window Like RAM

2026-10-19T10:40:00+00:00

Paper: Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica & Joseph E. Gonzalez (UC Berkeley), MemGPT: Towards LLMs as Operating Systems (arXiv, 2023)

One-line TL;DR: Instead of waiting for bigger context windows, MemGPT treats the prompt like a computer’s RAM and lets the model itself move information between that small fast space and larger external storage, using function calls.

The problem

An LLM can only “see” what fits in its context window. Long chats and big document collections don’t fit, so something has to decide what stays in view. Operating systems solved a version of this decades ago: programs act as if they have lots of memory, while the OS quietly pages data between fast RAM and slower disk. The paper calls its version virtual context management.

How it works

MemGPT splits memory into two places.

Main context — what’s actually in the prompt — has three parts:

System instructions: read-only rules, including how to use the memory functions.
Working context: a small, fixed-size scratchpad of key facts (for example, things learned about the user). The model can only change it through function calls.
A FIFO message queue: the recent conversation. Its first entry is a running summary of older messages that were pushed out.

External context lives outside the prompt:

Recall storage: the full message history, searchable on demand.
Archival storage: a store for arbitrary text the model chooses to save or look up later.

Main context is the prompt; external context is everything the model can page back in. The model itself moves information between them.

The clever part is who moves things around: the model does. Its output is parsed as function calls (search recall, write to archival, edit working context), and the results are fed back in. A few mechanics make this work:

Memory pressure warnings. When the prompt gets close to full (the paper’s example uses 70%), the system adds a warning so the model can save what matters before it’s lost.
Eviction with summaries. When the prompt is full, a chunk of the oldest messages is evicted and folded into the running summary. Evicted messages aren’t deleted — they stay in recall storage.
Heartbeats. A function call can ask to keep going (request_heartbeat), so the model can chain several lookups before replying.
Events. User messages, system alerts and scheduled timers all wake the agent up, so it doesn’t only act when a person types.

What they tested

Two settings, each with a “can it really remember?” task:

Multi-session chat. In a deep memory retrieval task, the agent is asked about something from an earlier session. MemGPT answered far more of these correctly than the same models with a fixed context. In a “conversation opener” task, its openers drew on what it knew about the person, and the authors found that saving facts to working context was key.
Document analysis. For question answering over many documents, MemGPT could page through more search results than fit in one prompt. In a nested key-value lookup task, where one value points to another key, fixed-context models broke down as nesting got deeper, while MemGPT with GPT-4 kept answering by looking things up repeatedly.

Why it matters for builders

Memory can be a set of tools, not just a bigger prompt. Giving the model explicit read/write/search functions makes memory something you can inspect and log.
Separate “always in view” from “look it up.” A small working context of key facts plus searchable history is a pattern that shows up again and again in agent memory designs.
Plan for overflow. Warning before the window fills, and summarizing instead of silently truncating, is a simple idea worth copying.

Honest caveats

It leans on the model’s function-calling skill. The authors report that results dropped noticeably with GPT-3.5, which was weaker at calling functions.
Retrieval is still the bottleneck. With embedding search, the right document often wasn’t near the top, and the agent tended to stop paging before exhausting the results.
More model isn’t always better. On the nested lookup task, the GPT-4 Turbo version did worse than GPT-4, and some versions gave up after too few lookups.
Self-managed memory is a judgment call by the model. What gets saved, summarized or forgotten depends on the model’s choices, which is harder to predict than a fixed rule.

Earlier in the series: LoCoMo (a benchmark for exactly this kind of long-term memory), Sentence-BERT (the embeddings behind most semantic search), and Complementary Learning Systems (a brain-inspired split between fast and slow memory).

Terms like virtual context, context window, episodic memory and top-k retrieval are defined in the agent memory glossary.

Paper TL;DR: Hopfield (1982) — Memory as a Valley You Roll Into

2026-10-18T22:00:00+00:00

Paper: J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, PNAS 79(8), 1982

One-line TL;DR: connect lots of simple on/off neurons with symmetric weights and the network gets an energy that only goes down as it runs — so stored memories become stable valleys, and a partial or noisy cue rolls downhill into the full memory.

Almost everything I’ve built around memory started here. It’s also one of the most readable classic papers I know: five pages, very little math, a physicist explaining computation.

The setup

Neurons are either on or off (+1 or −1).
Every neuron connects to every other, and the connection strengths are symmetric: the weight from A to B equals the weight from B to A.
Storing a memory uses a Hebbian rule — “neurons that fire together wire together.” For each pattern you want to remember, strengthen connections between neurons that agree and weaken those between neurons that disagree. Store several patterns by adding up their contributions.
Recalling means picking neurons one at a time and flipping each to agree with the weighted vote of its neighbors, until nothing changes.

The key insight: an energy that only goes down

Hopfield’s move was to define an energy for the whole network’s state and show that, with symmetric weights and one-at-a-time updates, every update either lowers the energy or leaves it unchanged. The network can’t cycle forever; it has to settle.

Stored patterns sit at the bottoms of valleys in that energy landscape. Start anywhere nearby — a corrupted photo, half a phone number — and the network slides down into the nearest stored memory. That’s content-addressable memory: you retrieve by what you remember, not by where it’s stored. It’s also robust: knock out some neurons or connections and recall still mostly works.

Twenty-five lines of NumPy

Store five random patterns in 100 neurons, flip 30% of the bits of one, and let it settle:

import numpy as np

rng = np.random.default_rng(1)
N, P = 100, 5
patterns = rng.choice([-1, 1], size=(P, N))
W = (patterns.T @ patterns) / N        # Hebbian storage
np.fill_diagonal(W, 0)                 # no self-connections

def energy(s):
    return -0.5 * s @ W @ s

def recall(s, sweeps=5):
    s = s.copy()
    for _ in range(sweeps):
        for i in rng.permutation(N):   # one neuron at a time
            s[i] = 1 if W[i] @ s >= 0 else -1
    return s

target = patterns[0]
noisy = target.copy()
noisy[rng.choice(N, 30, replace=False)] *= -1

out = recall(noisy)
print((noisy == target).mean(), (out == target).mean())   # 0.7 -> 1.0
print(round(energy(noisy), 1), round(energy(out), 1))     # -8.6 -> -50.5

70% correct goes to 100% correct, and the energy drops as it does.

The limits

Capacity is small. Hopfield found the network reliably holds only about 0.15 memories per neuron. Push past that and recall falls apart.
Spurious memories. The landscape also grows valleys nobody stored — blends of real memories — and a cue can get stuck in one.
Similar memories interfere. Patterns that overlap a lot blur into each other.

Those limits are exactly what the 2020 follow-up fixes: modern Hopfield networks sharpen the valleys so capacity grows exponentially — and their update turns out to be transformer attention.

Why it still matters to me

Databases retrieve by key. Vector stores retrieve by nearest neighbor. Hopfield networks retrieve by settling — pattern completion from a partial cue, which is much closer to how remembering feels. That idea is the reason Attractor is called Attractor.

That completes the six papers from Part 1. The whole list, with short TL;DRs, lives on my papershelf.

For a one-paragraph definition of associative (content-addressable) memory, see the agent memory glossary.

Paper TL;DR: Sentence-BERT — The Quiet Workhorse Behind Semantic Search

2026-10-18T21:00:00+00:00

Paper: Nils Reimers & Iryna Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)

One-line TL;DR: fine-tune BERT so that every sentence gets its own fixed-size embedding you can compare with cosine similarity — turning semantic search over thousands of sentences from hours into seconds.

It’s the least flashy paper in this series and the one my code depends on most. Every time Attractor, VaultMem or daylog “searches by meaning,” it’s using a model descended from this paper.

The problem: BERT is a slow judge

In 2019, BERT was the best tool for deciding whether two sentences meant the same thing — but only if you fed it both sentences together. That works for scoring one pair. It falls apart for search.

The paper’s example: to find the most similar pair in a collection of 10,000 sentences, BERT would need around 50 million pairwise passes — roughly 65 hours on a GPU. And you can’t precompute anything, because every score depends on both sentences at once.

The obvious shortcut — run each sentence through BERT alone and average its outputs (or take the special [CLS] token) — produced embeddings that were often worse than old-fashioned averaged word vectors.

The fix: train it to embed

Sentence-BERT puts two copies of the same BERT side by side (a siamese network — shared weights), runs one sentence through each, pools each output into a single vector, and trains the whole thing so that:

sentences that mean similar things land close together, and
sentences that don’t land far apart.

They trained on natural-language-inference data (pairs labeled entailment / contradiction / neutral) and on similarity-scored pairs. Afterward you throw away the “pair” setup: each sentence goes through the model once, you store its vector, and comparing two sentences is a cheap cosine similarity.

That same 10,000-sentence search drops to about 5 seconds.

Why it still matters

It made vector search practical. Embed once, store in an index (pgvector, Qdrant, HNSW), and query in milliseconds. That’s the foundation of nearly every RAG system and agent memory store.
The library outlived the paper. The sentence-transformers package that came out of this work is still how most people get embedding models running in a few lines.
Small models are often enough. The 384-dimension model I use in Attractor runs comfortably on a CPU — handy when the whole point is keeping data on hardware you own.

Caveats worth knowing

Bi-encoders like SBERT trade a little accuracy for speed. When precision matters, a common pattern is to retrieve with embeddings, then re-rank the top results with a slower cross-encoder.
“Similar meaning” is whatever the training data said it was. Domain-specific text (logs, configs, medical notes) often needs a domain-tuned model.
Embeddings find related text; they don’t reason. Multi-hop questions — the ones LoCoMo showed are hardest — need more than nearest neighbors.

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems, LoCoMo and Memory as Asset.

Embeddings, HNSW and re-ranking each get a short definition in the agent memory glossary.

Paper TL;DR: Memory as Asset — Whose Memory Is It Anyway?

2026-10-18T20:00:00+00:00

Paper: Yanqi Pan, Qinghao Huang & Weihao Yang, Memory as Asset: From Agent-centric to Human-centric Memory Management (arXiv, March 2026)

One-line TL;DR: the memory an AI keeps about you is designed today as the agent’s internal resource; this paper argues it should be your asset — owned by you, shared only under your permissions, and able to grow over time.

This is the paper that reshaped VaultMem. I found it while I was stuck on a vague feeling that something about agent memory was backwards, and it gave that feeling a name.

The problem: agent-centric memory

Most memory systems treat what an agent remembers about you as part of the agent — stored by the platform, shaped for the platform’s use, and stuck there. The authors call this agent-centric memory. The consequences are familiar: you can’t take your history with you, you can’t decide who sees it, and every product rebuilds its own isolated “memory island” about the same person.

The proposal: memory as a personal asset

They propose flipping the default, built on three principles:

Memory in hand — individuals own and control their memories, not centralized platforms.
Memory groups — memories can be shared and combined, but only through permissions the owner sets.
Collective memory evolution — memory isn’t frozen: it’s recorded, consolidated and abstracted into higher-level knowledge over time, and can be exchanged between groups under control.

Each memory object carries its content, context, an explicit owner, an access policy and a version. Ownership and permissions are part of the data itself, not an afterthought in some admin panel.

The three layers

Personal storage — fast storage of each person’s private memory objects.
Evolution — updates memory from interactions: recording experiences, merging related memories, abstracting knowledge. It can run locally for privacy or remotely for compute.
Memory exchange network — a decentralized way to discover, authenticate and exchange memory assets.

Notably, the paper doesn’t insist everything stays on your device. If a provider hosts your memory, it should be encrypted so only you can read it.

How it shaped what I built

That last point is where VaultMem lives. I treat it as an implementation of the first layer: the platform stores your memories, but they’re encrypted with a key only you hold, so the operator cryptographically can’t read them. Everything in the VaultMem series — client-side encryption, encrypted search, keeping PII away from the model provider — follows from taking “memory in hand” literally.

Caveats worth knowing

The authors are upfront that this is an early draft: the terms and design are still evolving.
It’s a conceptual and architectural paper — there are no benchmarks or experiments yet. The claim that existing systems meet at most two of the three principles is an argument, not a measurement.
The exchange network raises hard questions (trust, incentives, revocation) that the paper sketches rather than solves.

I still think it’s the right question to ask before building any memory feature: whose memory is this?

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems and LoCoMo.

Zero-knowledge storage and client-side encryption are defined in the privacy section of the agent memory glossary.

Paper TL;DR: LoCoMo — How Good Is Your Agent's Memory, Really?

2026-10-18T19:00:00+00:00

Paper: Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents (ACL 2024)

One-line TL;DR: LoCoMo is a benchmark of very long, multi-session conversations with questions that probe whether a system really remembers — and even strong long-context models and retrieval setups fall well short of people on it.

If you’re building memory for an agent, this is the paper that keeps you honest.

What’s in the benchmark

Most chat benchmarks are a few turns long. LoCoMo conversations run for dozens of sessions and hundreds of turns, spread over simulated weeks and months. The authors generated them with LLM agents that each had a persona and a timeline of life events, then had people review and fix them so the conversations stay consistent. Some turns include shared images, too.

On top of the conversations they built three tasks: answering questions, summarizing events, and generating responses. The question-answering part is the one I care about, and its categories are what make it useful:

Single-hop — one fact from one place in the history.
Multi-hop — needs to connect facts from different sessions.
Temporal — when did something happen, or what happened before or after.
Open-domain — combine the conversation with general knowledge.
Adversarial — questions about things that never happened, where the right answer is “that wasn’t mentioned.”

What they found

Even with long context windows or retrieval-augmented generation, models trail human performance by a wide margin. Retrieval helps, but the hard categories stay hard — especially reasoning about time and resisting the urge to make up answers to adversarial questions. Bigger context alone doesn’t fix it.

How I used it

When I built VaultMem, my first benchmark was 18 synthetic memories I wrote myself. It looked great — which was the problem. LoCoMo gave me a test I didn’t design. On a sample of its questions, retrieval was strong for single-hop facts and weakest for multi-hop, which needs pieces that weren’t individually asked about. That’s a retrieval problem before it’s a model problem, and it changed what I worked on next.

The takeaway for builders

Test against data you didn’t write. A memory system always looks good on its author’s examples.
Score by category, not just overall. “85% accurate” hides that temporal and multi-hop questions might be failing.
Measure the “I don’t know.” A memory that confidently invents a past is worse than one that admits a gap.

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, and Complementary Learning Systems.

Recall@k and MRR, the usual ways to score retrieval, are explained in the agent memory glossary.

Paper TL;DR: Complementary Learning Systems — Why Agents Need Two Kinds of Memory

2026-10-18T18:00:00+00:00

Paper: McClelland, McNaughton & O’Reilly, Why there are complementary learning systems in the hippocampus and neocortex, Psychological Review (1995)

One-line TL;DR: the brain learns with two systems on purpose — a fast one that grabs specific experiences and a slow one that gradually extracts general knowledge — because a single system can’t do both without wrecking what it already knows.

It’s a 1995 psychology paper, and it’s one of the most useful things I’ve read about designing memory for AI agents.

The problem: learning fast breaks things

The paper starts from a puzzle that anyone who has fine-tuned a neural network will recognize. If you teach a network something new by adjusting the same shared weights that hold everything else, the new learning can overwrite the old. This is catastrophic interference: learn B, forget A.

The fix that works in artificial networks is to learn slowly and interleave — mix new examples with old ones so the weights find a setting that serves both. But that’s slow. You can’t wait for hundreds of repetitions to remember where you parked this morning.

The brain’s answer: two learners

The authors argue the brain splits the job:

Hippocampus — the fast learner. Stores specific episodes quickly, often after a single exposure, using sparse, separate codes so similar memories don’t collide. Great for “what happened at Tuesday’s meeting.”
Neocortex — the slow learner. Learns gradually, with overlapping representations that capture structure shared across many experiences. Great for “how meetings at my company usually go.”

The bridge between them is replay. Over time — notably during sleep — the hippocampus reactivates recent episodes, and the neocortex learns from those replays interleaved with older knowledge. That’s consolidation: specific memories slowly become general knowledge without overwriting what came before.

The two-learner idea from the paper, with the agent-memory equivalent of each part.

The paper backs this with evidence from people with hippocampal damage, who struggle to form new episodic memories yet keep older knowledge, and with network simulations showing why interleaved learning avoids interference.

Why it matters for AI agents

Read with agents in mind, the paper is basically a design doc:

Keep an episodic store separate from the model. Fine-tuning a model on every conversation is the “learn fast in shared weights” mistake. A separate, fast-writing memory — a log or vector store of specific events — is the hippocampus.
Consolidate on a schedule, not on every write. Periodically summarize or distill many episodes into durable facts and preferences, mixing in what’s already known instead of overwriting it.
Let memories change over time. Things that keep getting recalled should get stronger; one-off details can fade. A store that treats every memory as equally permanent forever is missing half the design.

This is the model I borrowed for Attractor: memories carry the context they were formed in, track how often they’re used, and are meant to strengthen or decay rather than sit frozen.

Caveats worth knowing

The theory has been extended since 1995 — the neocortex can learn new things quickly when they fit an existing schema — so “neocortex is always slow” is a simplification.
Brains aren’t databases; the analogy is a guide for design choices, not a blueprint.

Next in the series: the benchmark that told me how good my own memory system actually was. Earlier parts: six papers on AI memory and Hopfield Networks Is All You Need.

Quick definitions of consolidation and decay are in the agent memory glossary.

Paper TL;DR: Hopfield Networks Is All You Need — Attention Is a Memory Lookup

2026-10-18T17:00:00+00:00

Paper: Ramsauer, Schäfl, Lehner et al., Hopfield Networks is All You Need (2020)

One-line TL;DR: a continuous version of the 1982 Hopfield network can store exponentially many memories, usually retrieves one in a single step — and that single step is exactly the attention operation inside every transformer.

This is the paper that made me stop thinking of Hopfield networks as history. Here’s the idea in plain language, plus a tiny demo you can run.

The old idea: memories as valleys

In Hopfield’s 1982 network, you store patterns (say, strings of +1 and −1) in the connection weights. The network has an “energy,” and each stored pattern sits at the bottom of a valley. Start from a corrupted pattern, let the network update, and it rolls downhill into the nearest stored memory. That’s content-addressable memory: you find a memory by giving a piece of it, not by knowing its address.

The catch: the classic network only holds about 0.14 patterns per neuron. Store more than that and the valleys blur together into spurious memories.

The new idea: sharpen the valleys

Ramsauer et al. swap the old quadratic energy for one built on log-sum-exp — a smooth version of “max.” Intuitively, each stored pattern now carves a much sharper, deeper valley, so many more of them fit without interfering. The paper shows the capacity grows exponentially with the dimension of the patterns, and that retrieval typically converges after one update.

That one update is:

new_state = X · softmax(β · Xᵀ · state)

where the columns of X are the stored patterns and β controls how sharp the valleys are.

Wait — that’s attention

Now look at transformer attention:

output = V · softmax(Kᵀ · q / √d)

Set the query q to the current state, use the stored patterns as both keys K and values V, and pick β = 1/√d. It’s the same equation. Every attention head in an LLM is doing a one-step associative memory lookup: compare the query against everything stored, softmax the similarities, return a weighted blend that’s usually dominated by the best match.

The paper goes further, showing how different heads and layers behave like different retrieval regimes — some settle on one memory, some average over many — but the core equivalence is the part that stuck with me.

Twenty lines of NumPy

Store five random ±1 patterns, flip 40% of the bits in one of them, and do a single update:

import numpy as np

rng = np.random.default_rng(0)
d, n = 64, 5
X = rng.choice([-1.0, 1.0], size=(d, n))   # stored memories (columns)

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

def retrieve(xi, beta=0.5):
    return X @ softmax(beta * X.T @ xi)      # one modern-Hopfield update

target = X[:, 2]
noisy = target.copy()
noisy[rng.choice(d, size=int(0.4 * d), replace=False)] *= -1

out = retrieve(noisy)
print("overlap before:", int(noisy @ target), "/", d)       # 14 / 64
print("overlap after: ", int(np.sign(out) @ target), "/", d) # 64 / 64

One step takes a pattern that’s only loosely related to the target back to a perfect copy. The softmax weights tell the story: about 0.97 on the right memory and crumbs on the rest — exactly what an attention map looks like.

Why I care, as someone building agent memory

Most agent memory today is “embed everything, return the top-k nearest neighbors.” This paper reframes that: nearest-neighbor search is a crude, non-differentiable cousin of what attention already does well. It nudged me to build Attractor around retrieval by partial cues and context, and to think of memory as something that settles rather than something you look up.

Caveats worth knowing

“Exponential capacity” is a statement about well-separated patterns in high dimensions; real embeddings are messy and correlated.
The one-step convergence holds when β is large enough relative to how similar the stored patterns are — similar memories still blur together.
Treating attention as memory is a lens, not a full explanation of what transformers learn.

If you read only one modern paper about associative memory, make it this one. The rest of what I’m reading is on my papershelf, and Part 1 of this series covers the other papers behind my memory projects.

Terms like top-k retrieval and associative memory are defined in plain language in the agent memory glossary.

Paper TL;DR: Six Papers That Shaped How I Think About AI Memory

2026-10-18T16:00:00+00:00

Most of my side projects over the past year have circled one question: how should an AI system remember things? Not just “where do we put the vectors,” but how memory gets formed, found again, strengthened or forgotten — and who gets to read it.

I didn’t start with papers. I started building, got stuck, and went looking for how other people had thought about the problem. These six are the ones that actually changed what I built. Here’s each one in a few sentences, in plain language, plus what I took from it. If you only read one, read the first.

1. Hopfield (1982) — memory as a place you roll into

Neural networks and physical systems with emergent collective computational abilities · PNAS

TL;DR: Wire up simple on/off neurons with symmetric connections and the network gets an “energy” that can only go down as it updates. Stored memories sit at the bottoms of valleys in that landscape. Give it a partial or noisy pattern and it slides downhill into the full memory. You don’t look memories up by key — you fall into them.

What I took from it: this was the “oh” moment behind Attractor. Recall from partial cues feels much closer to how I actually remember things than a database lookup does.

2. Ramsauer et al. (2020) — attention is a Hopfield update

Hopfield Networks is All You Need · arXiv

TL;DR: A “modern” Hopfield network with continuous states can store exponentially many patterns and usually retrieves one in a single step. And that single update step is mathematically the same operation as transformer attention.

What I took from it: the 1982 idea isn’t a museum piece — it’s sitting inside every LLM. That made me take attractor-style retrieval seriously as a design for agent memory, not just a curiosity.

3. McClelland, McNaughton & O’Reilly (1995) — two learners, not one

Why there are complementary learning systems in the hippocampus and neocortex · Psychological Review

TL;DR: The brain uses a fast learner that grabs specific episodes (hippocampus) and a slow learner that gradually pulls out general structure (neocortex). Replaying episodes into the slow system lets it learn without overwriting what it already knows.

What I took from it: a vocabulary for something agents get wrong — treating every memory the same forever. It’s why Attractor tracks access and why I think memories should strengthen with use and fade without it.

4. Reimers & Gurevych (2019) — embeddings you can actually search

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · arXiv

TL;DR: Plain BERT has to compare sentences two at a time, which is painfully slow for search. Training it in a “siamese” setup gives every sentence its own embedding you can compare with cosine similarity, so finding the closest match among thousands takes seconds instead of hours.

What I took from it: the unglamorous workhorse. The sentence-transformers models behind semantic search in Attractor, VaultMem and daylog all come from this line of work.

5. Maharana et al. (2024) — testing memory honestly

Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo) · arXiv

TL;DR: A benchmark of very long, multi-session conversations with questions that need single facts, multi-step reasoning, time awareness, or the sense to say “that never happened.” Even strong long-context models and RAG setups struggle with it.

What I took from it: my own little synthetic test was flattering me. Running VaultMem against LoCoMo gave me numbers I could actually trust — and showed me where retrieval was weak.

6. Pan, Huang & Yang (2026) — whose memory is it?

Memory as Asset: From Agent-centric to Human-centric Memory Management · arXiv

TL;DR: Argues that agent memory today is built for the agent rather than the person, and proposes treating the memory an AI keeps about you as an asset you own, share under your own permissions, and grow over time.

What I took from it: this one reshaped VaultMem entirely. It gave words to the feeling that an agent’s memory of your life shouldn’t be readable by whoever runs the agent. VaultMem ended up implementing its first layer.

What ties them together

Reading these in a row, I keep landing on the same picture: good memory is found by meaning and context, changes over time, and belongs to the person it’s about. Most systems today nail the first, ignore the second, and get the third backwards.

All six live on my papershelf with the rest of what I’m reading. If any of the vocabulary here is new, the agent memory glossary has short definitions. If there’s a paper on memory or agents you think I should read next, I’d genuinely love to hear about it.