<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://www.avinashgosavi.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://www.avinashgosavi.com/" rel="alternate" type="text/html" /><updated>2026-11-09T01:32:59+00:00</updated><id>https://www.avinashgosavi.com/feed.xml</id><title type="html">Avinash Gosavi</title><subtitle>Portfolio and blog of Avinash Gosavi — Staff Software Engineer on distributed systems, API design and reliability, with privacy-first AI tools on the side.</subtitle><author><name>Avinash Gosavi</name></author><entry><title type="html">Paper TL;DR: HNSW — How Vector Search Skips Most of the Haystack</title><link href="https://www.avinashgosavi.com/post/paper-tldr-hnsw/" rel="alternate" type="text/html" title="Paper TL;DR: HNSW — How Vector Search Skips Most of the Haystack" /><published>2026-10-20T20:45:00+00:00</published><updated>2026-10-23T01:35:48+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-hnsw</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-hnsw/"><![CDATA[<p><strong>Paper:</strong> Yu. A. Malkov &amp; D. A. Yashunin, <a href="https://arxiv.org/abs/1603.09320"><em>Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs</em></a> (arXiv, first posted March 2016; later published in IEEE Transactions on Pattern Analysis and Machine Intelligence, <a href="https://doi.org/10.1109/TPAMI.2018.2889473">DOI 10.1109/TPAMI.2018.2889473</a>)</p>

<p><strong>One-line TL;DR:</strong> Stack a few neighbor graphs on top of each other — sparse ones with long jumps at the top, a dense one with every item at the bottom — and you can find the closest vectors to a query by hopping down the stack instead of checking everything.</p>

<h2 id="the-question">The question</h2>

<p>Every memory or <a href="/post/agent-memory-glossary/#rag">RAG</a> system eventually asks the same thing: <em>which stored <a href="/post/agent-memory-glossary/#embedding">embeddings</a> are closest to this query vector?</em> Checking every vector works, but the cost grows linearly with the collection. Exact shortcuts like k-d trees stop helping once vectors have hundreds of dimensions. So practical systems use <em>approximate</em> nearest neighbor (ANN) search: accept that you’ll occasionally miss a true neighbor, and measure how often you don’t with <a href="/post/agent-memory-glossary/#recall-at-k">recall</a>.</p>

<p>One family of ANN methods builds a <em>proximity graph</em> — each item linked to its near neighbors — and searches by greedily walking toward the query. The authors’ own earlier method, Navigable Small World (NSW), did this but had two problems: search cost grew faster than logarithmically as the data grew, and on low-dimensional or tightly clustered data the greedy walk could get stuck in a “false” local minimum. HNSW is their fix.</p>

<h2 id="how-it-works">How it works</h2>

<p>The paper describes the idea as a <a href="https://en.wikipedia.org/wiki/Skip_list">skip list</a> where the linked lists are replaced by proximity graphs.</p>

<ul>
  <li><strong>Layers.</strong> Every item lives in the bottom layer. When it’s inserted, it also draws a random top layer from an exponentially decaying distribution, so only a few items reach the upper layers. Upper layers are sparse and their links span long distances; the bottom layer is dense with short links.</li>
  <li><strong>Search.</strong> Start at a fixed entry point on the top layer. On each layer, greedily move to whichever neighbor is closer to the query until none is, then drop down a layer and continue from there. On the bottom layer, keep a wider list of candidates (size <code class="language-plaintext highlighter-rouge">ef</code>) and return the best k. A bigger <code class="language-plaintext highlighter-rouge">ef</code> means higher recall and more work.</li>
  <li><strong>Insertion.</strong> A new item is placed by running the same descent, then — on each layer from its own top layer down — linking to up to <code class="language-plaintext highlighter-rouge">M</code> neighbors found by a wider search (<code class="language-plaintext highlighter-rouge">efConstruction</code>). If a neighbor ends up with too many links, its list is pruned back to a cap.</li>
  <li><strong>A smarter way to pick neighbors.</strong> Rather than always linking to the closest candidates, the heuristic keeps a candidate only if it’s closer to the new item than to any neighbor already chosen. That spreads links out in different directions, which keeps separate clusters connected — exactly where plain NSW got stuck.</li>
</ul>

<blockquote class="diagram-text"><p><strong>Diagram: How an HNSW search hops down the layers</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-hnsw/">see it in the post</a>)</p><p>A sketch of three stacked neighbor graphs. The top layer has only two items joined by one long link; the middle layer has four; the bottom layer has every item, each linked to close neighbors. A search starts at the entry point on the top layer, hops to the neighbor closer to the query, and when no neighbor is closer it drops to the same item one layer down and continues. On the bottom layer it keeps a list of ef candidates around the query and returns the best k.</p><p><em>A sketch, not real data: upper layers let the search take long jumps, the bottom layer lets it finish the job up close.</em></p></blockquote>

<p>The knobs the paper discusses are few: <code class="language-plaintext highlighter-rouge">M</code> (links per item; it suggests roughly 5–48, higher for high-recall or high-dimensional data), <code class="language-plaintext highlighter-rouge">efConstruction</code> (build quality) and <code class="language-plaintext highlighter-rouge">ef</code> (search effort). The graph links cost roughly 60–450 bytes per item on top of the vectors themselves.</p>

<h2 id="what-they-found">What they found</h2>

<ul>
  <li><strong>Roughly logarithmic search.</strong> The authors argue search cost scales as O(log N) under idealized assumptions, and see logarithmic scaling empirically on low-dimensional data. Construction is roughly O(N log N) there.</li>
  <li><strong>Fastest in their vector benchmarks.</strong> Single-threaded on the ann-benchmarks suite, HNSW clearly beat NSW, FLANN, Annoy, VP-tree and FALCONN (an LSH library) on SIFT, GloVe, DEEP and CoPhIR, and stayed ahead on random 4-dimensional data.</li>
  <li><strong>Not just for vectors.</strong> Because it only needs a distance function, it also led on non-vector data — sparse text, topic histograms, image signatures, even DNA strings compared by edit distance. The biggest gain over NSW was roughly three orders of magnitude on one low-dimensional, clustered dataset.</li>
  <li><strong>Speed versus memory against compression.</strong> On 200 million SIFT vectors, HNSW used more RAM than Faiss’s product-quantization index (about 64 GB versus 23.5–30 GB) but was more accurate, faster to search and much faster to build.</li>
</ul>

<h2 id="why-it-matters-for-builders">Why it matters for builders</h2>

<ol>
  <li><strong>It’s probably already in your stack.</strong> The paper notes that Faiss added its own HNSW implementation, and RAG’s Wikipedia index used a FAISS HNSW index (see the <a href="/post/paper-tldr-retrieval-augmented-generation/">RAG TL;DR</a>). <a href="/post/vaultmem-part-2-multimodal-temporal-backends/">VaultMem</a> has an optional <code class="language-plaintext highlighter-rouge">HNSWVectorIndex</code> that wraps the <a href="https://github.com/nmslib/hnswlib">hnswlib</a> library.</li>
  <li><strong>Small collections don’t need it.</strong> The graph has real overhead, so brute force is often fine early on — VaultMem Part 2 falls back to exact cosine search below 1,000 memories and shows where the crossover happens.</li>
  <li><strong>Recall is a dial, not a constant.</strong> <code class="language-plaintext highlighter-rouge">ef</code> trades speed for recall at query time, without rebuilding. When a memory system “forgets” something it stored, the index settings are one more place to look — alongside the embeddings and the <a href="/post/agent-memory-glossary/#re-ranking">re-ranking</a> step.</li>
  <li><strong>The index decides which candidates you even see.</strong> Better embeddings (<a href="/post/paper-tldr-sentence-bert/">Sentence-BERT</a>) and better prompt placement (<a href="/post/paper-tldr-lost-in-the-middle/">Lost in the Middle</a>) only help with the neighbors the index returns.</li>
</ol>

<h2 id="honest-caveats">Honest caveats</h2>

<ul>
  <li><strong>The O(log N) story has gaps.</strong> The guarantee leans on idealized graphs the heuristic doesn’t actually build, and the authors say high-dimensional behavior isn’t proven. On the 200M-vector test, query time drifted somewhat from pure logarithmic growth.</li>
  <li><strong>Memory hungry.</strong> All those links live alongside the vectors, so it needs more RAM than compressed indexes like product quantization.</li>
  <li><strong>No deletes or updates — at the time.</strong> The paper lists element updates and removal as future work. That matters for memory systems that edit or <a href="/post/agent-memory-glossary/#memory-unlearning">unlearn</a> facts; check what your library supports today.</li>
  <li><strong>Hard to distribute.</strong> Every search starts at the top layer, which can become a hotspot, and the authors found that simply splitting data across machines scales poorly in total throughput.</li>
  <li><strong>Benchmarks are a snapshot.</strong> Results depend on the hardware and the library versions of the time, and some baselines were older builds.</li>
</ul>

<p>Earlier in the series: <a href="/post/paper-tldr-retrieval-augmented-generation/">RAG</a> (the retrieve-then-generate loop this index sits inside) and <a href="/post/paper-tldr-sentence-bert/">Sentence-BERT</a> (the embeddings it searches over).</p>

<p>Terms like <a href="/post/agent-memory-glossary/#hnsw"><em>HNSW</em></a>, <a href="/post/agent-memory-glossary/#embedding"><em>embedding</em></a>, <a href="/post/agent-memory-glossary/#semantic-search"><em>semantic search</em></a>, <a href="/post/agent-memory-glossary/#top-k-retrieval"><em>top-k retrieval</em></a> and <a href="/post/agent-memory-glossary/#recall-at-k"><em>Recall@k</em></a> are defined in the <a href="/post/agent-memory-glossary/">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><summary type="html"><![CDATA[A TL;DR of HNSW (Malkov & Yashunin): a layered graph that finds approximate nearest vectors in roughly logarithmic time, behind much of today's vector search.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-hnsw.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-hnsw.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Paper TL;DR: RAG — The Paper That Named Retrieval-Augmented Generation</title><link href="https://www.avinashgosavi.com/post/paper-tldr-retrieval-augmented-generation/" rel="alternate" type="text/html" title="Paper TL;DR: RAG — The Paper That Named Retrieval-Augmented Generation" /><published>2026-10-20T15:45:00+00:00</published><updated>2026-10-23T01:35:48+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-retrieval-augmented-generation</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-retrieval-augmented-generation/"><![CDATA[<p><strong>Paper:</strong> Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel &amp; Douwe Kiela (Facebook AI Research, University College London, New York University), <a href="https://arxiv.org/abs/2005.11401v4"><em>Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks</em></a> (arXiv, 2020; revised 2021)</p>

<p><strong>One-line TL;DR:</strong> Instead of hoping a language model memorized the right fact, give it a search step: look up a few relevant Wikipedia passages, generate the answer from them, and train the search and the writer together.</p>

<h2 id="the-question">The question</h2>

<p>By 2020, large language models clearly stored a lot of facts in their weights. But the paper points out the downsides: that knowledge is hard to expand or revise, the model can’t show where an answer came from, and it can hallucinate. Earlier hybrid models that paired a language model with a retriever (REALM and ORQA) had only been tried on <em>extractive</em> question answering — pulling answer spans out of passages rather than writing new text. This paper asks whether one general recipe can combine the two: a model that retrieves documents <em>and</em> generates free-form answers, fine-tuned end to end for many different tasks.</p>

<h2 id="how-it-works">How it works</h2>

<p>The paper splits knowledge into two kinds of memory:</p>

<ul>
  <li><strong>Parametric memory</strong> — what’s baked into the model’s weights. Here that’s BART-large, a pre-trained sequence-to-sequence generator.</li>
  <li><strong>Non-parametric memory</strong> — an external index you can search. Here that’s a December 2018 Wikipedia dump cut into 100-word chunks (about 21 million of them), each turned into an <a href="/post/agent-memory-glossary/#embedding">embedding</a> by a BERT document encoder from DPR (Dense Passage Retrieval). The vectors live in a FAISS index using an <a href="/post/paper-tldr-hnsw/">HNSW</a> approximation, so lookups stay fast.</li>
</ul>

<p>At question time, a second BERT encoder turns the input into a query vector, the index returns the <a href="/post/agent-memory-glossary/#top-k-retrieval">top-k</a> passages, and BART generates an answer conditioned on the input plus each passage. Nobody labels which passage is “right”: the retriever learns only from whether the final answer comes out well.</p>

<p>Two variants decide how the passages get combined:</p>

<ul>
  <li><strong>RAG-Sequence</strong> uses the same passage for the whole answer, then weighs the answers it would give for each passage.</li>
  <li><strong>RAG-Token</strong> can lean on a different passage for each word, so one answer can stitch together facts from several documents.</li>
</ul>

<p>One practical shortcut: only the query encoder and the generator are trained. The document encoder — and so the whole 21-million-vector index — stays frozen, because re-embedding Wikipedia during training is expensive. The authors report they didn’t need that step for strong results.</p>

<blockquote class="diagram-text"><p><strong>Diagram: How RAG retrieves, then generates</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-retrieval-augmented-generation/">see it in the post</a>)</p><p>The input goes to a BERT query encoder, which turns it into a query vector. The vector searches a FAISS index of about 21 million 100-word Wikipedia chunks from December 2018, whose document encoder stays frozen. The top-k passages go to a BART generator, the model's parametric memory, which writes the answer. The only training signal is whether the final answer is right, and it updates just the query encoder and the generator. The index can be swapped for one built from a 2016 Wikipedia dump, which changes what the model knows without retraining.</p><p><em>Only the query encoder and the generator are trained. The index stays frozen, so swapping it for an older Wikipedia dump changes what the model knows.</em></p></blockquote>

<h2 id="what-they-found">What they found</h2>

<ul>
  <li><strong>Strong open-domain question answering.</strong> On Natural Questions, WebQuestions and CuratedTrec, RAG beat the previous best systems in the paper’s comparison, while writing answers instead of copying spans. On TriviaQA the picture is mixed: it led on one test split but not the standard one.</li>
  <li><strong>It can answer even when the passage doesn’t spell it out.</strong> On Natural Questions, RAG still got about 11.8% of answers right in cases where the exact answer wasn’t in any retrieved passage — something a pure extractor can’t do.</li>
  <li><strong>More factual, more specific text.</strong> In a human evaluation of generated Jeopardy-style questions, raters judged RAG more factual than plain BART far more often than the other way round (42.7% vs. 7.1% of cases), and more specific too.</li>
  <li><strong>Small next to giant models.</strong> RAG has 626M trainable parameters; the largest closed-book model it’s compared to, T5-11B, has 11 billion — yet RAG scored higher on open-domain QA.</li>
  <li><strong>Swap the index, update the knowledge.</strong> The authors built a second index from a 2016 Wikipedia dump and asked “Who is {position}?” about 82 world leaders who changed between 2016 and 2018. With the matching index RAG got roughly 70% right; with the mismatched one, 4–12%. Changing what the model knows meant replacing the index, not retraining.</li>
</ul>

<h2 id="why-it-matters-for-builders">Why it matters for builders</h2>

<ol>
  <li><strong>It’s the template most “chat with your docs” systems still follow.</strong> Embed chunks, search by meaning, paste the top results into the prompt, generate. Agent memory systems do the same thing with memories instead of Wikipedia pages — the <a href="/post/agent-memory-glossary/#storing-and-finding-memories">glossary’s “Storing and finding memories” section</a> walks through each piece.</li>
  <li><strong>Updating knowledge is a data problem, not a training problem.</strong> The hot-swap experiment is the cleanest argument for keeping facts outside the weights: if something changes, you change the index. The same idea is why memory systems store facts with timestamps and <a href="/post/agent-memory-glossary/#versioned-facts">versions</a> rather than hoping the model “knows” the latest.</li>
  <li><strong>Provenance comes for free.</strong> Because the answer is conditioned on specific passages, you can show where it came from — useful for debugging and for trust.</li>
  <li><strong>Retrieval quality still caps everything.</strong> The retriever here is learned only through the final answer, and the paper reports it can fail badly (see caveats). Later work in this series picks up from there: <a href="/post/paper-tldr-lost-in-the-middle/">Lost in the Middle</a> shows that even good passages help less when they land mid-prompt, and <a href="/post/paper-tldr-memgpt/">MemGPT</a> treats the context window as something to manage rather than fill.</li>
</ol>

<h2 id="honest-caveats">Honest caveats</h2>

<ul>
  <li><strong>This is 2020-era plumbing.</strong> “RAG” today usually means an instruction-tuned LLM plus an off-the-shelf retriever, with no joint training. The paper’s specific recipe — BART, DPR, a frozen Wikipedia index — is mostly of historical interest now; the <em>idea</em> is what stuck.</li>
  <li><strong>Retrieval can collapse.</strong> In early story-generation experiments, the retriever learned to return the same documents no matter the input, after which the generator ignored them and RAG behaved just like plain BART. The authors suggest tasks with long outputs or less need for facts may give the retriever too weak a signal.</li>
  <li><strong>Not state of the art everywhere.</strong> On FEVER fact verification, RAG came within about 4.3% of the best pipeline (3-way) without being given gold evidence — close, but not ahead.</li>
  <li><strong>The index is only as good as its source.</strong> The authors note that Wikipedia will likely never be fully factual or free of bias, and that the same misuse concerns as other text generators apply.</li>
  <li><strong>Big infrastructure for its time.</strong> The paper notes the Wikipedia index initially needed about 100 GB of CPU memory, cut to 36 GB with FAISS compression.</li>
</ul>

<p>Earlier in the series: <a href="/post/paper-tldr-sentence-bert/">Sentence-BERT</a> (the kind of embeddings that make the search step work), <a href="/post/paper-tldr-lost-in-the-middle/">Lost in the Middle</a> (where retrieved passages land in the prompt) and <a href="/post/paper-tldr-memgpt/">MemGPT</a> (an agent that pages its own memory in and out). Next in the series: <a href="/post/paper-tldr-hnsw/">HNSW</a>, the graph index behind fast nearest-neighbor lookups like this one.</p>

<p>Terms like <a href="/post/agent-memory-glossary/#rag"><em>retrieval-augmented generation</em></a>, <a href="/post/agent-memory-glossary/#embedding"><em>embedding</em></a>, <a href="/post/agent-memory-glossary/#semantic-search"><em>semantic search</em></a>, <a href="/post/agent-memory-glossary/#top-k-retrieval"><em>top-k retrieval</em></a> and <a href="/post/agent-memory-glossary/#hnsw"><em>HNSW</em></a> are defined in the <a href="/post/agent-memory-glossary/">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><summary type="html"><![CDATA[A TL;DR of the RAG paper (Lewis et al., 2020): a language model paired with a searchable Wikipedia index, so swapping the index updates what it knows.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-retrieval-augmented-generation.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-retrieval-augmented-generation.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Paper TL;DR: Lost in the Middle — Where You Put Context Matters</title><link href="https://www.avinashgosavi.com/post/paper-tldr-lost-in-the-middle/" rel="alternate" type="text/html" title="Paper TL;DR: Lost in the Middle — Where You Put Context Matters" /><published>2026-10-20T10:45:00+00:00</published><updated>2026-10-23T01:35:48+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-lost-in-the-middle</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-lost-in-the-middle/"><![CDATA[<p><strong>Paper:</strong> Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni &amp; Percy Liang (Stanford, UC Berkeley, Samaya AI), <a href="https://arxiv.org/abs/2307.03172v3"><em>Lost in the Middle: How Language Models Use Long Contexts</em></a> (arXiv, 2023)</p>

<p><strong>One-line TL;DR:</strong> Give a model a long prompt with one useful passage in it, and how well it answers depends on <em>where</em> that passage sits: best at the very start or very end, noticeably worse in the middle — even for models built for long contexts.</p>

<h2 id="the-question">The question</h2>

<p>Retrieval-augmented generation and agent memory both work the same way at the last step: fetch some passages, paste them into the prompt, and hope the model reads them. Bigger context windows make it tempting to paste in more. This paper asks a simple question first: does the model actually use everything in a long prompt equally well?</p>

<h2 id="how-they-tested-it">How they tested it</h2>

<p>Two tasks, each designed so the authors could move the one relevant piece of information around while keeping everything else the same.</p>

<ul>
  <li><strong>Multi-document question answering.</strong> Questions from NaturalQuestions-Open, each paired with one Wikipedia passage that contains the answer plus distractor passages that look relevant but don’t. Prompts held 10, 20 or 30 passages, and the answer passage was moved from first to last.</li>
  <li><strong>Key-value retrieval.</strong> A synthetic stress test: a JSON object of random UUID keys and values (75, 140 or 300 pairs), and the model has to return the value for one key. No language cues, just “find this and copy it.”</li>
</ul>

<p>They ran open models (MPT-30B-Instruct, LongChat-13B) and API models (GPT-3.5-Turbo, Claude-1.3, and their longer-context versions, plus GPT-4 on a subset).</p>

<h2 id="what-they-found">What they found</h2>

<ul>
  <li><strong>A U-shaped curve.</strong> Accuracy was highest when the answer passage came at the very start or the very end, and dropped when it sat in the middle — by more than 20% in some settings. The authors describe this as a primacy bias plus a recency bias.</li>
  <li><strong>The middle can be worse than nothing.</strong> In one 20-document setting, GPT-3.5-Turbo with the answer buried mid-prompt scored lower than the same model answering with no documents at all.</li>
  <li><strong>A longer window didn’t fix it.</strong> When a prompt fit in both a model and its extended-context version, the two scored almost the same. A bigger window lets you paste more; it doesn’t make the model read it better.</li>
  <li><strong>More retrieved documents stop helping early.</strong> In an open-domain QA case study, the retriever kept finding the answer more often as it returned more passages, but the model’s accuracy flattened out well before that. Going from 20 to 50 passages bought only a point or two, at much higher cost and latency.</li>
  <li><strong>Asking the question twice helps for lookups.</strong> Putting the query both before and after the data made key-value retrieval near-perfect, but it barely changed the multi-document QA results.</li>
</ul>

<blockquote class="diagram-text"><p><strong>Diagram: The U-shaped curve from Lost in the Middle</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-lost-in-the-middle/">see it in the post</a>)</p><p>A qualitative sketch, not the paper's data. The horizontal axis is where the one useful passage sits in a prompt of many passages, from first to last. The vertical axis is answer accuracy. The curve is high when the passage is first (primacy bias), dips lowest when it is in the middle, and rises again when it is last (recency bias).</p><p><em>A sketch of the shape the paper reports, not its numbers: the same passage helps most at the edges of the prompt and least in the middle.</em></p></blockquote>

<h2 id="why-it-matters-for-builders">Why it matters for builders</h2>

<ol>
  <li><strong>Order is part of retrieval.</strong> Ranking isn’t just about <em>which</em> passages make the cut but <em>where</em> they land. Putting the best matches first (and not burying them under weaker ones) is cheap and directly addresses this. A <a href="/post/agent-memory-glossary/#re-ranking">re-ranking</a> pass earns its keep here.</li>
  <li><strong>Fewer, better passages beat more passages.</strong> The case study suggests a tighter <a href="/post/agent-memory-glossary/#top-k-retrieval">top-k</a> can be nearly as accurate and a lot cheaper. It also connects to the idea of <a href="/post/agent-memory-glossary/#sufficient-evidence">sufficient evidence</a>: the goal is enough context, not all of it.</li>
  <li><strong>Test position, not just recall.</strong> A memory system can retrieve the right fact and still fail if the fact lands mid-prompt. Benchmarks like <a href="/post/paper-tldr-locomo-long-term-memory-benchmark/">LoCoMo</a> measure the end answer; this paper’s trick of moving the gold passage around is an easy extra check to run on your own pipeline.</li>
  <li><strong>Keep the important stuff at the edges.</strong> Designs like <a href="/post/paper-tldr-memgpt/">MemGPT</a> keep a small, always-visible working context near the top of the prompt. This paper is one reason that kind of layout makes sense.</li>
</ol>

<h2 id="honest-caveats">Honest caveats</h2>

<ul>
  <li><strong>These are 2023 models.</strong> The results cover GPT-3.5, Claude-1.3, MPT and LongChat, with only partial GPT-4 runs because of cost. Newer models may show a weaker (or different) position effect, so it’s worth measuring on whatever you actually use.</li>
  <li><strong>The “why” is still open.</strong> The authors test a few explanations — architecture, instruction tuning, model size — and call those analyses preliminary. Encoder-decoder models were fairly robust within their training length but showed the U-shape on longer inputs; base models without instruction tuning showed it too.</li>
  <li><strong>One answer per prompt.</strong> Each test has a single relevant passage. Real questions often need several facts combined, which this setup doesn’t measure.</li>
  <li><strong>Greedy decoding only,</strong> and the key-value task is deliberately artificial, so it shows a capability gap rather than typical real-world behavior.</li>
</ul>

<p>Earlier in the series: <a href="/post/paper-tldr-memgpt/">MemGPT</a> (paging what goes into the context window), <a href="/post/paper-tldr-generative-agents/">Generative Agents</a> (ranking memories by recency, importance and relevance) and <a href="/post/paper-tldr-sentence-bert/">Sentence-BERT</a> (the embeddings behind most of the retrieval step). Next in the series: <a href="/post/paper-tldr-retrieval-augmented-generation/">RAG</a>, the 2020 paper that named the retrieve-then-generate setup this paper stress-tests.</p>

<p>Terms like <a href="/post/agent-memory-glossary/#context-window"><em>context window</em></a>, <a href="/post/agent-memory-glossary/#position-bias"><em>position bias</em></a>, <a href="/post/agent-memory-glossary/#re-ranking"><em>re-ranking</em></a> and <a href="/post/agent-memory-glossary/#top-k-retrieval"><em>top-k retrieval</em></a> are defined in the <a href="/post/agent-memory-glossary/">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><summary type="html"><![CDATA[A TL;DR of Lost in the Middle (Liu et al., 2023): models use facts at the start and end of a long prompt far better than the middle, and what it means for RAG.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-lost-in-the-middle.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-lost-in-the-middle.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Paper TL;DR: Generative Agents — Memory Streams, Reflection and Plans</title><link href="https://www.avinashgosavi.com/post/paper-tldr-generative-agents/" rel="alternate" type="text/html" title="Paper TL;DR: Generative Agents — Memory Streams, Reflection and Plans" /><published>2026-10-19T21:40:00+00:00</published><updated>2026-10-23T01:35:48+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-generative-agents</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-generative-agents/"><![CDATA[<p><strong>Paper:</strong> Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang &amp; Michael S. Bernstein (Stanford, Google Research, Google DeepMind), <a href="https://arxiv.org/abs/2304.03442"><em>Generative Agents: Interactive Simulacra of Human Behavior</em></a> (UIST 2023)</p>

<p><strong>One-line TL;DR:</strong> Give an LLM a running log of everything it observes, a way to score which memories matter right now, and a habit of periodically writing down higher-level conclusions — and simulated characters start to behave believably over days, not just single replies.</p>

<h2 id="the-setup">The setup</h2>

<p>The authors built <strong>Smallville</strong>, a small sandbox town populated by 25 characters. Each one is driven by an LLM (gpt-3.5-turbo in the paper) and starts from a single paragraph describing who they are: job, relationships, habits. From there the characters wake up, go to work, chat, form opinions of each other and make plans — all generated, step by step, by the model.</p>

<p>The interesting part isn’t the town. It’s the memory design that keeps each character consistent over time, which the paper splits into three pieces: a <strong>memory stream</strong>, <strong>reflection</strong> and <strong>planning</strong>.</p>

<h2 id="1-the-memory-stream">1. The memory stream</h2>

<p>Every character keeps a <strong>memory stream</strong>: a plain list of memories, each written in natural language with a creation time and a “last accessed” time. The basic memories are <strong>observations</strong> — things the character perceived, such as (to give an illustrative example) “a neighbor is watering the garden.”</p>

<p>The stream grows far too large to fit into a prompt, so each time a character needs to act, it retrieves only the most useful memories. Each memory gets three scores:</p>

<ul>
  <li><strong>Recency:</strong> recently used memories score higher, fading exponentially with each in-game hour (the paper uses a decay factor of 0.995).</li>
  <li><strong>Importance:</strong> when a memory is created, the model rates it from 1 (mundane, like cleaning a room) to 10 (poignant, like asking someone out).</li>
  <li><strong>Relevance:</strong> cosine similarity between the memory’s embedding and the current situation’s embedding.</li>
</ul>

<p>Each score is scaled to 0–1 and the three are added up (the paper weights them equally). The top-scoring memories that fit into the context window go into the prompt.</p>

<h2 id="2-reflection">2. Reflection</h2>

<p>Raw observations are too low-level to generalize from. Fifty separate observations of a neighbor reading in the library don’t, by themselves, tell a character “she cares a lot about her research.”</p>

<p>So characters <strong>reflect</strong>. When the summed importance of recent memories passes a threshold (150 in the paper — roughly two or three times per simulated day), the model looks at the 100 most recent memories, asks itself three high-level questions about them, retrieves memories relevant to each question, and writes down insights that cite the memories they came from.</p>

<p>Those insights go back into the same memory stream, where they can be retrieved — and reflected on again. The result is a <strong>reflection tree</strong>: raw observations at the leaves, more abstract conclusions higher up.</p>

<blockquote class="diagram-text"><p><strong>Diagram: How a generative agent remembers, retrieves and reflects</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-generative-agents/">see it in the post</a>)</p><p>Each character keeps a memory stream of natural-language entries with a creation time and a last-accessed time: observations, plans and reflections. When the character needs to act, every memory is scored on recency (decaying each game hour), importance (rated 1 to 10 by the model) and relevance (cosine similarity of embeddings); the scores are scaled to 0 to 1 and added. The top memories go into the prompt to choose the next action. When the importance of recent memories adds up past a threshold, the character reflects, and the insights it writes are stored back in the memory stream.</p><p><em>Everything lives in one stream. Retrieval ranks it three ways; reflection writes new, higher-level memories back into it (dashed loop).</em></p></blockquote>

<h2 id="3-planning">3. Planning</h2>

<p>To keep behavior coherent across a day, each character drafts a rough plan (about five to eight chunks), then breaks each chunk into hour-long actions, then into 5–15 minute steps. Plans are stored in the memory stream too. At every step, the model decides whether something it just noticed is worth reacting to; if so, it rewrites the plan from that point.</p>

<h2 id="what-they-tested">What they tested</h2>

<p><strong>Interviews.</strong> The authors “interviewed” characters with questions about themselves, their memories, plans, reactions and reflections. 100 human evaluators ranked answers from the full architecture, versions with pieces removed, and human crowdworkers role-playing the same characters. The full architecture ranked most believable; removing reflection, then planning, then memory each made answers noticeably worse. The crowdworker answers landed below every version that still had memory.</p>

<p><strong>Emergent behavior over two game days.</strong> One character was seeded with the idea of throwing a Valentine’s Day party. Without further scripting, word spread from 1 character to 13 of 25, and 5 of the 12 who were invited showed up. A mayoral candidacy spread from 1 character to 8 the same way.</p>

<h2 id="why-it-matters-for-builders">Why it matters for builders</h2>

<ol>
  <li><strong>Rank memories on more than similarity.</strong> Recency + importance + relevance is a simple, readable scoring rule, and variations of it show up across agent memory systems. <a href="/post/vaultmem-encrypted-memory-for-ai-agents/">VaultMem</a>, for example, re-ranks on cosine similarity plus a significance score that decays over time.</li>
  <li><strong>Store conclusions, not just events.</strong> Reflections are cheap summaries the agent can retrieve later, and keeping links back to the source memories makes them auditable.</li>
  <li><strong>Memory and planning are the same store.</strong> Writing plans into the memory stream means “what am I doing next?” is answered by the same retrieval as “what happened?”</li>
</ol>

<h2 id="honest-caveats">Honest caveats</h2>

<ul>
  <li><strong>Retrieval still misses.</strong> The authors show characters failing to recall something they had heard, or recalling a detail of an event but not the event itself.</li>
  <li><strong>Embellishment.</strong> Characters sometimes added details nobody had said, or blended in the model’s world knowledge — one described a neighbor as the author of a famous 18th-century economics book.</li>
  <li><strong>Too polite.</strong> The authors attribute overly formal, overly agreeable dialogue to instruction tuning; one character rarely said no to suggestions, and her own interests drifted as a result.</li>
  <li><strong>Expensive.</strong> Running 25 characters for two game days took multiple days and, in the authors’ words, “thousands of dollars in token credits.”</li>
  <li><strong>Believability isn’t accuracy.</strong> The evaluation measured whether behavior <em>seemed</em> human, not whether memories were retrieved correctly. Benchmarks like <a href="/post/paper-tldr-locomo-long-term-memory-benchmark/">LoCoMo</a> test that more directly.</li>
</ul>

<p>Earlier in the series: <a href="/post/paper-tldr-memgpt/">MemGPT</a> (an OS-style alternative where the model pages its own memory), <a href="/post/paper-tldr-complementary-learning-systems/">Complementary Learning Systems</a> (the brain’s version of turning episodes into general knowledge — close in spirit to reflection), and <a href="/post/paper-tldr-sentence-bert/">Sentence-BERT</a> (the kind of embeddings behind the relevance score). Next in the series: <a href="/post/paper-tldr-lost-in-the-middle/">Lost in the Middle</a>, on what happens to retrieved text once it lands in a long prompt.</p>

<p>Terms like <a href="/post/agent-memory-glossary/#reflection"><em>reflection</em></a>, <a href="/post/agent-memory-glossary/#consolidation"><em>consolidation</em></a> and <a href="/post/agent-memory-glossary/#decay"><em>decay</em></a> are defined in the <a href="/post/agent-memory-glossary/#how-memories-change-over-time">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><summary type="html"><![CDATA[A TL;DR of Generative Agents (Park et al., 2023): how 25 LLM characters in a sandbox town remember, reflect and plan, and which parts builders still copy.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-generative-agents.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-generative-agents.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Paper TL;DR: MemGPT — Treating the Context Window Like RAM</title><link href="https://www.avinashgosavi.com/post/paper-tldr-memgpt/" rel="alternate" type="text/html" title="Paper TL;DR: MemGPT — Treating the Context Window Like RAM" /><published>2026-10-19T10:40:00+00:00</published><updated>2026-10-23T01:35:48+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-memgpt</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-memgpt/"><![CDATA[<p><strong>Paper:</strong> Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica &amp; Joseph E. Gonzalez (UC Berkeley), <a href="https://arxiv.org/abs/2310.08560"><em>MemGPT: Towards LLMs as Operating Systems</em></a> (arXiv, 2023)</p>

<p><strong>One-line TL;DR:</strong> Instead of waiting for bigger context windows, MemGPT treats the prompt like a computer’s RAM and lets the model itself move information between that small fast space and larger external storage, using function calls.</p>

<h2 id="the-problem">The problem</h2>

<p>An LLM can only “see” what fits in its context window. Long chats and big document collections don’t fit, so something has to decide what stays in view. Operating systems solved a version of this decades ago: programs act as if they have lots of memory, while the OS quietly pages data between fast RAM and slower disk. The paper calls its version <strong>virtual context management</strong>.</p>

<h2 id="how-it-works">How it works</h2>

<p>MemGPT splits memory into two places.</p>

<p><strong>Main context</strong> — what’s actually in the prompt — has three parts:</p>

<ul>
  <li><strong>System instructions:</strong> read-only rules, including how to use the memory functions.</li>
  <li><strong>Working context:</strong> a small, fixed-size scratchpad of key facts (for example, things learned about the user). The model can only change it through function calls.</li>
  <li><strong>A FIFO message queue:</strong> the recent conversation. Its first entry is a running summary of older messages that were pushed out.</li>
</ul>

<p><strong>External context</strong> lives outside the prompt:</p>

<ul>
  <li><strong>Recall storage:</strong> the full message history, searchable on demand.</li>
  <li><strong>Archival storage:</strong> a store for arbitrary text the model chooses to save or look up later.</li>
</ul>

<blockquote class="diagram-text"><p><strong>Diagram: MemGPT's two tiers of memory</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-memgpt/">see it in the post</a>)</p><p>Main context is what is in the prompt: read-only system instructions, a small working context of key facts edited through function calls, and a FIFO message queue that starts with a summary of older messages. External context sits outside the prompt: recall storage holds the full message history and archival storage holds any text the model chooses to save. The oldest messages are evicted from the queue into recall storage, and the model searches and saves across both tiers with function calls.</p><p><em>Main context is the prompt; external context is everything the model can page back in. The model itself moves information between them.</em></p></blockquote>

<p>The clever part is who moves things around: <strong>the model does</strong>. Its output is parsed as function calls (search recall, write to archival, edit working context), and the results are fed back in. A few mechanics make this work:</p>

<ul>
  <li><strong>Memory pressure warnings.</strong> When the prompt gets close to full (the paper’s example uses 70%), the system adds a warning so the model can save what matters before it’s lost.</li>
  <li><strong>Eviction with summaries.</strong> When the prompt is full, a chunk of the oldest messages is evicted and folded into the running summary. Evicted messages aren’t deleted — they stay in recall storage.</li>
  <li><strong>Heartbeats.</strong> A function call can ask to keep going (<code class="language-plaintext highlighter-rouge">request_heartbeat</code>), so the model can chain several lookups before replying.</li>
  <li><strong>Events.</strong> User messages, system alerts and scheduled timers all wake the agent up, so it doesn’t only act when a person types.</li>
</ul>

<h2 id="what-they-tested">What they tested</h2>

<p>Two settings, each with a “can it really remember?” task:</p>

<ul>
  <li><strong>Multi-session chat.</strong> In a deep memory retrieval task, the agent is asked about something from an earlier session. MemGPT answered far more of these correctly than the same models with a fixed context. In a “conversation opener” task, its openers drew on what it knew about the person, and the authors found that saving facts to working context was key.</li>
  <li><strong>Document analysis.</strong> For question answering over many documents, MemGPT could page through more search results than fit in one prompt. In a nested key-value lookup task, where one value points to another key, fixed-context models broke down as nesting got deeper, while MemGPT with GPT-4 kept answering by looking things up repeatedly.</li>
</ul>

<h2 id="why-it-matters-for-builders">Why it matters for builders</h2>

<ol>
  <li><strong>Memory can be a set of tools, not just a bigger prompt.</strong> Giving the model explicit read/write/search functions makes memory something you can inspect and log.</li>
  <li><strong>Separate “always in view” from “look it up.”</strong> A small working context of key facts plus searchable history is a pattern that shows up again and again in agent memory designs.</li>
  <li><strong>Plan for overflow.</strong> Warning before the window fills, and summarizing instead of silently truncating, is a simple idea worth copying.</li>
</ol>

<h2 id="honest-caveats">Honest caveats</h2>

<ul>
  <li><strong>It leans on the model’s function-calling skill.</strong> The authors report that results dropped noticeably with GPT-3.5, which was weaker at calling functions.</li>
  <li><strong>Retrieval is still the bottleneck.</strong> With embedding search, the right document often wasn’t near the top, and the agent tended to stop paging before exhausting the results.</li>
  <li><strong>More model isn’t always better.</strong> On the nested lookup task, the GPT-4 Turbo version did worse than GPT-4, and some versions gave up after too few lookups.</li>
  <li><strong>Self-managed memory is a judgment call by the model.</strong> What gets saved, summarized or forgotten depends on the model’s choices, which is harder to predict than a fixed rule.</li>
</ul>

<p>Earlier in the series: <a href="/post/paper-tldr-locomo-long-term-memory-benchmark/">LoCoMo</a> (a benchmark for exactly this kind of long-term memory), <a href="/post/paper-tldr-sentence-bert/">Sentence-BERT</a> (the embeddings behind most semantic search), and <a href="/post/paper-tldr-complementary-learning-systems/">Complementary Learning Systems</a> (a brain-inspired split between fast and slow memory). Next in the series: <a href="/post/paper-tldr-generative-agents/">Generative Agents</a>, which ranks a long memory stream by recency, importance and relevance.</p>

<p>Terms like <a href="/post/agent-memory-glossary/#virtual-context"><em>virtual context</em></a>, <a href="/post/agent-memory-glossary/#context-window"><em>context window</em></a>, <a href="/post/agent-memory-glossary/#episodic-memory"><em>episodic memory</em></a> and <a href="/post/agent-memory-glossary/#top-k-retrieval"><em>top-k retrieval</em></a> are defined in the <a href="/post/agent-memory-glossary/">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><summary type="html"><![CDATA[A TL;DR of MemGPT (Packer et al., 2023): how an LLM can page information in and out of a small context window with function calls, like an operating system.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-memgpt.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-memgpt.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">An Agent Memory Glossary: Terms in Plain Language</title><link href="https://www.avinashgosavi.com/post/agent-memory-glossary/" rel="alternate" type="text/html" title="An Agent Memory Glossary: Terms in Plain Language" /><published>2026-10-19T02:45:00+00:00</published><updated>2026-10-23T01:35:48+00:00</updated><id>https://www.avinashgosavi.com/post/agent-memory-glossary</id><content type="html" xml:base="https://www.avinashgosavi.com/post/agent-memory-glossary/"><![CDATA[<p>The vocabulary around AI agent memory borrows from neuroscience, databases, search and cryptography all at once, which makes it easy to talk past each other. This is a working glossary — short, plain definitions, with links to where each idea shows up in my own projects. I’ll keep adding to it.</p>

<h2 id="kinds-of-memory">Kinds of memory</h2>

<p id="context-window" class="glossary-term"><strong>Context window (working memory).</strong> What the model can see right now: the prompt, the recent conversation and anything retrieved for this turn. Fast and precise, but limited in size and gone when the session ends. Everything else on this list exists because of that limit.</p>

<p id="episodic-memory" class="glossary-term"><strong>Episodic memory.</strong> Specific events: <em>what happened, when, with whom.</em> “On Tuesday you said the demo slipped to Friday.” In <a href="/post/vaultmem-encrypted-memory-for-ai-agents/">VaultMem</a> these are one of the memory types the agent can store and retrieve.</p>

<p id="semantic-memory" class="glossary-term"><strong>Semantic memory.</strong> General facts and knowledge, detached from when they were learned. “Flash Attention reduces memory traffic by tiling.” Often distilled from many episodes.</p>

<p id="procedural-memory" class="glossary-term"><strong>Procedural memory.</strong> How to do things: workflows, preferences about process, steps that worked before. “When debugging, check the logs before the config.”</p>

<p id="persona-memory" class="glossary-term"><strong>Persona (profile) memory.</strong> Stable facts about the person the agent works with — name, role, standing preferences. Small, high-value and read on almost every turn.</p>

<p id="virtual-context" class="glossary-term"><strong>Virtual context (paging).</strong> Treating the context window like a computer’s RAM: the agent moves information in and out of the prompt — saving facts to external storage, searching old messages, summarizing what gets pushed out — usually through function calls. Popularized by <a href="/post/paper-tldr-memgpt/">MemGPT</a>.</p>

<h2 id="storing-and-finding-memories">Storing and finding memories</h2>

<p id="embedding" class="glossary-term"><strong>Embedding.</strong> A list of numbers that represents the meaning of a piece of text, so that similar meanings land close together. Most memory systems store an embedding next to each memory. See the <a href="/post/paper-tldr-sentence-bert/">Sentence-BERT TL;DR</a> for where modern sentence embeddings come from.</p>

<p id="semantic-search" class="glossary-term"><strong>Semantic search.</strong> Finding memories by meaning instead of exact words: embed the query, then return the stored memories whose embeddings are closest.</p>

<p id="top-k-retrieval" class="glossary-term"><strong>Top-k retrieval.</strong> Returning the <em>k</em> closest matches (say, the top 10) and putting them into the context window. Simple and effective; also where most retrieval mistakes start.</p>

<p id="rag" class="glossary-term"><strong>Retrieval-augmented generation (RAG).</strong> Search first, then write: fetch the passages most relevant to the input and have the model generate its answer from them, instead of relying only on what’s stored in its weights. The term comes from a 2020 paper that also showed you can update what the system knows by swapping the search index — see the <a href="/post/paper-tldr-retrieval-augmented-generation/">RAG TL;DR</a>.</p>

<p id="hnsw" class="glossary-term"><strong>Vector index (HNSW).</strong> A data structure that makes “find the closest vectors” fast at scale. HNSW (Hierarchical Navigable Small World) builds layered graphs of neighbors so a search can hop quickly toward the right region instead of comparing against every vector. (<a href="/post/paper-tldr-hnsw/">Paper TL;DR</a>)</p>

<p id="re-ranking" class="glossary-term"><strong>Re-ranking.</strong> A second, slower and more careful pass over the top results — often a cross-encoder that reads the query and each candidate together — to put the truly relevant ones first.</p>

<p id="position-bias" class="glossary-term"><strong>Position bias (lost in the middle).</strong> Models tend to use information at the start and end of a long prompt better than information buried in the middle, so <em>where</em> retrieved memories land in the prompt affects whether they get used. Putting the strongest matches first, and keeping top-k small, helps. See the <a href="/post/paper-tldr-lost-in-the-middle/">Lost in the Middle TL;DR</a>.</p>

<p id="hybrid-search" class="glossary-term"><strong>Hybrid search.</strong> Combining meaning-based search with keyword search (or filters like dates and tags), because embeddings are bad at exact names, numbers and IDs.</p>

<p id="context-frame" class="glossary-term"><strong>Context frame.</strong> The situation a memory was formed in — topic, people involved, task, session — stored alongside it so retrieval can filter by context, not just content. This is the core idea in <a href="/post/attractor-memory-engine/">Attractor</a>.</p>

<p id="sufficient-evidence" class="glossary-term"><strong>Sufficient evidence.</strong> The idea that a memory system’s job isn’t just to return <em>relevant</em> memories but a set that is <em>enough</em> to answer. Ranking memories one at a time can miss this: several records may say the same thing (so the set gets bloated), or two weak-looking records may only make sense together (so both get dropped). One recent paper defines a sufficient set as a small subset that leads the agent to the same decision it would make with its full memory (<a href="https://neurips.cc/virtual/2026/169578">Beyond Marginal Influence</a>).</p>

<h2 id="how-memories-change-over-time">How memories change over time</h2>

<p id="consolidation" class="glossary-term"><strong>Consolidation.</strong> Turning many specific episodes into durable, general knowledge — summarizing a month of conversations into “prefers short answers,” for example. The brain’s version is covered in the <a href="/post/paper-tldr-complementary-learning-systems/">Complementary Learning Systems TL;DR</a>.</p>

<p id="reflection" class="glossary-term"><strong>Reflection.</strong> Having the agent periodically read its recent memories and write down higher-level conclusions (“she cares a lot about her research”), stored back as memories that point to the evidence they came from. Introduced for LLM agents in <a href="/post/paper-tldr-generative-agents/">Generative Agents</a>, which also ranks memories by recency, importance and relevance together.</p>

<p id="decay" class="glossary-term"><strong>Decay (forgetting).</strong> Letting memories that are never used fade in importance, so old noise doesn’t crowd out what matters. Forgetting is a feature.</p>

<p id="event-time" class="glossary-term"><strong>Event time vs. write time.</strong> Two different timestamps for one memory: when the thing happened, and when it was saved. A photo from 2019 imported today should come back for “what did I do in 2019?”, not for “what did I add today?”. <a href="/post/vaultmem-part-2-multimodal-temporal-backends/#temporal-search">VaultMem Part 2</a> keeps them as separate fields (<code class="language-plaintext highlighter-rouge">captured_at</code> and <code class="language-plaintext highlighter-rouge">created_at</code>) for exactly this reason.</p>

<p id="versioned-facts" class="glossary-term"><strong>Versioned facts (temporal state).</strong> Keeping every value a fact has had — “lives in Boston”, then “lives in Seattle” — as a list of versions linked to where each came from, instead of overwriting the old one. The newest version answers “where does she live?”, while the older ones still answer “where did she live before?”. The <a href="https://www.emergentmind.com/papers/2610.21900">TEM paper</a> adds one more step: older versions are marked as historical before the model sees them, and the authors report that this label alone gave large gains on questions about repeatedly updated facts.</p>

<p id="associative-memory" class="glossary-term"><strong>Associative (content-addressable) memory.</strong> Retrieving a whole memory from a partial or noisy piece of it, rather than by key or address. Hopfield networks are the classic example — see <a href="/post/paper-tldr-hopfield-1982/">Hopfield (1982)</a> and <a href="/post/paper-tldr-hopfield-networks-is-all-you-need/">the modern version</a>.</p>

<h2 id="privacy">Privacy</h2>

<p id="zero-knowledge-storage" class="glossary-term"><strong>Zero-knowledge storage.</strong> The service storing your data cannot read it, because it’s encrypted with a key only you hold. The operator sees ciphertext. This is VaultMem’s core design (<a href="/post/vaultmem-your-ai-agent-shouldnt-read-your-diary/">why it matters, in plain language</a>).</p>

<p id="client-side-encryption" class="glossary-term"><strong>Client-side encryption.</strong> Encrypting data before it leaves your device (or your session), so the server only ever handles encrypted bytes. The usual way to get zero-knowledge storage.</p>

<p id="pii-sanitization" class="glossary-term"><strong>PII sanitization.</strong> Detecting and masking personal details — names, emails, phone numbers — before text is sent to a model provider, then restoring them in the response. It closes a gap encryption at rest can’t: the model still has to read the text. See <a href="/post/vaultmem-part-3-pii-sanitizer/">VaultMem Part 3</a>.</p>

<p id="memory-unlearning" class="glossary-term"><strong>Memory unlearning (forget requests).</strong> Deleting a specific fact when the user asks — and only that fact. Done well, the fact can no longer be recalled or pieced together from other memories, while everything unrelated survives. The <a href="https://www.emergentmind.com/papers/2610.18052">Bemula benchmark</a> tests exactly this and reports two common failures: the fact stays extractable with simple prompts, or the system deletes far more than it was asked to.</p>

<h2 id="measuring-it">Measuring it</h2>

<p id="recall-at-k" class="glossary-term"><strong>Recall@k.</strong> Of the questions asked, how often the right memory appears somewhere in the top <em>k</em> results. Measures “did we find it at all.”</p>

<p id="mrr" class="glossary-term"><strong>MRR (mean reciprocal rank).</strong> Rewards putting the right memory near the top: 1 if it’s first, ½ if second, ⅓ if third, averaged over all questions.</p>

<p id="locomo" class="glossary-term"><strong>LoCoMo.</strong> A benchmark of very long, multi-session conversations with questions across single-hop, multi-hop, temporal, open-domain and adversarial categories. See the <a href="/post/paper-tldr-locomo-long-term-memory-benchmark/">LoCoMo TL;DR</a>.</p>

<hr />

<p>Missing a term, or think one of these is off? I’d like to hear it — I’ll keep this page up to date.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Agent memory" /><category term="Explainers" /><summary type="html"><![CDATA[Plain-language definitions of the terms that come up when building memory for AI agents — episodic vs. semantic memory, embeddings, HNSW, re-ranking and more.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/agent-memory-glossary.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/agent-memory-glossary.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Paper TL;DR: Hopfield (1982) — Memory as a Valley You Roll Into</title><link href="https://www.avinashgosavi.com/post/paper-tldr-hopfield-1982/" rel="alternate" type="text/html" title="Paper TL;DR: Hopfield (1982) — Memory as a Valley You Roll Into" /><published>2026-10-18T22:00:00+00:00</published><updated>2026-10-23T01:35:48+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-hopfield-1982</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-hopfield-1982/"><![CDATA[<p><strong>Paper:</strong> J. J. Hopfield, <a href="https://www.pnas.org/doi/10.1073/pnas.79.8.2554"><em>Neural networks and physical systems with emergent collective computational abilities</em></a>, PNAS 79(8), 1982</p>

<p><strong>One-line TL;DR:</strong> connect lots of simple on/off neurons with symmetric weights and the network gets an energy that only goes down as it runs — so stored memories become stable valleys, and a partial or noisy cue rolls downhill into the full memory.</p>

<p>Almost everything I’ve built around memory started here. It’s also one of the most readable classic papers I know: five pages, very little math, a physicist explaining computation.</p>

<h2 id="the-setup">The setup</h2>

<ul>
  <li><strong>Neurons</strong> are either on or off (+1 or −1).</li>
  <li><strong>Every neuron connects to every other</strong>, and the connection strengths are <strong>symmetric</strong>: the weight from A to B equals the weight from B to A.</li>
  <li><strong>Storing a memory</strong> uses a Hebbian rule — “neurons that fire together wire together.” For each pattern you want to remember, strengthen connections between neurons that agree and weaken those between neurons that disagree. Store several patterns by adding up their contributions.</li>
  <li><strong>Recalling</strong> means picking neurons one at a time and flipping each to agree with the weighted vote of its neighbors, until nothing changes.</li>
</ul>

<h2 id="the-key-insight-an-energy-that-only-goes-down">The key insight: an energy that only goes down</h2>

<p>Hopfield’s move was to define an <strong>energy</strong> for the whole network’s state and show that, with symmetric weights and one-at-a-time updates, every update either lowers the energy or leaves it unchanged. The network can’t cycle forever; it has to settle.</p>

<p>Stored patterns sit at the bottoms of valleys in that energy landscape. Start anywhere nearby — a corrupted photo, half a phone number — and the network slides down into the nearest stored memory. That’s <strong>content-addressable memory</strong>: you retrieve by <em>what</em> you remember, not by <em>where</em> it’s stored. It’s also robust: knock out some neurons or connections and recall still mostly works.</p>

<blockquote class="diagram-text"><p><strong>Diagram: Storing and recalling a memory in a Hopfield network</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-hopfield-1982/">see it in the post</a>)</p><p>Storing: the Hebbian rule strengthens connections between neurons that agree, and several patterns are stored by adding up their contributions. Recalling: start from a noisy cue in which some neurons are flipped; neurons update one at a time to agree with the weighted vote of their neighbors, so the wrong ones flip back until the stored pattern reappears and nothing changes. Underneath, the network's energy only goes down or stays the same with each update, like rolling downhill into a valley, and it settles at the bottom, which is the stored memory.</p><p><em>Illustration with 25 neurons: store with the Hebbian rule, then a noisy cue rolls downhill into the stored pattern.</em></p></blockquote>

<h2 id="twenty-five-lines-of-numpy">Twenty-five lines of NumPy</h2>

<p>Store five random patterns in 100 neurons, flip 30% of the bits of one, and let it settle:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">numpy</span> <span class="k">as</span> <span class="n">np</span>

<span class="n">rng</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="nf">default_rng</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">N</span><span class="p">,</span> <span class="n">P</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">5</span>
<span class="n">patterns</span> <span class="o">=</span> <span class="n">rng</span><span class="p">.</span><span class="nf">choice</span><span class="p">([</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="n">P</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
<span class="n">W</span> <span class="o">=</span> <span class="p">(</span><span class="n">patterns</span><span class="p">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">patterns</span><span class="p">)</span> <span class="o">/</span> <span class="n">N</span>        <span class="c1"># Hebbian storage
</span><span class="n">np</span><span class="p">.</span><span class="nf">fill_diagonal</span><span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>                 <span class="c1"># no self-connections
</span>
<span class="k">def</span> <span class="nf">energy</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
    <span class="k">return</span> <span class="o">-</span><span class="mf">0.5</span> <span class="o">*</span> <span class="n">s</span> <span class="o">@</span> <span class="n">W</span> <span class="o">@</span> <span class="n">s</span>

<span class="k">def</span> <span class="nf">recall</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">sweeps</span><span class="o">=</span><span class="mi">5</span><span class="p">):</span>
    <span class="n">s</span> <span class="o">=</span> <span class="n">s</span><span class="p">.</span><span class="nf">copy</span><span class="p">()</span>
    <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="n">sweeps</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">rng</span><span class="p">.</span><span class="nf">permutation</span><span class="p">(</span><span class="n">N</span><span class="p">):</span>   <span class="c1"># one neuron at a time
</span>            <span class="n">s</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">W</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">@</span> <span class="n">s</span> <span class="o">&gt;=</span> <span class="mi">0</span> <span class="k">else</span> <span class="o">-</span><span class="mi">1</span>
    <span class="k">return</span> <span class="n">s</span>

<span class="n">target</span> <span class="o">=</span> <span class="n">patterns</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">noisy</span> <span class="o">=</span> <span class="n">target</span><span class="p">.</span><span class="nf">copy</span><span class="p">()</span>
<span class="n">noisy</span><span class="p">[</span><span class="n">rng</span><span class="p">.</span><span class="nf">choice</span><span class="p">(</span><span class="n">N</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="n">replace</span><span class="o">=</span><span class="bp">False</span><span class="p">)]</span> <span class="o">*=</span> <span class="o">-</span><span class="mi">1</span>

<span class="n">out</span> <span class="o">=</span> <span class="nf">recall</span><span class="p">(</span><span class="n">noisy</span><span class="p">)</span>
<span class="nf">print</span><span class="p">((</span><span class="n">noisy</span> <span class="o">==</span> <span class="n">target</span><span class="p">).</span><span class="nf">mean</span><span class="p">(),</span> <span class="p">(</span><span class="n">out</span> <span class="o">==</span> <span class="n">target</span><span class="p">).</span><span class="nf">mean</span><span class="p">())</span>   <span class="c1"># 0.7 -&gt; 1.0
</span><span class="nf">print</span><span class="p">(</span><span class="nf">round</span><span class="p">(</span><span class="nf">energy</span><span class="p">(</span><span class="n">noisy</span><span class="p">),</span> <span class="mi">1</span><span class="p">),</span> <span class="nf">round</span><span class="p">(</span><span class="nf">energy</span><span class="p">(</span><span class="n">out</span><span class="p">),</span> <span class="mi">1</span><span class="p">))</span>     <span class="c1"># -8.6 -&gt; -50.5
</span></code></pre></div></div>

<p>70% correct goes to 100% correct, and the energy drops as it does.</p>

<h2 id="the-limits">The limits</h2>

<ul>
  <li><strong>Capacity is small.</strong> Hopfield found the network reliably holds only about <strong>0.15 memories per neuron</strong>. Push past that and recall falls apart.</li>
  <li><strong>Spurious memories.</strong> The landscape also grows valleys nobody stored — blends of real memories — and a cue can get stuck in one.</li>
  <li><strong>Similar memories interfere.</strong> Patterns that overlap a lot blur into each other.</li>
</ul>

<p>Those limits are exactly what the 2020 follow-up fixes: <a href="/post/paper-tldr-hopfield-networks-is-all-you-need/">modern Hopfield networks</a> sharpen the valleys so capacity grows exponentially — and their update turns out to be transformer attention.</p>

<h2 id="why-it-still-matters-to-me">Why it still matters to me</h2>

<p>Databases retrieve by key. Vector stores retrieve by nearest neighbor. Hopfield networks retrieve by <strong>settling</strong> — pattern completion from a partial cue, which is much closer to how remembering feels. That idea is the reason <a href="/post/attractor-memory-engine/">Attractor</a> is called Attractor.</p>

<p>That completes the six papers from <a href="/post/paper-tldr-six-papers-on-memory/">Part 1</a>. The whole list, with short TL;DRs, lives on my <a href="/research/#papershelf">papershelf</a>. The series keeps going with <a href="/post/paper-tldr-memgpt/">MemGPT</a>, which treats an LLM’s context window like an operating system’s memory.</p>

<p>For a one-paragraph definition of <a href="/post/agent-memory-glossary/#associative-memory"><em>associative (content-addressable) memory</em></a>, see the <a href="/post/agent-memory-glossary/">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><summary type="html"><![CDATA[A TL;DR of John Hopfield's 1982 paper on associative memory: energy landscapes, Hebbian storage, recall from partial cues and its limits — with a NumPy demo.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-hopfield-1982.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-hopfield-1982.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Paper TL;DR: Sentence-BERT — The Quiet Workhorse Behind Semantic Search</title><link href="https://www.avinashgosavi.com/post/paper-tldr-sentence-bert/" rel="alternate" type="text/html" title="Paper TL;DR: Sentence-BERT — The Quiet Workhorse Behind Semantic Search" /><published>2026-10-18T21:00:00+00:00</published><updated>2026-10-23T12:36:36+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-sentence-bert</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-sentence-bert/"><![CDATA[<p><strong>Paper:</strong> Nils Reimers &amp; Iryna Gurevych, <a href="https://arxiv.org/abs/1908.10084"><em>Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks</em></a> (EMNLP 2019)</p>

<p><strong>One-line TL;DR:</strong> fine-tune BERT so that every sentence gets its own fixed-size embedding you can compare with cosine similarity — turning semantic search over thousands of sentences from hours into seconds.</p>

<p>It’s the least flashy paper in this series and the one my code depends on most. Every time <a href="/post/attractor-memory-engine/">Attractor</a>, <a href="/series/#vaultmem">VaultMem</a> or <a href="/post/daylog-security-architecture/">daylog</a> “searches by meaning,” it’s using a model descended from this paper.</p>

<h2 id="the-problem-bert-is-a-slow-judge">The problem: BERT is a slow judge</h2>

<p>In 2019, BERT was the best tool for deciding whether two sentences meant the same thing — but only if you fed it <strong>both sentences together</strong>. That works for scoring one pair. It falls apart for search.</p>

<p>The paper’s example: to find the most similar pair in a collection of 10,000 sentences, BERT would need around 50 million pairwise passes — roughly <strong>65 hours</strong> on a GPU. And you can’t precompute anything, because every score depends on both sentences at once.</p>

<p>The obvious shortcut — run each sentence through BERT alone and average its outputs (or take the special <code class="language-plaintext highlighter-rouge">[CLS]</code> token) — produced embeddings that were often <em>worse</em> than old-fashioned averaged word vectors.</p>

<h2 id="the-fix-train-it-to-embed">The fix: train it to embed</h2>

<p>Sentence-BERT puts two copies of the same BERT side by side (a <strong>siamese</strong> network — shared weights), runs one sentence through each, pools each output into a single vector, and trains the whole thing so that:</p>

<ul>
  <li>sentences that mean similar things land close together, and</li>
  <li>sentences that don’t land far apart.</li>
</ul>

<p>They trained on natural-language-inference data (pairs labeled entailment / contradiction / neutral) and on similarity-scored pairs. Afterward you throw away the “pair” setup: each sentence goes through the model <strong>once</strong>, you store its vector, and comparing two sentences is a cheap cosine similarity.</p>

<p>That same 10,000-sentence search drops to about <strong>5 seconds</strong>.</p>

<blockquote class="diagram-text"><p><strong>Diagram: Plain BERT versus Sentence-BERT for comparing sentences</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-sentence-bert/">see it in the post</a>)</p><p>Top: plain BERT reads both sentences together and outputs one similarity score for that pair, so nothing can be precomputed; finding the most similar pair among 10,000 sentences takes about 50 million passes, roughly 65 hours on a GPU. Bottom: Sentence-BERT runs each sentence through BERT on its own, with two copies sharing the same weights, pools the output into one vector per sentence, and compares the two vectors with cosine similarity. Each vector is computed once and stored, so the same search takes about 5 seconds.</p><p><em>Plain BERT has to re-read every pair together. Sentence-BERT gives each sentence its own vector once, so comparing two is just a cosine similarity.</em></p></blockquote>

<h2 id="why-it-still-matters">Why it still matters</h2>

<ul>
  <li><strong>It made vector search practical.</strong> Embed once, store in an index (pgvector, Qdrant, <a href="/post/paper-tldr-hnsw/">HNSW</a>), and query in milliseconds. That’s the foundation of nearly every <a href="/post/paper-tldr-retrieval-augmented-generation/">RAG</a> system and agent memory store.</li>
  <li><strong>The library outlived the paper.</strong> The <code class="language-plaintext highlighter-rouge">sentence-transformers</code> package that came out of this work is still how most people get embedding models running in a few lines.</li>
  <li><strong>Small models are often enough.</strong> The 384-dimension model I use in <a href="/post/attractor-memory-engine/">Attractor</a> runs comfortably on a CPU — handy when the whole point is <a href="/uses/">keeping data on hardware you own</a>.</li>
</ul>

<h2 id="caveats-worth-knowing">Caveats worth knowing</h2>

<ul>
  <li>Bi-encoders like SBERT trade a little accuracy for speed. When precision matters, a common pattern is to retrieve with embeddings, then <strong>re-rank</strong> the top results with a slower cross-encoder.</li>
  <li>“Similar meaning” is whatever the training data said it was. Domain-specific text (logs, configs, medical notes) often needs a domain-tuned model.</li>
  <li>Embeddings find <em>related</em> text; they don’t reason. Multi-hop questions — the ones <a href="/post/paper-tldr-locomo-long-term-memory-benchmark/">LoCoMo</a> showed are hardest — need more than nearest neighbors.</li>
</ul>

<p>Earlier in the series: <a href="/post/paper-tldr-six-papers-on-memory/">six papers on AI memory</a>, <a href="/post/paper-tldr-hopfield-networks-is-all-you-need/">Hopfield Networks Is All You Need</a>, <a href="/post/paper-tldr-complementary-learning-systems/">Complementary Learning Systems</a>, <a href="/post/paper-tldr-locomo-long-term-memory-benchmark/">LoCoMo</a> and <a href="/post/paper-tldr-memory-as-asset/">Memory as Asset</a>. Next in the series: <a href="/post/paper-tldr-hopfield-1982/">Hopfield (1982)</a>, the original associative memory that recalls from a partial cue.</p>

<p><a href="/post/agent-memory-glossary/#embedding"><em>Embeddings</em></a>, <a href="/post/agent-memory-glossary/#hnsw"><em>HNSW</em></a> and <a href="/post/agent-memory-glossary/#re-ranking"><em>re-ranking</em></a> each get a short definition in the <a href="/post/agent-memory-glossary/#storing-and-finding-memories">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><summary type="html"><![CDATA[A TL;DR of Sentence-BERT (Reimers & Gurevych, 2019): why BERT is too slow for search, how siamese training fixes it, and why it still powers semantic search.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-sentence-bert.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-sentence-bert.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Paper TL;DR: Memory as Asset — Whose Memory Is It Anyway?</title><link href="https://www.avinashgosavi.com/post/paper-tldr-memory-as-asset/" rel="alternate" type="text/html" title="Paper TL;DR: Memory as Asset — Whose Memory Is It Anyway?" /><published>2026-10-18T20:00:00+00:00</published><updated>2026-10-24T05:39:10+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-memory-as-asset</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-memory-as-asset/"><![CDATA[<p><strong>Paper:</strong> Yanqi Pan, Qinghao Huang &amp; Weihao Yang, <a href="https://arxiv.org/abs/2603.14212"><em>Memory as Asset: From Agent-centric to Human-centric Memory Management</em></a> (arXiv, March 2026)</p>

<p><strong>One-line TL;DR:</strong> the memory an AI keeps about you is designed today as the agent’s internal resource; this paper argues it should be <em>your</em> asset — owned by you, shared only under your permissions, and able to grow over time.</p>

<p>This is the paper that reshaped <a href="/series/#vaultmem">VaultMem</a>. I found it while I was stuck on a vague feeling that something about agent memory was backwards, and it gave that feeling a name.</p>

<h2 id="the-problem-agent-centric-memory">The problem: agent-centric memory</h2>

<p>Most memory systems treat what an agent remembers about you as part of the agent — stored by the platform, shaped for the platform’s use, and stuck there. The authors call this <strong>agent-centric</strong> memory. The consequences are familiar: you can’t take your history with you, you can’t decide who sees it, and every product rebuilds its own isolated “memory island” about the same person.</p>

<h2 id="the-proposal-memory-as-a-personal-asset">The proposal: memory as a personal asset</h2>

<p>They propose flipping the default, built on three principles:</p>

<ul>
  <li><strong>Memory in hand</strong> — individuals own and control their memories, not centralized platforms.</li>
  <li><strong>Memory groups</strong> — memories can be shared and combined, but only through permissions the owner sets.</li>
  <li><strong>Collective memory evolution</strong> — memory isn’t frozen: it’s recorded, consolidated and abstracted into higher-level knowledge over time, and can be exchanged between groups under control.</li>
</ul>

<p>Each memory object carries its content, context, an explicit <strong>owner</strong>, an <strong>access policy</strong> and a <strong>version</strong>. Ownership and permissions are part of the data itself, not an afterthought in some admin panel.</p>

<h2 id="the-three-layers">The three layers</h2>

<ol>
  <li><strong>Personal storage</strong> — fast storage of each person’s private memory objects.</li>
  <li><strong>Evolution</strong> — updates memory from interactions: recording experiences, merging related memories, abstracting knowledge. It can run locally for privacy or remotely for compute.</li>
  <li><strong>Memory exchange network</strong> — a decentralized way to discover, authenticate and exchange memory assets.</li>
</ol>

<p>Notably, the paper doesn’t insist everything stays on your device. If a provider hosts your memory, it should be encrypted so only you can read it.</p>

<blockquote class="diagram-text"><p><strong>Diagram: Memory as Asset: memory objects and the three layers</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-memory-as-asset/">see it in the post</a>)</p><p>Each memory object carries its content, its context and a version, plus an explicit owner and an access policy, so ownership travels with the data. Memory objects are kept in layer 1, personal storage: each person's private memory objects. VaultMem implements this layer, encrypted with a key only the user holds. Layer 2, evolution, updates memory from interactions: it records experiences, merges related memories and abstracts knowledge, and can run locally or remotely. Layer 3, the memory exchange network, discovers, authenticates and exchanges memory assets; the paper sketches it rather than solving it. Overall: owned by you, shared only through your permissions.</p><p><em>The paper's design in one picture: owner and access policy live inside every memory object, and three layers store, evolve and exchange those objects. VaultMem is the first layer.</em></p></blockquote>

<h2 id="how-it-shaped-what-i-built">How it shaped what I built</h2>

<p>That last point is where VaultMem lives. I treat it as an implementation of the <strong>first layer</strong>: the platform stores your memories, but they’re encrypted with a key only you hold, so the operator cryptographically can’t read them. Everything in <a href="/series/#vaultmem">the VaultMem series</a> — client-side encryption, encrypted search, <a href="/post/vaultmem-part-3-pii-sanitizer/">keeping PII away from the model provider</a> — follows from taking “memory in hand” literally.</p>

<h2 id="caveats-worth-knowing">Caveats worth knowing</h2>

<ul>
  <li>The authors are upfront that this is an <strong>early draft</strong>: the terms and design are still evolving.</li>
  <li>It’s a <strong>conceptual and architectural</strong> paper — there are no benchmarks or experiments yet. The claim that existing systems meet at most two of the three principles is an argument, not a measurement.</li>
  <li>The exchange network raises hard questions (trust, incentives, revocation) that the paper sketches rather than solves.</li>
</ul>

<p>I still think it’s the right question to ask before building any memory feature: <em>whose memory is this?</em></p>

<p>Earlier in the series: <a href="/post/paper-tldr-six-papers-on-memory/">six papers on AI memory</a>, <a href="/post/paper-tldr-hopfield-networks-is-all-you-need/">Hopfield Networks Is All You Need</a>, <a href="/post/paper-tldr-complementary-learning-systems/">Complementary Learning Systems</a> and <a href="/post/paper-tldr-locomo-long-term-memory-benchmark/">LoCoMo</a>. Next in the series: <a href="/post/paper-tldr-sentence-bert/">Sentence-BERT</a>, the sentence embeddings behind most semantic search.</p>

<p><a href="/post/agent-memory-glossary/#zero-knowledge-storage"><em>Zero-knowledge storage</em></a> and <a href="/post/agent-memory-glossary/#client-side-encryption"><em>client-side encryption</em></a> are defined in the privacy section of the <a href="/post/agent-memory-glossary/#privacy">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><category term="Privacy" /><summary type="html"><![CDATA[A TL;DR of Memory as Asset (Pan, Huang & Yang, 2026): moving AI memory from agent-centric to human-centric, its three principles, and how it shaped VaultMem.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-memory-as-asset.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-memory-as-asset.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Paper TL;DR: LoCoMo — How Good Is Your Agent’s Memory, Really?</title><link href="https://www.avinashgosavi.com/post/paper-tldr-locomo-long-term-memory-benchmark/" rel="alternate" type="text/html" title="Paper TL;DR: LoCoMo — How Good Is Your Agent’s Memory, Really?" /><published>2026-10-18T19:00:00+00:00</published><updated>2026-10-23T01:35:48+00:00</updated><id>https://www.avinashgosavi.com/post/paper-tldr-locomo-long-term-memory-benchmark</id><content type="html" xml:base="https://www.avinashgosavi.com/post/paper-tldr-locomo-long-term-memory-benchmark/"><![CDATA[<p><strong>Paper:</strong> Maharana et al., <a href="https://arxiv.org/abs/2402.17753"><em>Evaluating Very Long-Term Conversational Memory of LLM Agents</em></a> (ACL 2024)</p>

<p><strong>One-line TL;DR:</strong> LoCoMo is a benchmark of very long, multi-session conversations with questions that probe whether a system really remembers — and even strong long-context models and retrieval setups fall well short of people on it.</p>

<p>If you’re building memory for an agent, this is the paper that keeps you honest.</p>

<h2 id="whats-in-the-benchmark">What’s in the benchmark</h2>

<p>Most chat benchmarks are a few turns long. LoCoMo conversations run for <strong>dozens of sessions and hundreds of turns</strong>, spread over simulated weeks and months. The authors generated them with LLM agents that each had a persona and a timeline of life events, then had people review and fix them so the conversations stay consistent. Some turns include shared images, too.</p>

<p>On top of the conversations they built three tasks: answering questions, summarizing events, and generating responses. The question-answering part is the one I care about, and its categories are what make it useful:</p>

<ul>
  <li><strong>Single-hop</strong> — one fact from one place in the history.</li>
  <li><strong>Multi-hop</strong> — needs to connect facts from different sessions.</li>
  <li><strong>Temporal</strong> — when did something happen, or what happened before or after.</li>
  <li><strong>Open-domain</strong> — combine the conversation with general knowledge.</li>
  <li><strong>Adversarial</strong> — questions about things that <em>never</em> happened, where the right answer is “that wasn’t mentioned.”</li>
</ul>

<blockquote class="diagram-text"><p><strong>Diagram: How the LoCoMo benchmark is built</strong> (<a href="https://www.avinashgosavi.com/post/paper-tldr-locomo-long-term-memory-benchmark/">see it in the post</a>)</p><p>Each LoCoMo conversation averages about 300 turns and 9 thousand tokens, over up to 35 sessions spread across simulated weeks and months. Three tasks are built on top: question answering, event summarization, and multi-modal dialogue generation. Question answering splits into five categories: single-hop, one fact from one place; multi-hop, connecting facts across sessions; temporal, when something happened or what came before or after; open-domain, combining the conversation with general knowledge; and adversarial, questions about things that never happened, where the right answer is that it was not mentioned.</p><p><em>Scoring each category on its own keeps a strong overall number from hiding weak temporal or multi-hop answers.</em></p></blockquote>

<h2 id="what-they-found">What they found</h2>

<p>Even with long context windows or <a href="/post/paper-tldr-retrieval-augmented-generation/">retrieval-augmented generation</a>, models trail human performance by a wide margin. Retrieval helps, but the hard categories stay hard — especially reasoning about time and resisting the urge to make up answers to adversarial questions. Bigger context alone doesn’t fix it.</p>

<h2 id="how-i-used-it">How I used it</h2>

<p>When I built <a href="/series/#vaultmem">VaultMem</a>, my first benchmark was 18 synthetic memories I wrote myself. It looked great — which was the problem. LoCoMo gave me a test I didn’t design. <a href="/post/vaultmem-part-2-multimodal-temporal-backends/">On a sample of its questions</a>, retrieval was strong for single-hop facts and weakest for multi-hop, which needs pieces that weren’t individually asked about. That’s a retrieval problem before it’s a model problem, and it changed what I worked on next.</p>

<h2 id="the-takeaway-for-builders">The takeaway for builders</h2>

<ol>
  <li><strong>Test against data you didn’t write.</strong> A memory system always looks good on its author’s examples.</li>
  <li><strong>Score by category, not just overall.</strong> “85% accurate” hides that temporal and multi-hop questions might be failing.</li>
  <li><strong>Measure the “I don’t know.”</strong> A memory that confidently invents a past is worse than one that admits a gap.</li>
</ol>

<p>Earlier in the series: <a href="/post/paper-tldr-six-papers-on-memory/">six papers on AI memory</a>, <a href="/post/paper-tldr-hopfield-networks-is-all-you-need/">Hopfield Networks Is All You Need</a>, and <a href="/post/paper-tldr-complementary-learning-systems/">Complementary Learning Systems</a>. Next in the series: <a href="/post/paper-tldr-memory-as-asset/">Memory as Asset</a>, on who should own an agent’s memory.</p>

<p><a href="/post/agent-memory-glossary/#recall-at-k"><em>Recall@k</em></a> and <a href="/post/agent-memory-glossary/#mrr"><em>MRR</em></a>, the usual ways to score retrieval, are explained in the <a href="/post/agent-memory-glossary/#measuring-it">agent memory glossary</a>.</p>]]></content><author><name>Avinash Gosavi</name></author><category term="Research" /><category term="Agent memory" /><summary type="html"><![CDATA[A TL;DR of the LoCoMo benchmark (Maharana et al., 2024) for very long-term conversational memory — what it tests, what it found, and how I used it on VaultMem.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.avinashgosavi.com/assets/og/paper-tldr-locomo-long-term-memory-benchmark.png" /><media:content medium="image" url="https://www.avinashgosavi.com/assets/og/paper-tldr-locomo-long-term-memory-benchmark.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>