Agent memory · Avinash Gosavi

Paper TL;DR: HNSW — How Vector Search Skips Most of the Haystack

2026-10-20T20:45:00+00:00

Paper: Yu. A. Malkov & D. A. Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (arXiv, first posted March 2016; later published in IEEE Transactions on Pattern Analysis and Machine Intelligence, DOI 10.1109/TPAMI.2018.2889473)

One-line TL;DR: Stack a few neighbor graphs on top of each other — sparse ones with long jumps at the top, a dense one with every item at the bottom — and you can find the closest vectors to a query by hopping down the stack instead of checking everything.

The question

Every memory or RAG system eventually asks the same thing: which stored embeddings are closest to this query vector? Checking every vector works, but the cost grows linearly with the collection. Exact shortcuts like k-d trees stop helping once vectors have hundreds of dimensions. So practical systems use approximate nearest neighbor (ANN) search: accept that you’ll occasionally miss a true neighbor, and measure how often you don’t with recall.

One family of ANN methods builds a proximity graph — each item linked to its near neighbors — and searches by greedily walking toward the query. The authors’ own earlier method, Navigable Small World (NSW), did this but had two problems: search cost grew faster than logarithmically as the data grew, and on low-dimensional or tightly clustered data the greedy walk could get stuck in a “false” local minimum. HNSW is their fix.

How it works

The paper describes the idea as a skip list where the linked lists are replaced by proximity graphs.

Layers. Every item lives in the bottom layer. When it’s inserted, it also draws a random top layer from an exponentially decaying distribution, so only a few items reach the upper layers. Upper layers are sparse and their links span long distances; the bottom layer is dense with short links.
Search. Start at a fixed entry point on the top layer. On each layer, greedily move to whichever neighbor is closer to the query until none is, then drop down a layer and continue from there. On the bottom layer, keep a wider list of candidates (size ef) and return the best k. A bigger ef means higher recall and more work.
Insertion. A new item is placed by running the same descent, then — on each layer from its own top layer down — linking to up to M neighbors found by a wider search (efConstruction). If a neighbor ends up with too many links, its list is pruned back to a cap.
A smarter way to pick neighbors. Rather than always linking to the closest candidates, the heuristic keeps a candidate only if it’s closer to the new item than to any neighbor already chosen. That spreads links out in different directions, which keeps separate clusters connected — exactly where plain NSW got stuck.

Diagram: How an HNSW search hops down the layers (see it in the post)
A sketch of three stacked neighbor graphs. The top layer has only two items joined by one long link; the middle layer has four; the bottom layer has every item, each linked to close neighbors. A search starts at the entry point on the top layer, hops to the neighbor closer to the query, and when no neighbor is closer it drops to the same item one layer down and continues. On the bottom layer it keeps a list of ef candidates around the query and returns the best k.
A sketch, not real data: upper layers let the search take long jumps, the bottom layer lets it finish the job up close.

The knobs the paper discusses are few: M (links per item; it suggests roughly 5–48, higher for high-recall or high-dimensional data), efConstruction (build quality) and ef (search effort). The graph links cost roughly 60–450 bytes per item on top of the vectors themselves.

What they found

Roughly logarithmic search. The authors argue search cost scales as O(log N) under idealized assumptions, and see logarithmic scaling empirically on low-dimensional data. Construction is roughly O(N log N) there.
Fastest in their vector benchmarks. Single-threaded on the ann-benchmarks suite, HNSW clearly beat NSW, FLANN, Annoy, VP-tree and FALCONN (an LSH library) on SIFT, GloVe, DEEP and CoPhIR, and stayed ahead on random 4-dimensional data.
Not just for vectors. Because it only needs a distance function, it also led on non-vector data — sparse text, topic histograms, image signatures, even DNA strings compared by edit distance. The biggest gain over NSW was roughly three orders of magnitude on one low-dimensional, clustered dataset.
Speed versus memory against compression. On 200 million SIFT vectors, HNSW used more RAM than Faiss’s product-quantization index (about 64 GB versus 23.5–30 GB) but was more accurate, faster to search and much faster to build.

Why it matters for builders

It’s probably already in your stack. The paper notes that Faiss added its own HNSW implementation, and RAG’s Wikipedia index used a FAISS HNSW index (see the RAG TL;DR). VaultMem has an optional HNSWVectorIndex that wraps the hnswlib library.
Small collections don’t need it. The graph has real overhead, so brute force is often fine early on — VaultMem Part 2 falls back to exact cosine search below 1,000 memories and shows where the crossover happens.
Recall is a dial, not a constant. ef trades speed for recall at query time, without rebuilding. When a memory system “forgets” something it stored, the index settings are one more place to look — alongside the embeddings and the re-ranking step.
The index decides which candidates you even see. Better embeddings (Sentence-BERT) and better prompt placement (Lost in the Middle) only help with the neighbors the index returns.

Honest caveats

The O(log N) story has gaps. The guarantee leans on idealized graphs the heuristic doesn’t actually build, and the authors say high-dimensional behavior isn’t proven. On the 200M-vector test, query time drifted somewhat from pure logarithmic growth.
Memory hungry. All those links live alongside the vectors, so it needs more RAM than compressed indexes like product quantization.
No deletes or updates — at the time. The paper lists element updates and removal as future work. That matters for memory systems that edit or unlearn facts; check what your library supports today.
Hard to distribute. Every search starts at the top layer, which can become a hotspot, and the authors found that simply splitting data across machines scales poorly in total throughput.
Benchmarks are a snapshot. Results depend on the hardware and the library versions of the time, and some baselines were older builds.

Earlier in the series: RAG (the retrieve-then-generate loop this index sits inside) and Sentence-BERT (the embeddings it searches over).

Terms like HNSW, embedding, semantic search, top-k retrieval and Recall@k are defined in the agent memory glossary.

Paper TL;DR: RAG — The Paper That Named Retrieval-Augmented Generation

2026-10-20T15:45:00+00:00

Paper: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel & Douwe Kiela (Facebook AI Research, University College London, New York University), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, 2020; revised 2021)

One-line TL;DR: Instead of hoping a language model memorized the right fact, give it a search step: look up a few relevant Wikipedia passages, generate the answer from them, and train the search and the writer together.

The question

By 2020, large language models clearly stored a lot of facts in their weights. But the paper points out the downsides: that knowledge is hard to expand or revise, the model can’t show where an answer came from, and it can hallucinate. Earlier hybrid models that paired a language model with a retriever (REALM and ORQA) had only been tried on extractive question answering — pulling answer spans out of passages rather than writing new text. This paper asks whether one general recipe can combine the two: a model that retrieves documents and generates free-form answers, fine-tuned end to end for many different tasks.

How it works

The paper splits knowledge into two kinds of memory:

Parametric memory — what’s baked into the model’s weights. Here that’s BART-large, a pre-trained sequence-to-sequence generator.
Non-parametric memory — an external index you can search. Here that’s a December 2018 Wikipedia dump cut into 100-word chunks (about 21 million of them), each turned into an embedding by a BERT document encoder from DPR (Dense Passage Retrieval). The vectors live in a FAISS index using an HNSW approximation, so lookups stay fast.

At question time, a second BERT encoder turns the input into a query vector, the index returns the top-k passages, and BART generates an answer conditioned on the input plus each passage. Nobody labels which passage is “right”: the retriever learns only from whether the final answer comes out well.

Two variants decide how the passages get combined:

RAG-Sequence uses the same passage for the whole answer, then weighs the answers it would give for each passage.
RAG-Token can lean on a different passage for each word, so one answer can stitch together facts from several documents.

One practical shortcut: only the query encoder and the generator are trained. The document encoder — and so the whole 21-million-vector index — stays frozen, because re-embedding Wikipedia during training is expensive. The authors report they didn’t need that step for strong results.

Diagram: How RAG retrieves, then generates (see it in the post)
The input goes to a BERT query encoder, which turns it into a query vector. The vector searches a FAISS index of about 21 million 100-word Wikipedia chunks from December 2018, whose document encoder stays frozen. The top-k passages go to a BART generator, the model's parametric memory, which writes the answer. The only training signal is whether the final answer is right, and it updates just the query encoder and the generator. The index can be swapped for one built from a 2016 Wikipedia dump, which changes what the model knows without retraining.
Only the query encoder and the generator are trained. The index stays frozen, so swapping it for an older Wikipedia dump changes what the model knows.

What they found

Strong open-domain question answering. On Natural Questions, WebQuestions and CuratedTrec, RAG beat the previous best systems in the paper’s comparison, while writing answers instead of copying spans. On TriviaQA the picture is mixed: it led on one test split but not the standard one.
It can answer even when the passage doesn’t spell it out. On Natural Questions, RAG still got about 11.8% of answers right in cases where the exact answer wasn’t in any retrieved passage — something a pure extractor can’t do.
More factual, more specific text. In a human evaluation of generated Jeopardy-style questions, raters judged RAG more factual than plain BART far more often than the other way round (42.7% vs. 7.1% of cases), and more specific too.
Small next to giant models. RAG has 626M trainable parameters; the largest closed-book model it’s compared to, T5-11B, has 11 billion — yet RAG scored higher on open-domain QA.
Swap the index, update the knowledge. The authors built a second index from a 2016 Wikipedia dump and asked “Who is {position}?” about 82 world leaders who changed between 2016 and 2018. With the matching index RAG got roughly 70% right; with the mismatched one, 4–12%. Changing what the model knows meant replacing the index, not retraining.

Why it matters for builders

It’s the template most “chat with your docs” systems still follow. Embed chunks, search by meaning, paste the top results into the prompt, generate. Agent memory systems do the same thing with memories instead of Wikipedia pages — the glossary’s “Storing and finding memories” section walks through each piece.
Updating knowledge is a data problem, not a training problem. The hot-swap experiment is the cleanest argument for keeping facts outside the weights: if something changes, you change the index. The same idea is why memory systems store facts with timestamps and versions rather than hoping the model “knows” the latest.
Provenance comes for free. Because the answer is conditioned on specific passages, you can show where it came from — useful for debugging and for trust.
Retrieval quality still caps everything. The retriever here is learned only through the final answer, and the paper reports it can fail badly (see caveats). Later work in this series picks up from there: Lost in the Middle shows that even good passages help less when they land mid-prompt, and MemGPT treats the context window as something to manage rather than fill.

Honest caveats

This is 2020-era plumbing. “RAG” today usually means an instruction-tuned LLM plus an off-the-shelf retriever, with no joint training. The paper’s specific recipe — BART, DPR, a frozen Wikipedia index — is mostly of historical interest now; the idea is what stuck.
Retrieval can collapse. In early story-generation experiments, the retriever learned to return the same documents no matter the input, after which the generator ignored them and RAG behaved just like plain BART. The authors suggest tasks with long outputs or less need for facts may give the retriever too weak a signal.
Not state of the art everywhere. On FEVER fact verification, RAG came within about 4.3% of the best pipeline (3-way) without being given gold evidence — close, but not ahead.
The index is only as good as its source. The authors note that Wikipedia will likely never be fully factual or free of bias, and that the same misuse concerns as other text generators apply.
Big infrastructure for its time. The paper notes the Wikipedia index initially needed about 100 GB of CPU memory, cut to 36 GB with FAISS compression.

Earlier in the series: Sentence-BERT (the kind of embeddings that make the search step work), Lost in the Middle (where retrieved passages land in the prompt) and MemGPT (an agent that pages its own memory in and out). Next in the series: HNSW, the graph index behind fast nearest-neighbor lookups like this one.

Terms like retrieval-augmented generation, embedding, semantic search, top-k retrieval and HNSW are defined in the agent memory glossary.

Paper TL;DR: Lost in the Middle — Where You Put Context Matters

2026-10-20T10:45:00+00:00

Paper: Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni & Percy Liang (Stanford, UC Berkeley, Samaya AI), Lost in the Middle: How Language Models Use Long Contexts (arXiv, 2023)

One-line TL;DR: Give a model a long prompt with one useful passage in it, and how well it answers depends on where that passage sits: best at the very start or very end, noticeably worse in the middle — even for models built for long contexts.

The question

Retrieval-augmented generation and agent memory both work the same way at the last step: fetch some passages, paste them into the prompt, and hope the model reads them. Bigger context windows make it tempting to paste in more. This paper asks a simple question first: does the model actually use everything in a long prompt equally well?

How they tested it

Two tasks, each designed so the authors could move the one relevant piece of information around while keeping everything else the same.

Multi-document question answering. Questions from NaturalQuestions-Open, each paired with one Wikipedia passage that contains the answer plus distractor passages that look relevant but don’t. Prompts held 10, 20 or 30 passages, and the answer passage was moved from first to last.
Key-value retrieval. A synthetic stress test: a JSON object of random UUID keys and values (75, 140 or 300 pairs), and the model has to return the value for one key. No language cues, just “find this and copy it.”

They ran open models (MPT-30B-Instruct, LongChat-13B) and API models (GPT-3.5-Turbo, Claude-1.3, and their longer-context versions, plus GPT-4 on a subset).

What they found

A U-shaped curve. Accuracy was highest when the answer passage came at the very start or the very end, and dropped when it sat in the middle — by more than 20% in some settings. The authors describe this as a primacy bias plus a recency bias.
The middle can be worse than nothing. In one 20-document setting, GPT-3.5-Turbo with the answer buried mid-prompt scored lower than the same model answering with no documents at all.
A longer window didn’t fix it. When a prompt fit in both a model and its extended-context version, the two scored almost the same. A bigger window lets you paste more; it doesn’t make the model read it better.
More retrieved documents stop helping early. In an open-domain QA case study, the retriever kept finding the answer more often as it returned more passages, but the model’s accuracy flattened out well before that. Going from 20 to 50 passages bought only a point or two, at much higher cost and latency.
Asking the question twice helps for lookups. Putting the query both before and after the data made key-value retrieval near-perfect, but it barely changed the multi-document QA results.

Diagram: The U-shaped curve from Lost in the Middle (see it in the post)
A qualitative sketch, not the paper's data. The horizontal axis is where the one useful passage sits in a prompt of many passages, from first to last. The vertical axis is answer accuracy. The curve is high when the passage is first (primacy bias), dips lowest when it is in the middle, and rises again when it is last (recency bias).
A sketch of the shape the paper reports, not its numbers: the same passage helps most at the edges of the prompt and least in the middle.

Why it matters for builders

Order is part of retrieval. Ranking isn’t just about which passages make the cut but where they land. Putting the best matches first (and not burying them under weaker ones) is cheap and directly addresses this. A re-ranking pass earns its keep here.
Fewer, better passages beat more passages. The case study suggests a tighter top-k can be nearly as accurate and a lot cheaper. It also connects to the idea of sufficient evidence: the goal is enough context, not all of it.
Test position, not just recall. A memory system can retrieve the right fact and still fail if the fact lands mid-prompt. Benchmarks like LoCoMo measure the end answer; this paper’s trick of moving the gold passage around is an easy extra check to run on your own pipeline.
Keep the important stuff at the edges. Designs like MemGPT keep a small, always-visible working context near the top of the prompt. This paper is one reason that kind of layout makes sense.

Honest caveats

These are 2023 models. The results cover GPT-3.5, Claude-1.3, MPT and LongChat, with only partial GPT-4 runs because of cost. Newer models may show a weaker (or different) position effect, so it’s worth measuring on whatever you actually use.
The “why” is still open. The authors test a few explanations — architecture, instruction tuning, model size — and call those analyses preliminary. Encoder-decoder models were fairly robust within their training length but showed the U-shape on longer inputs; base models without instruction tuning showed it too.
One answer per prompt. Each test has a single relevant passage. Real questions often need several facts combined, which this setup doesn’t measure.
Greedy decoding only, and the key-value task is deliberately artificial, so it shows a capability gap rather than typical real-world behavior.

Earlier in the series: MemGPT (paging what goes into the context window), Generative Agents (ranking memories by recency, importance and relevance) and Sentence-BERT (the embeddings behind most of the retrieval step). Next in the series: RAG, the 2020 paper that named the retrieve-then-generate setup this paper stress-tests.

Terms like context window, position bias, re-ranking and top-k retrieval are defined in the agent memory glossary.

Paper TL;DR: Generative Agents — Memory Streams, Reflection and Plans

2026-10-19T21:40:00+00:00

Paper: Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang & Michael S. Bernstein (Stanford, Google Research, Google DeepMind), Generative Agents: Interactive Simulacra of Human Behavior (UIST 2023)

One-line TL;DR: Give an LLM a running log of everything it observes, a way to score which memories matter right now, and a habit of periodically writing down higher-level conclusions — and simulated characters start to behave believably over days, not just single replies.

The setup

The authors built Smallville, a small sandbox town populated by 25 characters. Each one is driven by an LLM (gpt-3.5-turbo in the paper) and starts from a single paragraph describing who they are: job, relationships, habits. From there the characters wake up, go to work, chat, form opinions of each other and make plans — all generated, step by step, by the model.

The interesting part isn’t the town. It’s the memory design that keeps each character consistent over time, which the paper splits into three pieces: a memory stream, reflection and planning.

1. The memory stream

Every character keeps a memory stream: a plain list of memories, each written in natural language with a creation time and a “last accessed” time. The basic memories are observations — things the character perceived, such as (to give an illustrative example) “a neighbor is watering the garden.”

The stream grows far too large to fit into a prompt, so each time a character needs to act, it retrieves only the most useful memories. Each memory gets three scores:

Recency: recently used memories score higher, fading exponentially with each in-game hour (the paper uses a decay factor of 0.995).
Importance: when a memory is created, the model rates it from 1 (mundane, like cleaning a room) to 10 (poignant, like asking someone out).
Relevance: cosine similarity between the memory’s embedding and the current situation’s embedding.

Each score is scaled to 0–1 and the three are added up (the paper weights them equally). The top-scoring memories that fit into the context window go into the prompt.

2. Reflection

Raw observations are too low-level to generalize from. Fifty separate observations of a neighbor reading in the library don’t, by themselves, tell a character “she cares a lot about her research.”

So characters reflect. When the summed importance of recent memories passes a threshold (150 in the paper — roughly two or three times per simulated day), the model looks at the 100 most recent memories, asks itself three high-level questions about them, retrieves memories relevant to each question, and writes down insights that cite the memories they came from.

Those insights go back into the same memory stream, where they can be retrieved — and reflected on again. The result is a reflection tree: raw observations at the leaves, more abstract conclusions higher up.

Diagram: How a generative agent remembers, retrieves and reflects (see it in the post)
Each character keeps a memory stream of natural-language entries with a creation time and a last-accessed time: observations, plans and reflections. When the character needs to act, every memory is scored on recency (decaying each game hour), importance (rated 1 to 10 by the model) and relevance (cosine similarity of embeddings); the scores are scaled to 0 to 1 and added. The top memories go into the prompt to choose the next action. When the importance of recent memories adds up past a threshold, the character reflects, and the insights it writes are stored back in the memory stream.
Everything lives in one stream. Retrieval ranks it three ways; reflection writes new, higher-level memories back into it (dashed loop).

3. Planning

To keep behavior coherent across a day, each character drafts a rough plan (about five to eight chunks), then breaks each chunk into hour-long actions, then into 5–15 minute steps. Plans are stored in the memory stream too. At every step, the model decides whether something it just noticed is worth reacting to; if so, it rewrites the plan from that point.

What they tested

Interviews. The authors “interviewed” characters with questions about themselves, their memories, plans, reactions and reflections. 100 human evaluators ranked answers from the full architecture, versions with pieces removed, and human crowdworkers role-playing the same characters. The full architecture ranked most believable; removing reflection, then planning, then memory each made answers noticeably worse. The crowdworker answers landed below every version that still had memory.

Emergent behavior over two game days. One character was seeded with the idea of throwing a Valentine’s Day party. Without further scripting, word spread from 1 character to 13 of 25, and 5 of the 12 who were invited showed up. A mayoral candidacy spread from 1 character to 8 the same way.

Why it matters for builders

Rank memories on more than similarity. Recency + importance + relevance is a simple, readable scoring rule, and variations of it show up across agent memory systems. VaultMem, for example, re-ranks on cosine similarity plus a significance score that decays over time.
Store conclusions, not just events. Reflections are cheap summaries the agent can retrieve later, and keeping links back to the source memories makes them auditable.
Memory and planning are the same store. Writing plans into the memory stream means “what am I doing next?” is answered by the same retrieval as “what happened?”

Honest caveats

Retrieval still misses. The authors show characters failing to recall something they had heard, or recalling a detail of an event but not the event itself.
Embellishment. Characters sometimes added details nobody had said, or blended in the model’s world knowledge — one described a neighbor as the author of a famous 18th-century economics book.
Too polite. The authors attribute overly formal, overly agreeable dialogue to instruction tuning; one character rarely said no to suggestions, and her own interests drifted as a result.
Expensive. Running 25 characters for two game days took multiple days and, in the authors’ words, “thousands of dollars in token credits.”
Believability isn’t accuracy. The evaluation measured whether behavior seemed human, not whether memories were retrieved correctly. Benchmarks like LoCoMo test that more directly.

Earlier in the series: MemGPT (an OS-style alternative where the model pages its own memory), Complementary Learning Systems (the brain’s version of turning episodes into general knowledge — close in spirit to reflection), and Sentence-BERT (the kind of embeddings behind the relevance score). Next in the series: Lost in the Middle, on what happens to retrieved text once it lands in a long prompt.

Terms like reflection, consolidation and decay are defined in the agent memory glossary.

Paper TL;DR: MemGPT — Treating the Context Window Like RAM

2026-10-19T10:40:00+00:00

Paper: Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica & Joseph E. Gonzalez (UC Berkeley), MemGPT: Towards LLMs as Operating Systems (arXiv, 2023)

One-line TL;DR: Instead of waiting for bigger context windows, MemGPT treats the prompt like a computer’s RAM and lets the model itself move information between that small fast space and larger external storage, using function calls.

The problem

An LLM can only “see” what fits in its context window. Long chats and big document collections don’t fit, so something has to decide what stays in view. Operating systems solved a version of this decades ago: programs act as if they have lots of memory, while the OS quietly pages data between fast RAM and slower disk. The paper calls its version virtual context management.

How it works

MemGPT splits memory into two places.

Main context — what’s actually in the prompt — has three parts:

System instructions: read-only rules, including how to use the memory functions.
Working context: a small, fixed-size scratchpad of key facts (for example, things learned about the user). The model can only change it through function calls.
A FIFO message queue: the recent conversation. Its first entry is a running summary of older messages that were pushed out.

External context lives outside the prompt:

Recall storage: the full message history, searchable on demand.
Archival storage: a store for arbitrary text the model chooses to save or look up later.

Diagram: MemGPT's two tiers of memory (see it in the post)
Main context is what is in the prompt: read-only system instructions, a small working context of key facts edited through function calls, and a FIFO message queue that starts with a summary of older messages. External context sits outside the prompt: recall storage holds the full message history and archival storage holds any text the model chooses to save. The oldest messages are evicted from the queue into recall storage, and the model searches and saves across both tiers with function calls.
Main context is the prompt; external context is everything the model can page back in. The model itself moves information between them.

The clever part is who moves things around: the model does. Its output is parsed as function calls (search recall, write to archival, edit working context), and the results are fed back in. A few mechanics make this work:

Memory pressure warnings. When the prompt gets close to full (the paper’s example uses 70%), the system adds a warning so the model can save what matters before it’s lost.
Eviction with summaries. When the prompt is full, a chunk of the oldest messages is evicted and folded into the running summary. Evicted messages aren’t deleted — they stay in recall storage.
Heartbeats. A function call can ask to keep going (request_heartbeat), so the model can chain several lookups before replying.
Events. User messages, system alerts and scheduled timers all wake the agent up, so it doesn’t only act when a person types.

What they tested

Two settings, each with a “can it really remember?” task:

Multi-session chat. In a deep memory retrieval task, the agent is asked about something from an earlier session. MemGPT answered far more of these correctly than the same models with a fixed context. In a “conversation opener” task, its openers drew on what it knew about the person, and the authors found that saving facts to working context was key.
Document analysis. For question answering over many documents, MemGPT could page through more search results than fit in one prompt. In a nested key-value lookup task, where one value points to another key, fixed-context models broke down as nesting got deeper, while MemGPT with GPT-4 kept answering by looking things up repeatedly.

Why it matters for builders

Memory can be a set of tools, not just a bigger prompt. Giving the model explicit read/write/search functions makes memory something you can inspect and log.
Separate “always in view” from “look it up.” A small working context of key facts plus searchable history is a pattern that shows up again and again in agent memory designs.
Plan for overflow. Warning before the window fills, and summarizing instead of silently truncating, is a simple idea worth copying.

Honest caveats

It leans on the model’s function-calling skill. The authors report that results dropped noticeably with GPT-3.5, which was weaker at calling functions.
Retrieval is still the bottleneck. With embedding search, the right document often wasn’t near the top, and the agent tended to stop paging before exhausting the results.
More model isn’t always better. On the nested lookup task, the GPT-4 Turbo version did worse than GPT-4, and some versions gave up after too few lookups.
Self-managed memory is a judgment call by the model. What gets saved, summarized or forgotten depends on the model’s choices, which is harder to predict than a fixed rule.

Earlier in the series: LoCoMo (a benchmark for exactly this kind of long-term memory), Sentence-BERT (the embeddings behind most semantic search), and Complementary Learning Systems (a brain-inspired split between fast and slow memory). Next in the series: Generative Agents, which ranks a long memory stream by recency, importance and relevance.

Terms like virtual context, context window, episodic memory and top-k retrieval are defined in the agent memory glossary.

An Agent Memory Glossary: Terms in Plain Language

2026-10-19T02:45:00+00:00

The vocabulary around AI agent memory borrows from neuroscience, databases, search and cryptography all at once, which makes it easy to talk past each other. This is a working glossary — short, plain definitions, with links to where each idea shows up in my own projects. I’ll keep adding to it.

Kinds of memory

Context window (working memory). What the model can see right now: the prompt, the recent conversation and anything retrieved for this turn. Fast and precise, but limited in size and gone when the session ends. Everything else on this list exists because of that limit.

Episodic memory. Specific events: what happened, when, with whom. “On Tuesday you said the demo slipped to Friday.” In VaultMem these are one of the memory types the agent can store and retrieve.

Semantic memory. General facts and knowledge, detached from when they were learned. “Flash Attention reduces memory traffic by tiling.” Often distilled from many episodes.

Procedural memory. How to do things: workflows, preferences about process, steps that worked before. “When debugging, check the logs before the config.”

Persona (profile) memory. Stable facts about the person the agent works with — name, role, standing preferences. Small, high-value and read on almost every turn.

Virtual context (paging). Treating the context window like a computer’s RAM: the agent moves information in and out of the prompt — saving facts to external storage, searching old messages, summarizing what gets pushed out — usually through function calls. Popularized by MemGPT.

Storing and finding memories

Embedding. A list of numbers that represents the meaning of a piece of text, so that similar meanings land close together. Most memory systems store an embedding next to each memory. See the Sentence-BERT TL;DR for where modern sentence embeddings come from.

Semantic search. Finding memories by meaning instead of exact words: embed the query, then return the stored memories whose embeddings are closest.

Top-k retrieval. Returning the k closest matches (say, the top 10) and putting them into the context window. Simple and effective; also where most retrieval mistakes start.

Retrieval-augmented generation (RAG). Search first, then write: fetch the passages most relevant to the input and have the model generate its answer from them, instead of relying only on what’s stored in its weights. The term comes from a 2020 paper that also showed you can update what the system knows by swapping the search index — see the RAG TL;DR.

Vector index (HNSW). A data structure that makes “find the closest vectors” fast at scale. HNSW (Hierarchical Navigable Small World) builds layered graphs of neighbors so a search can hop quickly toward the right region instead of comparing against every vector. (Paper TL;DR)

Re-ranking. A second, slower and more careful pass over the top results — often a cross-encoder that reads the query and each candidate together — to put the truly relevant ones first.

Position bias (lost in the middle). Models tend to use information at the start and end of a long prompt better than information buried in the middle, so where retrieved memories land in the prompt affects whether they get used. Putting the strongest matches first, and keeping top-k small, helps. See the Lost in the Middle TL;DR.

Hybrid search. Combining meaning-based search with keyword search (or filters like dates and tags), because embeddings are bad at exact names, numbers and IDs.

Context frame. The situation a memory was formed in — topic, people involved, task, session — stored alongside it so retrieval can filter by context, not just content. This is the core idea in Attractor.

Sufficient evidence. The idea that a memory system’s job isn’t just to return relevant memories but a set that is enough to answer. Ranking memories one at a time can miss this: several records may say the same thing (so the set gets bloated), or two weak-looking records may only make sense together (so both get dropped). One recent paper defines a sufficient set as a small subset that leads the agent to the same decision it would make with its full memory (Beyond Marginal Influence).

How memories change over time

Consolidation. Turning many specific episodes into durable, general knowledge — summarizing a month of conversations into “prefers short answers,” for example. The brain’s version is covered in the Complementary Learning Systems TL;DR.

Reflection. Having the agent periodically read its recent memories and write down higher-level conclusions (“she cares a lot about her research”), stored back as memories that point to the evidence they came from. Introduced for LLM agents in Generative Agents, which also ranks memories by recency, importance and relevance together.

Decay (forgetting). Letting memories that are never used fade in importance, so old noise doesn’t crowd out what matters. Forgetting is a feature.

Event time vs. write time. Two different timestamps for one memory: when the thing happened, and when it was saved. A photo from 2019 imported today should come back for “what did I do in 2019?”, not for “what did I add today?”. VaultMem Part 2 keeps them as separate fields (captured_at and created_at) for exactly this reason.

Versioned facts (temporal state). Keeping every value a fact has had — “lives in Boston”, then “lives in Seattle” — as a list of versions linked to where each came from, instead of overwriting the old one. The newest version answers “where does she live?”, while the older ones still answer “where did she live before?”. The TEM paper adds one more step: older versions are marked as historical before the model sees them, and the authors report that this label alone gave large gains on questions about repeatedly updated facts.

Associative (content-addressable) memory. Retrieving a whole memory from a partial or noisy piece of it, rather than by key or address. Hopfield networks are the classic example — see Hopfield (1982) and the modern version.

Privacy

Zero-knowledge storage. The service storing your data cannot read it, because it’s encrypted with a key only you hold. The operator sees ciphertext. This is VaultMem’s core design (why it matters, in plain language).

Client-side encryption. Encrypting data before it leaves your device (or your session), so the server only ever handles encrypted bytes. The usual way to get zero-knowledge storage.

PII sanitization. Detecting and masking personal details — names, emails, phone numbers — before text is sent to a model provider, then restoring them in the response. It closes a gap encryption at rest can’t: the model still has to read the text. See VaultMem Part 3.

Memory unlearning (forget requests). Deleting a specific fact when the user asks — and only that fact. Done well, the fact can no longer be recalled or pieced together from other memories, while everything unrelated survives. The Bemula benchmark tests exactly this and reports two common failures: the fact stays extractable with simple prompts, or the system deletes far more than it was asked to.

Measuring it

Recall@k. Of the questions asked, how often the right memory appears somewhere in the top k results. Measures “did we find it at all.”

MRR (mean reciprocal rank). Rewards putting the right memory near the top: 1 if it’s first, ½ if second, ⅓ if third, averaged over all questions.

LoCoMo. A benchmark of very long, multi-session conversations with questions across single-hop, multi-hop, temporal, open-domain and adversarial categories. See the LoCoMo TL;DR.

Missing a term, or think one of these is off? I’d like to hear it — I’ll keep this page up to date.

Paper TL;DR: Hopfield (1982) — Memory as a Valley You Roll Into

2026-10-18T22:00:00+00:00

Paper: J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, PNAS 79(8), 1982

One-line TL;DR: connect lots of simple on/off neurons with symmetric weights and the network gets an energy that only goes down as it runs — so stored memories become stable valleys, and a partial or noisy cue rolls downhill into the full memory.

Almost everything I’ve built around memory started here. It’s also one of the most readable classic papers I know: five pages, very little math, a physicist explaining computation.

The setup

Neurons are either on or off (+1 or −1).
Every neuron connects to every other, and the connection strengths are symmetric: the weight from A to B equals the weight from B to A.
Storing a memory uses a Hebbian rule — “neurons that fire together wire together.” For each pattern you want to remember, strengthen connections between neurons that agree and weaken those between neurons that disagree. Store several patterns by adding up their contributions.
Recalling means picking neurons one at a time and flipping each to agree with the weighted vote of its neighbors, until nothing changes.

The key insight: an energy that only goes down

Hopfield’s move was to define an energy for the whole network’s state and show that, with symmetric weights and one-at-a-time updates, every update either lowers the energy or leaves it unchanged. The network can’t cycle forever; it has to settle.

Stored patterns sit at the bottoms of valleys in that energy landscape. Start anywhere nearby — a corrupted photo, half a phone number — and the network slides down into the nearest stored memory. That’s content-addressable memory: you retrieve by what you remember, not by where it’s stored. It’s also robust: knock out some neurons or connections and recall still mostly works.

Diagram: Storing and recalling a memory in a Hopfield network (see it in the post)
Storing: the Hebbian rule strengthens connections between neurons that agree, and several patterns are stored by adding up their contributions. Recalling: start from a noisy cue in which some neurons are flipped; neurons update one at a time to agree with the weighted vote of their neighbors, so the wrong ones flip back until the stored pattern reappears and nothing changes. Underneath, the network's energy only goes down or stays the same with each update, like rolling downhill into a valley, and it settles at the bottom, which is the stored memory.
Illustration with 25 neurons: store with the Hebbian rule, then a noisy cue rolls downhill into the stored pattern.

Twenty-five lines of NumPy

Store five random patterns in 100 neurons, flip 30% of the bits of one, and let it settle:

import numpy as np

rng = np.random.default_rng(1)
N, P = 100, 5
patterns = rng.choice([-1, 1], size=(P, N))
W = (patterns.T @ patterns) / N        # Hebbian storage
np.fill_diagonal(W, 0)                 # no self-connections

def energy(s):
    return -0.5 * s @ W @ s

def recall(s, sweeps=5):
    s = s.copy()
    for _ in range(sweeps):
        for i in rng.permutation(N):   # one neuron at a time
            s[i] = 1 if W[i] @ s >= 0 else -1
    return s

target = patterns[0]
noisy = target.copy()
noisy[rng.choice(N, 30, replace=False)] *= -1

out = recall(noisy)
print((noisy == target).mean(), (out == target).mean())   # 0.7 -> 1.0
print(round(energy(noisy), 1), round(energy(out), 1))     # -8.6 -> -50.5

70% correct goes to 100% correct, and the energy drops as it does.

The limits

Capacity is small. Hopfield found the network reliably holds only about 0.15 memories per neuron. Push past that and recall falls apart.
Spurious memories. The landscape also grows valleys nobody stored — blends of real memories — and a cue can get stuck in one.
Similar memories interfere. Patterns that overlap a lot blur into each other.

Those limits are exactly what the 2020 follow-up fixes: modern Hopfield networks sharpen the valleys so capacity grows exponentially — and their update turns out to be transformer attention.

Why it still matters to me

Databases retrieve by key. Vector stores retrieve by nearest neighbor. Hopfield networks retrieve by settling — pattern completion from a partial cue, which is much closer to how remembering feels. That idea is the reason Attractor is called Attractor.

That completes the six papers from Part 1. The whole list, with short TL;DRs, lives on my papershelf. The series keeps going with MemGPT, which treats an LLM’s context window like an operating system’s memory.

For a one-paragraph definition of associative (content-addressable) memory, see the agent memory glossary.

Paper TL;DR: Sentence-BERT — The Quiet Workhorse Behind Semantic Search

2026-10-18T21:00:00+00:00

Paper: Nils Reimers & Iryna Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)

One-line TL;DR: fine-tune BERT so that every sentence gets its own fixed-size embedding you can compare with cosine similarity — turning semantic search over thousands of sentences from hours into seconds.

It’s the least flashy paper in this series and the one my code depends on most. Every time Attractor, VaultMem or daylog “searches by meaning,” it’s using a model descended from this paper.

The problem: BERT is a slow judge

In 2019, BERT was the best tool for deciding whether two sentences meant the same thing — but only if you fed it both sentences together. That works for scoring one pair. It falls apart for search.

The paper’s example: to find the most similar pair in a collection of 10,000 sentences, BERT would need around 50 million pairwise passes — roughly 65 hours on a GPU. And you can’t precompute anything, because every score depends on both sentences at once.

The obvious shortcut — run each sentence through BERT alone and average its outputs (or take the special [CLS] token) — produced embeddings that were often worse than old-fashioned averaged word vectors.

The fix: train it to embed

Sentence-BERT puts two copies of the same BERT side by side (a siamese network — shared weights), runs one sentence through each, pools each output into a single vector, and trains the whole thing so that:

sentences that mean similar things land close together, and
sentences that don’t land far apart.

They trained on natural-language-inference data (pairs labeled entailment / contradiction / neutral) and on similarity-scored pairs. Afterward you throw away the “pair” setup: each sentence goes through the model once, you store its vector, and comparing two sentences is a cheap cosine similarity.

That same 10,000-sentence search drops to about 5 seconds.

Diagram: Plain BERT versus Sentence-BERT for comparing sentences (see it in the post)
Top: plain BERT reads both sentences together and outputs one similarity score for that pair, so nothing can be precomputed; finding the most similar pair among 10,000 sentences takes about 50 million passes, roughly 65 hours on a GPU. Bottom: Sentence-BERT runs each sentence through BERT on its own, with two copies sharing the same weights, pools the output into one vector per sentence, and compares the two vectors with cosine similarity. Each vector is computed once and stored, so the same search takes about 5 seconds.
Plain BERT has to re-read every pair together. Sentence-BERT gives each sentence its own vector once, so comparing two is just a cosine similarity.

Why it still matters

It made vector search practical. Embed once, store in an index (pgvector, Qdrant, HNSW), and query in milliseconds. That’s the foundation of nearly every RAG system and agent memory store.
The library outlived the paper. The sentence-transformers package that came out of this work is still how most people get embedding models running in a few lines.
Small models are often enough. The 384-dimension model I use in Attractor runs comfortably on a CPU — handy when the whole point is keeping data on hardware you own.

Caveats worth knowing

Bi-encoders like SBERT trade a little accuracy for speed. When precision matters, a common pattern is to retrieve with embeddings, then re-rank the top results with a slower cross-encoder.
“Similar meaning” is whatever the training data said it was. Domain-specific text (logs, configs, medical notes) often needs a domain-tuned model.
Embeddings find related text; they don’t reason. Multi-hop questions — the ones LoCoMo showed are hardest — need more than nearest neighbors.

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems, LoCoMo and Memory as Asset. Next in the series: Hopfield (1982), the original associative memory that recalls from a partial cue.

Embeddings, HNSW and re-ranking each get a short definition in the agent memory glossary.

Paper TL;DR: Memory as Asset — Whose Memory Is It Anyway?

2026-10-18T20:00:00+00:00

Paper: Yanqi Pan, Qinghao Huang & Weihao Yang, Memory as Asset: From Agent-centric to Human-centric Memory Management (arXiv, March 2026)

One-line TL;DR: the memory an AI keeps about you is designed today as the agent’s internal resource; this paper argues it should be your asset — owned by you, shared only under your permissions, and able to grow over time.

This is the paper that reshaped VaultMem. I found it while I was stuck on a vague feeling that something about agent memory was backwards, and it gave that feeling a name.

The problem: agent-centric memory

Most memory systems treat what an agent remembers about you as part of the agent — stored by the platform, shaped for the platform’s use, and stuck there. The authors call this agent-centric memory. The consequences are familiar: you can’t take your history with you, you can’t decide who sees it, and every product rebuilds its own isolated “memory island” about the same person.

The proposal: memory as a personal asset

They propose flipping the default, built on three principles:

Memory in hand — individuals own and control their memories, not centralized platforms.
Memory groups — memories can be shared and combined, but only through permissions the owner sets.
Collective memory evolution — memory isn’t frozen: it’s recorded, consolidated and abstracted into higher-level knowledge over time, and can be exchanged between groups under control.

Each memory object carries its content, context, an explicit owner, an access policy and a version. Ownership and permissions are part of the data itself, not an afterthought in some admin panel.

The three layers

Personal storage — fast storage of each person’s private memory objects.
Evolution — updates memory from interactions: recording experiences, merging related memories, abstracting knowledge. It can run locally for privacy or remotely for compute.
Memory exchange network — a decentralized way to discover, authenticate and exchange memory assets.

Notably, the paper doesn’t insist everything stays on your device. If a provider hosts your memory, it should be encrypted so only you can read it.

Diagram: Memory as Asset: memory objects and the three layers (see it in the post)
Each memory object carries its content, its context and a version, plus an explicit owner and an access policy, so ownership travels with the data. Memory objects are kept in layer 1, personal storage: each person's private memory objects. VaultMem implements this layer, encrypted with a key only the user holds. Layer 2, evolution, updates memory from interactions: it records experiences, merges related memories and abstracts knowledge, and can run locally or remotely. Layer 3, the memory exchange network, discovers, authenticates and exchanges memory assets; the paper sketches it rather than solving it. Overall: owned by you, shared only through your permissions.
The paper's design in one picture: owner and access policy live inside every memory object, and three layers store, evolve and exchange those objects. VaultMem is the first layer.

How it shaped what I built

That last point is where VaultMem lives. I treat it as an implementation of the first layer: the platform stores your memories, but they’re encrypted with a key only you hold, so the operator cryptographically can’t read them. Everything in the VaultMem series — client-side encryption, encrypted search, keeping PII away from the model provider — follows from taking “memory in hand” literally.

Caveats worth knowing

The authors are upfront that this is an early draft: the terms and design are still evolving.
It’s a conceptual and architectural paper — there are no benchmarks or experiments yet. The claim that existing systems meet at most two of the three principles is an argument, not a measurement.
The exchange network raises hard questions (trust, incentives, revocation) that the paper sketches rather than solves.

I still think it’s the right question to ask before building any memory feature: whose memory is this?

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems and LoCoMo. Next in the series: Sentence-BERT, the sentence embeddings behind most semantic search.

Zero-knowledge storage and client-side encryption are defined in the privacy section of the agent memory glossary.

Paper TL;DR: LoCoMo — How Good Is Your Agent's Memory, Really?

2026-10-18T19:00:00+00:00

Paper: Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents (ACL 2024)

One-line TL;DR: LoCoMo is a benchmark of very long, multi-session conversations with questions that probe whether a system really remembers — and even strong long-context models and retrieval setups fall well short of people on it.

If you’re building memory for an agent, this is the paper that keeps you honest.

What’s in the benchmark

Most chat benchmarks are a few turns long. LoCoMo conversations run for dozens of sessions and hundreds of turns, spread over simulated weeks and months. The authors generated them with LLM agents that each had a persona and a timeline of life events, then had people review and fix them so the conversations stay consistent. Some turns include shared images, too.

On top of the conversations they built three tasks: answering questions, summarizing events, and generating responses. The question-answering part is the one I care about, and its categories are what make it useful:

Single-hop — one fact from one place in the history.
Multi-hop — needs to connect facts from different sessions.
Temporal — when did something happen, or what happened before or after.
Open-domain — combine the conversation with general knowledge.
Adversarial — questions about things that never happened, where the right answer is “that wasn’t mentioned.”

Diagram: How the LoCoMo benchmark is built (see it in the post)
Each LoCoMo conversation averages about 300 turns and 9 thousand tokens, over up to 35 sessions spread across simulated weeks and months. Three tasks are built on top: question answering, event summarization, and multi-modal dialogue generation. Question answering splits into five categories: single-hop, one fact from one place; multi-hop, connecting facts across sessions; temporal, when something happened or what came before or after; open-domain, combining the conversation with general knowledge; and adversarial, questions about things that never happened, where the right answer is that it was not mentioned.
Scoring each category on its own keeps a strong overall number from hiding weak temporal or multi-hop answers.

What they found

Even with long context windows or retrieval-augmented generation, models trail human performance by a wide margin. Retrieval helps, but the hard categories stay hard — especially reasoning about time and resisting the urge to make up answers to adversarial questions. Bigger context alone doesn’t fix it.

How I used it

When I built VaultMem, my first benchmark was 18 synthetic memories I wrote myself. It looked great — which was the problem. LoCoMo gave me a test I didn’t design. On a sample of its questions, retrieval was strong for single-hop facts and weakest for multi-hop, which needs pieces that weren’t individually asked about. That’s a retrieval problem before it’s a model problem, and it changed what I worked on next.

The takeaway for builders

Test against data you didn’t write. A memory system always looks good on its author’s examples.
Score by category, not just overall. “85% accurate” hides that temporal and multi-hop questions might be failing.
Measure the “I don’t know.” A memory that confidently invents a past is worse than one that admits a gap.

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, and Complementary Learning Systems. Next in the series: Memory as Asset, on who should own an agent’s memory.

Recall@k and MRR, the usual ways to score retrieval, are explained in the agent memory glossary.

Paper TL;DR: Complementary Learning Systems — Why Agents Need Two Kinds of Memory

2026-10-18T18:00:00+00:00

Paper: McClelland, McNaughton & O’Reilly, Why there are complementary learning systems in the hippocampus and neocortex, Psychological Review (1995)

One-line TL;DR: the brain learns with two systems on purpose — a fast one that grabs specific experiences and a slow one that gradually extracts general knowledge — because a single system can’t do both without wrecking what it already knows.

It’s a 1995 psychology paper, and it’s one of the most useful things I’ve read about designing memory for AI agents.

The problem: learning fast breaks things

The paper starts from a puzzle that anyone who has fine-tuned a neural network will recognize. If you teach a network something new by adjusting the same shared weights that hold everything else, the new learning can overwrite the old. This is catastrophic interference: learn B, forget A.

The fix that works in artificial networks is to learn slowly and interleave — mix new examples with old ones so the weights find a setting that serves both. But that’s slow. You can’t wait for hundreds of repetitions to remember where you parked this morning.

The brain’s answer: two learners

The authors argue the brain splits the job:

Hippocampus — the fast learner. Stores specific episodes quickly, often after a single exposure, using sparse, separate codes so similar memories don’t collide. Great for “what happened at Tuesday’s meeting.”
Neocortex — the slow learner. Learns gradually, with overlapping representations that capture structure shared across many experiences. Great for “how meetings at my company usually go.”

The bridge between them is replay. Over time — notably during sleep — the hippocampus reactivates recent episodes, and the neocortex learns from those replays interleaved with older knowledge. That’s consolidation: specific memories slowly become general knowledge without overwriting what came before.

Diagram: Two learners connected by replay (see it in the post)
A new experience is written quickly into the hippocampus, the fast learner that stores specific episodes; in an agent this is an episodic log or vector store. Replay, interleaved with older knowledge, slowly teaches the neocortex, the slow learner that extracts general structure; in an agent this is consolidated facts and preferences.
The two-learner idea from the paper, with the agent-memory equivalent of each part.

The paper backs this with evidence from people with hippocampal damage, who struggle to form new episodic memories yet keep older knowledge, and with network simulations showing why interleaved learning avoids interference.

Why it matters for AI agents

Read with agents in mind, the paper is basically a design doc:

Keep an episodic store separate from the model. Fine-tuning a model on every conversation is the “learn fast in shared weights” mistake. A separate, fast-writing memory — a log or vector store of specific events — is the hippocampus.
Consolidate on a schedule, not on every write. Periodically summarize or distill many episodes into durable facts and preferences, mixing in what’s already known instead of overwriting it.
Let memories change over time. Things that keep getting recalled should get stronger; one-off details can fade. A store that treats every memory as equally permanent forever is missing half the design.

This is the model I borrowed for Attractor: memories carry the context they were formed in, track how often they’re used, and are meant to strengthen or decay rather than sit frozen.

Caveats worth knowing

The theory has been extended since 1995 — the neocortex can learn new things quickly when they fit an existing schema — so “neocortex is always slow” is a simplification.
Brains aren’t databases; the analogy is a guide for design choices, not a blueprint.

Next in the series: the benchmark that told me how good my own memory system actually was. Earlier parts: six papers on AI memory and Hopfield Networks Is All You Need.

Quick definitions of consolidation and decay are in the agent memory glossary.

Paper TL;DR: Hopfield Networks Is All You Need — Attention Is a Memory Lookup

2026-10-18T17:00:00+00:00

Paper: Ramsauer, Schäfl, Lehner et al., Hopfield Networks is All You Need (2020)

One-line TL;DR: a continuous version of the 1982 Hopfield network can store exponentially many memories, usually retrieves one in a single step — and that single step is exactly the attention operation inside every transformer.

This is the paper that made me stop thinking of Hopfield networks as history. Here’s the idea in plain language, plus a tiny demo you can run.

The old idea: memories as valleys

In Hopfield’s 1982 network, you store patterns (say, strings of +1 and −1) in the connection weights. The network has an “energy,” and each stored pattern sits at the bottom of a valley. Start from a corrupted pattern, let the network update, and it rolls downhill into the nearest stored memory. That’s content-addressable memory: you find a memory by giving a piece of it, not by knowing its address.

The catch: the classic network only holds about 0.14 patterns per neuron. Store more than that and the valleys blur together into spurious memories.

The new idea: sharpen the valleys

Ramsauer et al. swap the old quadratic energy for one built on log-sum-exp — a smooth version of “max.” Intuitively, each stored pattern now carves a much sharper, deeper valley, so many more of them fit without interfering. The paper shows the capacity grows exponentially with the dimension of the patterns, and that retrieval typically converges after one update.

That one update is:

new_state = X · softmax(β · Xᵀ · state)

where the columns of X are the stored patterns and β controls how sharp the valleys are.

Wait — that’s attention

Now look at transformer attention:

output = V · softmax(Kᵀ · q / √d)

Set the query q to the current state, use the stored patterns as both keys K and values V, and pick β = 1/√d. It’s the same equation. Every attention head in an LLM is doing a one-step associative memory lookup: compare the query against everything stored, softmax the similarities, return a weighted blend that’s usually dominated by the best match.

The paper goes further, showing how different heads and layers behave like different retrieval regimes — some settle on one memory, some average over many — but the core equivalence is the part that stuck with me.

Twenty lines of NumPy

Store five random ±1 patterns, flip 40% of the bits in one of them, and do a single update:

import numpy as np

rng = np.random.default_rng(0)
d, n = 64, 5
X = rng.choice([-1.0, 1.0], size=(d, n))   # stored memories (columns)

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

def retrieve(xi, beta=0.5):
    return X @ softmax(beta * X.T @ xi)      # one modern-Hopfield update

target = X[:, 2]
noisy = target.copy()
noisy[rng.choice(d, size=int(0.4 * d), replace=False)] *= -1

out = retrieve(noisy)
print("overlap before:", int(noisy @ target), "/", d)       # 14 / 64
print("overlap after: ", int(np.sign(out) @ target), "/", d) # 64 / 64

One step takes a pattern that’s only loosely related to the target back to a perfect copy. The softmax weights tell the story: about 0.97 on the right memory and crumbs on the rest — exactly what an attention map looks like.

Diagram: One modern Hopfield update, step by step (see it in the post)
A noisy state, the query, starts with 40 percent of its bits flipped and an overlap of 14 out of 64 with stored pattern x3. It is compared against all five stored patterns x1 to x5, which act as the keys. A softmax with sharpness beta turns the similarities into weights: about 0.97 on x3 and crumbs on the rest. The output blends the stored patterns, which act as the values, by those weights. After this single step the new state matches x3 with an overlap of 64 out of 64. This is the same equation as transformer attention with the query set to the state, keys and values set to the stored patterns, and beta equal to one over the square root of d.
The twenty-line demo as a picture: one comparison, one softmax, one blend — and the noisy pattern snaps back to the memory it came from.

Why I care, as someone building agent memory

Most agent memory today is “embed everything, return the top-k nearest neighbors” (usually through an index like HNSW). This paper reframes that: nearest-neighbor search is a crude, non-differentiable cousin of what attention already does well. It nudged me to build Attractor around retrieval by partial cues and context, and to think of memory as something that settles rather than something you look up.

Caveats worth knowing

“Exponential capacity” is a statement about well-separated patterns in high dimensions; real embeddings are messy and correlated.
The one-step convergence holds when β is large enough relative to how similar the stored patterns are — similar memories still blur together.
Treating attention as memory is a lens, not a full explanation of what transformers learn.

If you read only one modern paper about associative memory, make it this one. Next in the series: Complementary Learning Systems, the brain’s split between a fast and a slow learner. The rest of what I’m reading is on my papershelf, and Part 1 of this series covers the other papers behind my memory projects.

Terms like top-k retrieval and associative memory are defined in plain language in the agent memory glossary.

Paper TL;DR: Six Papers That Shaped How I Think About AI Memory

2026-10-18T16:00:00+00:00

Most of my side projects over the past year have circled one question: how should an AI system remember things? Not just “where do we put the vectors,” but how memory gets formed, found again, strengthened or forgotten — and who gets to read it.

I didn’t start with papers. I started building, got stuck, and went looking for how other people had thought about the problem. These six are the ones that actually changed what I built. Here’s each one in a few sentences, in plain language, plus what I took from it. If you only read one, read the first.

Diagram: Which of the six papers fed into which project (see it in the post)
Hopfield 1982, Modern Hopfield networks and Complementary Learning Systems fed into Attractor. Sentence-BERT fed into Attractor, daylog and VaultMem. LoCoMo and Memory as Asset fed into VaultMem.
Where each paper's takeaway landed, from the notes below. Sentence-BERT is the only one under all three projects.

1. Hopfield (1982) — memory as a place you roll into

Neural networks and physical systems with emergent collective computational abilities · PNAS

TL;DR: Wire up simple on/off neurons with symmetric connections and the network gets an “energy” that can only go down as it updates. Stored memories sit at the bottoms of valleys in that landscape. Give it a partial or noisy pattern and it slides downhill into the full memory. You don’t look memories up by key — you fall into them.

What I took from it: this was the “oh” moment behind Attractor. Recall from partial cues feels much closer to how I actually remember things than a database lookup does.

2. Ramsauer et al. (2020) — attention is a Hopfield update

Hopfield Networks is All You Need · arXiv

TL;DR: A “modern” Hopfield network with continuous states can store exponentially many patterns and usually retrieves one in a single step. And that single update step is mathematically the same operation as transformer attention.

What I took from it: the 1982 idea isn’t a museum piece — it’s sitting inside every LLM. That made me take attractor-style retrieval seriously as a design for agent memory, not just a curiosity.

3. McClelland, McNaughton & O’Reilly (1995) — two learners, not one

Why there are complementary learning systems in the hippocampus and neocortex · Psychological Review

TL;DR: The brain uses a fast learner that grabs specific episodes (hippocampus) and a slow learner that gradually pulls out general structure (neocortex). Replaying episodes into the slow system lets it learn without overwriting what it already knows.

What I took from it: a vocabulary for something agents get wrong — treating every memory the same forever. It’s why Attractor tracks access and why I think memories should strengthen with use and fade without it.

4. Reimers & Gurevych (2019) — embeddings you can actually search

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · arXiv

TL;DR: Plain BERT has to compare sentences two at a time, which is painfully slow for search. Training it in a “siamese” setup gives every sentence its own embedding you can compare with cosine similarity, so finding the closest match among thousands takes seconds instead of hours.

What I took from it: the unglamorous workhorse. The sentence-transformers models behind semantic search in Attractor, VaultMem and daylog all come from this line of work.

5. Maharana et al. (2024) — testing memory honestly

Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo) · arXiv

TL;DR: A benchmark of very long, multi-session conversations with questions that need single facts, multi-step reasoning, time awareness, or the sense to say “that never happened.” Even strong long-context models and RAG setups struggle with it.

What I took from it: my own little synthetic test was flattering me. Running VaultMem against LoCoMo gave me numbers I could actually trust — and showed me where retrieval was weak.

6. Pan, Huang & Yang (2026) — whose memory is it?

Memory as Asset: From Agent-centric to Human-centric Memory Management · arXiv

TL;DR: Argues that agent memory today is built for the agent rather than the person, and proposes treating the memory an AI keeps about you as an asset you own, share under your own permissions, and grow over time.

What I took from it: this one reshaped VaultMem entirely. It gave words to the feeling that an agent’s memory of your life shouldn’t be readable by whoever runs the agent. VaultMem ended up implementing its first layer.

What ties them together

Reading these in a row, I keep landing on the same picture: good memory is found by meaning and context, changes over time, and belongs to the person it’s about. Most systems today nail the first, ignore the second, and get the third backwards.

All six live on my papershelf with the rest of what I’m reading. If any of the vocabulary here is new, the agent memory glossary has short definitions. If there’s a paper on memory or agents you think I should read next, I’d genuinely love to hear about it.

Next in the series: Hopfield Networks Is All You Need, how modern Hopfield networks store exponentially many memories and why their update rule is transformer attention. Every part is listed on the Paper TL;DR page.

Attractor: Building a Memory Engine Inspired by How the Brain Works

2026-04-05T00:00:00+00:00

I’ve been thinking about memory — not human memory, but how machines store and retrieve it. Most systems today are either key-value stores (fast, exact, dumb) or vector databases (semantic, but still just nearest-neighbour). Neither of them works the way a brain does.

So I started building something different. I called it Attractor.

The Problem with How Machines Remember

When you search a database, you give it a key and it returns a value. When you search a vector database, you give it a query and it returns the closest vectors by distance (usually via an approximate index like HNSW). Both are useful. But neither captures something important about how biological memory works — context.

Think about how you remember things. You don’t retrieve memories by key. You retrieve them by partial cues — a smell, a phrase, a situation. And the retrieval is shaped by context: what you were doing, who you were with, what problem you were trying to solve. Two memories with identical content mean different things depending on when and why they were formed.

Machines don’t do this. And for AI agents that need to reason across long conversations or accumulated knowledge, this is a real limitation.

Hopfield Networks — The Intriguing Part

In 1982, John Hopfield described a type of recurrent neural network that stores memories as energy minima [1]. The idea is elegant: each memory is a stable state — a basin of attraction — in an energy landscape. When you give the network a partial or noisy input, it doesn’t search for the closest match. It converges — like a ball rolling downhill — to the nearest stable state.

This is called content-addressable memory. You don’t need the exact key. You need something close enough to fall into the right basin.

Diagram: Energy landscape: memories as basins of attraction (see it in the post)
A qualitative sketch. The vertical axis is energy and the horizontal axis is the space of possible states. The curve has two valleys. Memory A, "switched from JWT to session tokens", sits at the bottom of the first valley; Memory B, "auth tokens should never be stored in localStorage", sits at the bottom of the second. A query partway up the slope of the first valley rolls downhill and settles on Memory A, without needing an exact match.
Memories sit at the bottom of basins. A partial cue doesn't need an exact match: it rolls downhill into the nearest one.

The name Attractor comes directly from this. In dynamical systems, an attractor is a state that a system converges to over time. Hopfield memories are attractors. Retrieval is the process of finding which attractor your input belongs to.

What makes this intriguing is that it mirrors something real about biological memory. Your brain isn’t doing a database lookup when you remember something. It’s doing something closer to energy minimisation — pattern completion from partial cues.

What’s also interesting: in 2020, researchers showed that modern Hopfield networks with continuous states have exponentially higher storage capacity than the original binary version [2] — and that the attention mechanism in transformers is mathematically equivalent to a single Hopfield update step. The architecture powering every LLM today is, at its core, a Hopfield retrieval operation.

What I Built — Phase 1

Attractor is a standalone memory service. Right now it’s Phase 1: a working foundation that stores memories with context and retrieves them by meaning.

Diagram: Attractor Phase 1 architecture (see it in the post)
A client — memlog, an agent, or any HTTP client — calls a FastAPI service over REST. FastAPI exposes POST /memories, POST /memories/search, GET /memories/:id and DELETE /memories/:id. It uses an encoder (sentence-transformers all-MiniLM-L6-v2, producing a 384-dimensional vector) and PgStore (an asyncpg connection pool for write, get, search and delete). Both lead to PostgreSQL with pgvector, which stores content as TEXT, context and tags as JSONB, vector as vector(384), access_count as INT and created_at as TIMESTAMPTZ. The service runs on port 7747, "loci" on a phone keypad.
Phase 1: a FastAPI service that embeds each memory and stores it, with its context, in Postgres + pgvector.

The data model is built around something I call a ContextFrame — the situational state at the time a memory was formed:

class ContextFrame(BaseModel):
    topic: str | None      # what was being discussed
    entities: list[str]    # who or what was involved
    task: str | None       # what was being done
    session_id: str | None # which session it came from

This isn’t metadata. It’s a retrieval key. When you search, you can filter by context — “find memories from this topic” or “find memories from this session” — and combine that with semantic search over the content itself.

The storage layer uses PostgreSQL with pgvector [3]. Each memory stores its content, context frame, tags, timestamps, access count, and a 384-dimensional embedding from a sentence-transformers model [4]. The embedding enables semantic search — retrieving by meaning rather than exact words.

The API is simple:

POST /memories          — store a memory with context
GET  /memories/:id      — retrieve by ID
POST /memories/search   — search by content + context filters
DELETE /memories/:id    — remove a memory

The CLI wraps the service:

attractor start    # starts the service on port 7747
attractor connect  # verifies it's running

Port 7747 is “loci” on a phone keypad — a nod to the method of loci, the memory palace technique [5], which was the project’s original name before I found a better one.

A Concrete Example

Here’s what makes it different from a plain vector search. Say you store these two memories:

"auth tokens should never be stored in localStorage"
  context: { topic: "security review", task: "reviewing PR #42" }

"switched from JWT to session tokens — stateless was causing issues"
  context: { topic: "auth refactor", task: "architecture decision" }

Now you search: "why did we change the auth approach" with context_filter: { topic: "auth refactor" }.

Diagram: Context filtering: same query, different results (see it in the post)
The query "why did we change the auth approach" is embedded as a 384-dimensional vector. With vector search only, both memories surface: "switched from JWT to session tokens" and "auth tokens should never be stored in localStorage". Running the same query with the context filter topic equals "auth refactor" surfaces only the session-tokens memory; the localStorage memory, formed during a security review, is filtered out.
Same query, two searches. Vector search alone returns both auth memories; adding the context filter keeps only the one formed during the auth refactor.

The second memory surfaces — not because the words match, but because:

The embedding is semantically close to the query
The context filter narrows to the right topic

Without the context filter, both memories are relevant. With it, you get the one that was formed in the right situation. That’s the difference.

The Roadmap

Phase 1 is the foundation. The more interesting work is ahead:

Phase 2 — Association graph: Memories aren’t just points in a vector space. They’re nodes in a graph with typed edges — supports, contradicts, caused_by, followed_by. Retrieval triggers spreading activation: surface the memory you asked for, and the ones connected to it.

Phase 3 — Decay and consolidation: Not all memories are equally important. The access_count field is already tracking how often each memory is retrieved. Phase 3 uses this to implement decay — memories that are never accessed fade, memories that are frequently retrieved strengthen. This mirrors how biological memory consolidation works [6].

Phase 4 — Hopfield retrieval: Replace vector similarity with a proper Hopfield weight matrix. Retrieval becomes pattern completion — give the network a partial cue and it converges to the full memory. This is the step that moves from “semantic search with context” to something genuinely brain-inspired.

Isn’t This Already Being Done?

Honestly — yes, in parts. Vector search with metadata filtering exists in Pinecone, Weaviate, and Qdrant. Memory layers for agents exist in LangChain and LlamaIndex. And the “Hopfield Networks is All You Need” paper already showed that transformer attention is mathematically a Hopfield update — so in a sense, every LLM is already doing content-addressable retrieval implicitly.

Phase 1 of Attractor is not novel. It’s a clean, focused implementation of something that exists in various forms.

The bet is on Phases 3 and 4. Most agent memory systems don’t implement decay — memories don’t weaken over time unless reinforced, they don’t consolidate the way biological memory does, and retrieval is always ranking rather than convergence. That’s the gap Attractor is trying to close.

Whether that gap matters in practice is an open question — and the honest answer is that it depends entirely on whether the difference shows up in real use. That’s why I’m building memlog on top of Attractor as a daily-use testbed: log thoughts, decisions, and notes every day, and see whether context-aware retrieval with decay actually surfaces better results than a plain vector search would. If it does, the model is right. If it doesn’t, the model needs revision.

Theory is cheap. The feedback loop is the thing.

Why This Matters for AI Agents

Most AI agents today are stateless between sessions or rely on crude context windows. The ones that do have memory bolt it on as an afterthought — store the last N messages, embed them, retrieve by cosine similarity.

Attractor is an attempt to do it properly. Not because the engineering is especially hard, but because the model of memory matters. If you treat memory as a lookup table, you get lookup-table behaviour. If you treat it as an energy landscape with context and decay, you get something closer to how intelligence actually works.

Phase 1 is live. pip install attractor-engine — start storing memories with context, retrieve by meaning.

The interesting phases are coming.

References

Hopfield, J.J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558. Plain-language summary: Paper TL;DR.
Ramsauer, H., Schäfl, B., Lehner, J., et al. (2020). Hopfield Networks is All You Need. arXiv:2008.02217. (Shows modern Hopfield networks have exponential capacity and that transformer attention is a Hopfield update.) Plain-language summary: Paper TL;DR.
pgvector. (2021). Open-source vector similarity search for Postgres. GitHub.
Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084. Plain-language summary: Paper TL;DR.
Yates, F.A. (1966). The Art of Memory. University of Chicago Press. (The definitive account of the method of loci — the memory palace technique.)
McClelland, J.L., McNaughton, B.L., & O’Reilly, R.C. (1995). Why there are complementary learning systems in the hippocampus and neocortex. Psychological Review, 102(3), 419–457. (The foundational paper on memory consolidation and why some memories strengthen while others decay.) Plain-language summary: Paper TL;DR.

Source: github.com/aag1091-alt/attractor

Context frames, consolidation, decay and associative memory are each defined in a sentence or two in the agent memory glossary.

VaultMem Part 3 — Hiding PII from Your LLM Provider

2026-04-01T00:00:00+00:00

Part 1 covered the core encryption model. Part 2 covered multi-modal ingestion and pluggable backends. This post covers v0.3.x — a new layer of privacy that addresses a problem encryption at rest cannot solve.

Install: pip install 'vaultmem[spacy]' · Code: github.com/aag1091-alt/vaultmem-sdk · Live demo: vaultmem-demo.streamlit.app

The gap encryption at rest doesn’t close

VaultMem stores every memory atom encrypted with AES-256-GCM. Your passphrase never leaves your device. The platform operator cryptographically cannot read your vault. The threat model comes from the Memory-as-Asset framework (Pan, Huang & Yang, arXiv:2603.14212; plain-language TL;DR): memory is a personal asset, not a platform resource.

But there is a second exposure point I hadn’t fully addressed: the moment you decrypt a memory and inject it into an LLM prompt.

Say your vault contains:

I met Sarah Chen at the NeurIPS conference. She works on ML infra at Google.
My accountant James Park can be reached at james@parkaccounting.com.

When you ask your assistant “what do I know about Sarah?”, VaultMem retrieves those memories, decrypts them, and sends them as context to the LLM provider — Groq, OpenAI, Anthropic, whoever you’re using. At that point, Sarah’s name, her employer, and James’s email address are all sitting in plaintext in an HTTP request to a third-party cloud.

Encryption at rest doesn’t help here. The data is decrypted before it goes out the door.

The solution: strip PII before injection, restore it after

VaultMem v0.3.0 ships a Sanitizer class that sits between your vault and your LLM call.

from vaultmem import VaultSession, Sanitizer

san = Sanitizer(backend="spacy")   # one instance per session

with VaultSession.open("./my_vault", passphrase="...") as s:
    memories = s.search("sarah chen", top_k=5)
    context = "\n".join(f"- {m.atom.content}" for m in memories)

    # Sanitize context AND query with the same instance
    sanitized_context, rmap = san.sanitize(context)
    sanitized_query, rmap   = san.sanitize(user_query)

    raw_reply = llm.chat(sanitized_query, context=sanitized_context)

    # Restore real values in the LLM's response
    final_reply = san.restore(raw_reply, rmap)

The LLM provider receives:

I met Jordan at the NeurIPS conference. She works on ML infra at Globex.
My accountant Casey can be reached at [EMAIL_1].

It reasons about this correctly, responds naturally, and VaultMem restores the real values client-side before the answer is shown to the user.

The provider never sees the real names, companies, or contact details.

Diagram: How the VaultMem sanitizer round-trips a prompt (see it in the post)
On your device, memories are decrypted from the vault, for example Sarah Chen, who works at Google, and James's email address. Sanitizer.sanitize swaps them for a pseudonym, a fictional company and a typed token: Jordan, Globex and EMAIL_1, using one mapping for the whole session. Only that sanitized text goes to the LLM provider. The provider's reply mentions Jordan. On your device again, Sanitizer.restore maps Jordan back to Sarah Chen, so the answer you read uses the real names.
Real names and contact details stay on your device; the provider only ever reasons about Jordan, Globex and [EMAIL_1].

Pseudonyms, not tokens

Person names are replaced with natural-sounding pseudonyms — Jordan, Casey, Morgan, Riley, Avery — rather than opaque tokens like [PERSON_1].

The reason is fluency. If the LLM sees [PERSON_1] in context and addresses the user with it, the response reads awkwardly. With a pseudonym, it can write “Based on what Jordan told you at NeurIPS…” and after restoration the user reads “Based on what Sarah Chen told you at NeurIPS…” — completely natural.

Organisations get fictional company names (Acme, Globex, Initech, Pied Piper). Structured PII — emails, phone numbers, SSNs, credit cards, IPs — gets typed tokens ([EMAIL_1], [PHONE_2]) because there’s no natural-sounding substitute for those.

The mapping is session-scoped and stable: if “Sarah Chen” appears in five different memories, she is always Jordan in that session. The LLM can reason about her as a single consistent person across all memories.

Two backends

Presidio supports different NLP backends. VaultMem exposes both:

# Lightweight — ~12 MB spaCy model, no PyTorch dependency
# Works on Streamlit Cloud, CI, anywhere torch is unavailable
san = Sanitizer(backend="spacy")

# High-accuracy — ~400 MB transformers model, requires PyTorch
# Best for local deployments or privacy-critical production use
san = Sanitizer(backend="transformers")

The spaCy backend (en_core_web_sm) is the default. It handles the common cases — person names, organisations, locations, emails, phone numbers — and is small enough to deploy anywhere without a GPU or a multi-gigabyte dependency install.

# spaCy (default)
pip install 'vaultmem[spacy]'
python -m spacy download en_core_web_sm

# transformers (high accuracy)
pip install 'vaultmem[presidio]'

What gets skipped

Not everything Presidio detects is PII. Temporal expressions (DATE_TIME), cardinal numbers, percentages, monetary values, and quantities are identifying in context but not on their own — and redacting them breaks the coherence of the context significantly.

VaultMem v0.3.2 introduced a skip list:

_SKIP_ENTITY_TYPES = frozenset({
    "DATE_TIME", "CARDINAL", "ORDINAL",
    "PERCENT", "MONEY", "QUANTITY", "LANGUAGE",
})

Before this, a memory like “Every Friday at 4pm I do a weekly review — takes 20 minutes” was returned to the LLM as “Every [DATE_TIME_3] at [DATE_TIME_4] I do a weekly review — takes [DATE_TIME_5]”. Correct in theory, unreadable in practice. The skip list fixes this.

The same release fixed two spaCy-specific label mismatches: spaCy reports organisations as ORGANIZATION (not ORG) and geopolitical entities as GPE (not LOCATION). Before the fix these fell through to typed tokens instead of receiving pseudonyms from the pools.

A subtle bug: case-insensitive deduplication

During testing I hit a bug that’s easy to miss until you actually run the system with real queries.

The memory stored "Sarah Chen" (title case). The user asked "who is sarah chen?" (lowercase). Presidio detected "sarah chen" as a PERSON — but the forward map looked it up with an exact string match. "sarah chen" != "Sarah Chen", so it was treated as a new entity and assigned a fresh pseudonym: Casey.

The LLM then received context about Jordan but a query about Casey. Its response: “There is no information about Casey.” Technically accurate. Completely useless.

The fix in v0.3.3: all forward-map keys are normalised to lowercase. A separate canonical map preserves the first-seen capitalisation so restoration returns "Sarah Chen", not "sarah chen". The variant seen first wins on casing; all subsequent variants map to the same pseudonym.

def _assign(self, real: str, entity_type: str) -> str:
    key = real.lower()
    if key in self._forward:
        return self._forward[key]          # "sarah chen" hits "Sarah Chen"'s entry
    self._canonical[key] = real            # first-seen casing for restoration
    ...

What this doesn’t solve

The Sanitizer addresses what leaves your device. It doesn’t address what the LLM infers or reconstructs. If your memory says “I grew up two blocks from the Eiffel Tower”, stripping names doesn’t prevent the LLM from inferring something about where you’re from.

The complete solution is TEE-based inference — running the model inside a trusted execution environment so the provider’s infrastructure never sees plaintext at all. That’s the direction the research is heading (see the arXiv paper above), but it’s not production-ready for general-purpose models yet.

PII stripping is the practical near-term answer. It’s not perfect, but it meaningfully shrinks the surface area of personal data your LLM provider sees on every query.

Changelog

v0.3.0 — Sanitizer class, transformers backend, pseudonym pools, restoration map
v0.3.1 — spaCy backend (en_core_web_sm), new [spacy] optional extra, backend= parameter
v0.3.2 — DATE_TIME skip list, ORGANIZATION/GPE label fixes for spaCy
v0.3.3 — case-insensitive entity deduplication

pip install 'vaultmem[spacy]' gets you the latest.

If you’re building an AI agent that touches personal data, I’d love to hear what you’re working on.

For short definitions of PII sanitization, zero-knowledge storage and the other terms in this series, see the agent memory glossary.

Your AI Agent Shouldn't Be Able to Read Your Diary

2026-03-28T00:00:00+00:00

Imagine you have a personal AI assistant. Over months, you tell it everything — your health worries, your relationship friction, your career anxieties, the habits you’re trying to break. It remembers. It helps.

Now ask yourself: who else can read that?

The honest answer, with every AI memory product that exists today, is: the company running it.

The problem isn’t the AI. It’s where the memory lives.

When you use an AI agent with persistent memory — whether that’s a health companion, a journaling assistant, a personal coach, or a productivity tool — your conversations get stored somewhere. That “somewhere” is a server owned by the platform. And the platform can read it.

This is true even when there’s a privacy policy. A privacy policy is a promise. It can be broken, subpoenaed, acquired, or quietly changed. A cryptographic guarantee cannot.

I spent a few weeks building something that fixes this at the foundation:

VaultMem — a Python library that gives AI agents encrypted, persistent, portable memory where the platform operator mathematically cannot read what users store.

Not “we promise not to look.” Not “we anonymize your data.” The key never leaves your session. The math makes it true regardless of what the policy says.

A real-world example: the health companion

Say you’re building a personal health AI — the kind of agent a person with a chronic condition might use to track symptoms, medications, and how they’re feeling day to day.

Without VaultMem, your architecture probably looks like this:

User tells agent “my migraines have been worse since I started the new medication”
Agent stores that in a database you control
You — the developer — can query that database
So can anyone who compromises it, or a legal subpoena, or your future acquirer

With VaultMem, it looks like this:

from vaultmem import VaultSession, LocalEmbedder

with VaultSession.open("./alice_vault", passphrase="alice_secret",
                        embedder=LocalEmbedder()) as s:
    s.add("Migraines worse since starting new medication on March 15th")
    s.add("Blood pressure 120/80 at checkup — within normal range")
    s.add("Feeling anxious about the scan results, haven't slept well")

    results = s.search("recent health concerns", top_k=3)

You — the developer — receive an encrypted blob you cannot open. Alice holds the key. Her memories are hers.

Diagram: Who can read an agent's memory, with and without VaultMem (see it in the post)
Typical agent memory: what you tell the agent is stored in the platform's database, readable by the platform, so the developer, anyone who breaches it, a subpoena or a future acquirer can read every memory. With VaultMem: you hold the key, the VaultSession encrypts in your session, and the platform stores only an encrypted blob, so the developer, a breach, a subpoena or an acquirer see only encrypted bytes.
Same app, same people around it. The difference is where the key lives: with VaultMem it never leaves your session.

The same principle applies to a therapist AI, a relationship coach, a financial planner, a personal journal. Anywhere the content is deeply personal and the user has good reason not to trust a third party with it.

What it does

The library handles the full memory lifecycle — not just encryption:

Four memory types auto-classified from content: episodic (events), semantic (facts), persona (preferences), procedural (how-tos)
Smart retrieval that surfaces recurring habits and patterns above one-off mentions — so “my morning coffee” surfaces a pattern summary, not just the last time you mentioned coffee
Time-aware search — ask “what was I doing last summer?” and it searches by when events actually happened, not when you typed them in
Multi-modal — photos, voice notes, PDFs ingested as encrypted memory atoms with EXIF dates and GPS stripped from metadata
Portable — the .vmem format is open and self-describing; your vault can move between apps
Production backends — S3 for storage, Postgres for indexing, HNSW for fast search at scale

Three core dependencies. No mandatory cloud service. No phone-home.

It’s early — and I’d love feedback

This is a work in progress. The library is usable and the core is solid, but I’m still building toward the vision.

Try the interactive demo (no install needed): vaultmem-demo.streamlit.app

It runs a pre-seeded vault with 36 example memories and shows you both sides — what you see (meaningful memories retrieved by your question) and what the platform sees (encrypted bytes that reveal nothing).

Install the SDK:

pip install vaultmem

Read the technical writeups:

The research: VaultMem implements Layer 1 of the Memory-as-Asset framework — a formal model for user-owned AI memory (plain-language TL;DR). The preprint is on Zenodo.

If you’re building an AI agent and want to give your users memory that’s genuinely theirs — or if you have thoughts on where this should go — I’d love to hear from you.

VaultMem Part 2 — Multi-Modal Memory, Temporal Search, and Pluggable Backends

2026-03-28T00:00:00+00:00

Part 1 covered why I built VaultMem, how the encryption works, and the core memory model. This post covers what came next in v0.2.0 and v0.2.1 — the parts that took the library from a solid foundation to something that can handle real-world agent use cases: multi-modal memory ingestion, time-aware retrieval, production-grade storage backends, O(log N) ANN search, and a pluggable query normalizer.

Install: pip install vaultmem · Code: github.com/aag1091-alt/vaultmem-sdk · Live demo: vaultmem-demo.streamlit.app

Text-only memory is a significant limitation for a personal AI agent. The moments that matter most — a photo from a trip, a voice note recorded mid-commute, a scanned medical document — aren’t text.

VaultMem v0.2.0 added add_media(), which ingests an image, audio file, PDF, or video and stores it as a fully encrypted memory atom. The raw bytes go into vault/media/{uuid}.enc (AES-256-GCM, same key architecture as text atoms). The extracted content becomes the searchable atom.

with VaultSession.open("./vault", passphrase, embedder=embedder) as s:
    # Image — extracts EXIF date, GPS coordinates, optional OCR
    atom = s.add_media("holiday_photo.jpg", passphrase)
    print(atom.content)      # "Photo taken 2023-07-14 in Split, Croatia"
    print(atom.captured_at)  # EXIF timestamp as Unix int
    print(atom.location)     # {"lat": 43.5, "lon": 16.4, "place": "Split"}

    # Audio — Whisper transcription
    atom = s.add_media("voice_note.mp3", passphrase)

    # PDF — PyMuPDF text extraction
    atom = s.add_media("contract.pdf", passphrase)

    # Retrieve raw bytes later (decrypted on demand, not in search results)
    raw = s.get_media(atom.id, passphrase)

The media bytes are never loaded into RAM during search — only the extracted text atom participates in embedding and retrieval. This keeps the privacy model intact: even if you extract metadata, the raw file stays encrypted behind the passphrase.

Each extractor is a pluggable Protocol, so adding new file types is straightforward:

from vaultmem import MediaExtractor, MediaExtractionResult, MediaIngester

class HeicExtractor:
    MIME_TYPES = {"image/heic", "image/heif"}

    def extract(self, path, mime_type):
        return MediaExtractionResult(
            content_type=mime_type,
            transcript="...",
            captured_at=...,
        )

ingester = MediaIngester(extra_extractors=[HeicExtractor()])

Install the extras to activate the built-in extractors:

pip install "vaultmem[media]"   # Pillow, piexif, openai-whisper, PyMuPDF, ffmpeg-python

Temporal search

The second gap I wanted to close: time. A personal agent should be able to answer “what was I doing in 2019?” or “show me everything from last summer” — not just “what’s most semantically similar to this query?”

This required two new timestamp fields on MemoryObject:

created_at — when the atom was written to the vault (set automatically, immutable)
captured_at — when the real-world event happened (set manually or from EXIF/metadata)

These are deliberately separate. A photo taken in 2019 but added to the vault today has captured_at in 2019 and created_at today. The distinction matters for any use case involving retroactive journaling or importing old media.

Browse by time window

from datetime import datetime, timezone

def ts(year, month=1, day=1):
    return int(datetime(year, month, day, tzinfo=timezone.utc).timestamp())

with VaultSession.open("./vault", passphrase, embedder=embedder) as s:
    memories_2019 = s.search_by_time(ts(2019), ts(2020))
    q3_2021 = s.search_by_time(ts(2021, 7), ts(2021, 10))

    # What was written to the vault in the last 24 hours
    import time
    recent = s.diff(int(time.time()) - 86400, int(time.time()))

Natural-language time queries

I wanted to avoid the experience of asking “what happened summer 2019?” and getting results from all time. The TimeQueryParser extracts a timestamp range from the query string, strips the time phrase, and passes the remainder to the embedding — all in pure Python with no dependencies:

from vaultmem import TimeQueryParser

from_ts, to_ts, remainder = TimeQueryParser.parse("what happened summer 2019")
# from_ts  → 2019-06-01 UTC
# to_ts    → 2019-09-01 UTC
# remainder → "what happened"

Supported phrases: season + year, month + year, bare year, “last year”, “this month”, “last N days”, “yesterday”.

Pass parse_time=True to search() and it handles everything automatically:

# Strips "in 2019" → timestamp range; embeds "what did I do"
results = s.search("what did I do in 2019", top_k=10, parse_time=True)

Diagram: How VaultMem splits a natural-language time query (see it in the post)
The query "what happened summer 2019" goes to TimeQueryParser.parse, which runs in pure Python with no dependencies. It splits the query in two. The time phrase, summer 2019, becomes a timestamp range from 2019-06-01 to 2019-09-01 UTC. The remainder, what happened, is what gets embedded. search with parse_time=True uses both, so the results come from summer 2019 instead of from all time.
The time phrase becomes a timestamp range; only what's left of the query is embedded.

Pluggable storage backends

The default storage model — two files on the local filesystem — is the right starting point. But a production agent deployment might need S3 for durability, Postgres for searchability, or both. I didn’t want to bake those in as optional flags on the session object; I wanted them to be proper extension points.

v0.2.0 introduced three ABCs:

ABC	Default	Production
`BlobStore`	`FileBlobStore`	`S3BlobStore`
`SearchIndex`	`SQLiteSearchIndex`	`PostgresSearchIndex`
`VectorIndex`	— (exact O(N) cosine)	`HNSWVectorIndex`

All three are injected at session creation:

from vaultmem import (
    VaultSession, LocalEmbedder,
    S3BlobStore, PostgresSearchIndex, HNSWVectorIndex,
)

with VaultSession.open(
    "./vault", passphrase,
    embedder=LocalEmbedder(),
    blob_store=S3BlobStore(bucket="my-vault-atoms"),
    search_index=PostgresSearchIndex("postgresql://user:pass@host/db"),
    vector_index=HNSWVectorIndex(),
) as s:
    results = s.search("coffee habits", top_k=10)

The important design decision: the vault format stays the same regardless of which backends you use. FileBlobStore and S3BlobStore store the same encrypted bytes. SQLiteSearchIndex and PostgresSearchIndex store the same metadata. Migrating between backends is O(N) byte-copies, no decryption required:

from vaultmem import migrate_vault

migrate_vault(
    src_blob_store=FileBlobStore("./old"),
    dst_blob_store=S3BlobStore(bucket="new-vault"),
    passphrase=passphrase,
)

HNSW and the O(N) problem

At small scale, exact cosine search over all atoms in RAM is fine. At 10,000 atoms it becomes the bottleneck — the benchmark showed ~148ms per query, all of it in the embedding comparison loop.

HNSWVectorIndex wraps hnswlib and plugs into the same VectorIndex ABC. The HNSW graph is built incrementally as atoms are added and serialized encrypted to vault/vector_index.hnsw.enc on flush. At session open it’s loaded once into RAM. Search then runs in O(log N):

Atoms	Exact cosine	HNSW
1,000	~15ms	~0.3ms
10,000	~148ms	~0.8ms
100,000	~1,500ms	~1.2ms

Below 1,000 atoms, the implementation falls back to exact cosine automatically — the HNSW graph has overhead that isn’t worth paying at small N. (For how the HNSW graph itself works, see the Paper TL;DR on HNSW.)

pip install "vaultmem[ann]"   # hnswlib

Query normalizer

A subtler problem surfaced while building the interactive demo: the question framing users naturally type — “What do I know about Sarah Chen?” — dilutes the embedding. A bag-of-words or hash-projection embedder assigns equal weight to “what”, “do”, “i”, “know”, “about” and “Sarah”, “Chen”. The meaningful terms are swamped by question scaffolding.

Sentence-transformers handle this natively through attention — they learn that “Sarah Chen” carries the semantic weight in the sentence. But not every deployment uses a 22M-parameter ML model.

v0.2.1 introduced a QueryNormalizer Protocol:

from vaultmem import QueryNormalizer  # Protocol — implement normalize(text) -> str

The built-in RegexQueryNormalizer strips common English question preamble:

from vaultmem import RegexQueryNormalizer

# "What do I know about Sarah Chen?" → "Sarah Chen"
# "How does Flash Attention work?"   → "Flash Attention work"
# "Am I vegetarian?"                 → "vegetarian"

More interesting is the custom path — since it’s a Protocol, any object with a normalize(self, text: str) -> str method works. The interactive demo uses a Groq-backed normalizer with a server-side API key, so users get LLM-quality query extraction without needing their own key:

class GroqQueryNormalizer:
    def __init__(self, api_key: str) -> None:
        from groq import Groq
        self._client = Groq(api_key=api_key)

    def normalize(self, text: str) -> str:
        resp = self._client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            max_tokens=20,
            messages=[
                {"role": "system",
                 "content": "Extract key search terms. Return only keywords."},
                {"role": "user", "content": text},
            ],
        )
        return resp.choices[0].message.content.strip() or text

The normalizer is injected once at session creation and runs on all searches where normalize_query=True:

with VaultSession.open("./vault", passphrase,
                        query_normalizer=GroqQueryNormalizer(api_key),
                        embedder=embedder) as s:
    results = s.search("What do I know about Sarah?", normalize_query=True)

Diagram: How VaultMem's query normalizer strips question framing before search (see it in the post)
The question "What do I know about Sarah Chen?" goes to a QueryNormalizer, any object with normalize(text) returning a string. Two kinds plug in: the built-in RegexQueryNormalizer, which strips common English question preamble, or your own class, such as the demo's Groq-backed normalizer that asks an LLM to return only keywords. Either way the result is "Sarah Chen"; the question scaffolding "What do I know about" is dropped. search with normalize_query=True then embeds only those key terms.
The normalizer runs before embedding, so the question's scaffolding words don't drown out the names that matter.

Data classes

One design detail worth calling out: memory doesn’t all decay at the same rate, and not all memory deserves the same cryptographic protection.

VaultMem has three data classes, set at vault creation:

Class	Use case	Argon2id memory	Decay λ	Half-life
`GENERAL`	Standard	64 MiB	0.005	~5 months
`MEDICAL`	Health data	128 MiB + mlock	0.002	~12 months
`ARCHIVAL`	Long-term records	64 MiB	0.0007	~3 years

MEDICAL doubles the Argon2id memory cost, making brute-force twice as expensive. ARCHIVAL slows the significance decay so memories stay surfaced for years rather than months — appropriate for journals, legal records, or anything you want to last.

VaultSession.create("./health_vault", passphrase, owner="alice", data_class="MEDICAL")

The LoCoMo benchmark

I wanted a more rigorous evaluation than the 18-atom synthetic benchmark I described in Part 1. The LoCoMo dataset is a collection of long-form conversational memory — 1,982 QA pairs across five categories (single-hop factual, multi-hop reasoning, temporal, adversarial, open-domain). It’s the closest thing to a realistic test of a personal memory system. (For how the benchmark was built, see the LoCoMo Paper TL;DR.)

VaultMem’s Recall@10 and MRR scores on a random sample:

Category	Recall@10	MRR
Single-hop factual	0.91	0.87
Temporal	0.84	0.79
Multi-hop	0.73	0.68
Adversarial	0.78	0.71

Multi-hop is the hardest — it requires reasoning across atoms that weren’t individually asked about, which is a retrieval problem before it’s even an LLM problem.

The interactive demo

The fastest way to see all of this in action is the live demo. It runs a pre-seeded vault with 36 memories across all four types — enough variety to demonstrate the retrieval model without you having to type anything first.

Every retrieved result shows its relevancy score, memory type, and retrieval tier. The right panel shows the raw encrypted bytes of current.vmem and current.atoms — what the platform operator actually sees. Each browser tab gets an isolated vault; your data can’t be seen by or contaminated by other visitors.

Some things worth trying:

Question	What it exercises
What do I know about Sarah Chen?	Groq query normalizer → EPISODIC retrieval
How does Flash Attention work?	SEMANTIC retrieval
What’s my coffee order?	PERSONA retrieval
How do I debug problems?	PROCEDURAL retrieval
Tell me about my cat	Natural language → EPISODIC

What’s next

The library is at v0.2.1. The things I’m thinking about for v0.3.0:

Composite and Affinity generation — right now the tiers exist in the retrieval model but atoms are only created at ATOM granularity. Automated COMPOSITE and AFFINITY synthesis from accumulated atoms is the next big addition.
Cross-agent memory sharing — Layer 2 of the Memory-as-Asset framework. A shared encrypted memory space where multiple agents can read without the platform being able to.
LoCoMo full benchmark — running the complete 1,982-pair dataset rather than a sample.

The paper targeting ACM CCS 2026 is in progress.

VaultMem Part 3 — Hiding PII from Your LLM Provider →

pip install vaultmem · GitHub · Preprint · Live demo

VaultMem — Building Encrypted Memory for AI Agents

2026-03-21T00:00:00+00:00

While building Rena (code) — a voice AI health agent I submitted for the Google Gemini Live Agent Challenge — I needed persistent memory. Something that could build a user profile from conversations across sessions and recall it the next time you speak to the agent. Libraries like mem0 or Zep would have worked fine for that use case.

But the more I thought about it, the more a different question started bothering me: what if I wanted a truly private agent — one where the platform running it genuinely cannot read my conversations? And beyond that, how do you store raw conversations as structured memory in the first place?

I couldn’t find anything that answered both. So I spent a few days building it: a memory library for AI agents where the platform developer cryptographically cannot read what the user stores.

Install: pip install vaultmem · Code: github.com/aag1091-alt/vaultmem-sdk · Preprint: doi.org/10.5281/zenodo.19154079

The problem

Every AI memory library I looked at — mem0, Zep, LangMem, Letta — stores user memories in plaintext on the platform’s servers. The platform operator can read everything their users have ever told the agent. This isn’t a privacy policy failure; it’s a structural one. When the key lives with the operator, the guarantee lives with the operator.

This matters more than it sounds. The deepest use case for persistent AI memory isn’t task management — it’s candor. An agent that remembers across sessions only becomes genuinely useful when you can be completely honest with it. A person working through a difficult decision. Someone with early-stage dementia building a longitudinal record of their life. A user who just wants their preferences remembered without those preferences living in a corporate database.

For that to work, the privacy guarantee has to be mathematical, not a privacy policy.

Where the research led

While digging into how existing systems handle memory, I stumbled on a paper: Memory as Asset by Pan, Huang & Yang (2026) (plain-language TL;DR). It formalizes something I’d been circling around informally — that personal memory isn’t just data to be stored; it’s an asset that belongs to the user, with ownership, permissions, and portability as first-class concerns.

The paper defines a layered architecture. Layer 1 is personal storage: encrypted, user-owned, portable. Layer 2 is cross-session intelligence. Layer 3 is collaborative memory between agents. No existing library implements even Layer 1 with real cryptographic guarantees.

What the paper gave me more than anything was vocabulary. Before reading it, I was thinking vaguely about “storing conversations.” After, I was thinking about episodic memory, semantic memory, significance decay, and granularity hierarchies. That shift changed what I built — instead of a simple key-value store with embeddings, VaultMem ended up with a proper memory model grounded in the framework. The paper became the specification.

What VaultMem is

VaultMem is an embeddable Python library. A platform developer adds it as a dependency, and their users get encrypted, portable, private memory — without the developer being able to read it.

The analogy I keep coming back to: VaultMem is to AI memory what 1Password is to credentials. The library doesn’t know what it’s storing. The platform receives opaque ciphertext it can store, back up, and serve back — but cannot decrypt.

from vaultmem import VaultSession, OllamaEmbedder

embedder = OllamaEmbedder("http://localhost:11434")

# Create an encrypted vault
with VaultSession.create("./vault", passphrase="s3cr3t", owner="alice",
                          embedder=embedder) as s:
    s.add("I met Bob at the AI conference yesterday")
    s.add("I prefer concise bullet-point answers")
    s.add("My go-to language for data work is Python")

# Search across sessions
with VaultSession.open("./vault", passphrase="s3cr3t",
                        embedder=embedder) as s:
    results = s.search("Bob conference")
    for r in results:
        print(f"[{r.tier}] {r.score:.3f}  {r.atom.content}")

Output:

[ATOM] 0.847  I met Bob at the AI conference yesterday

How the encryption works

The key architecture has two layers:

User passphrase
  → Argon2id(passphrase, salt)      ← 64 MiB, 3 iterations
  → KEK (Key Encryption Key)         ← in RAM only; zeroed after open
  → AES-GCM-Dec(KEK, wrapped_MEK)
  → MEK (Master Encryption Key)      ← in RAM during session only
  → AES-256-GCM(memory atom)        ← to disk

The Master Encryption Key (MEK) is generated once at vault creation and stored only as a ciphertext wrapped with the user’s passphrase-derived key. It never touches disk in plaintext. When the session closes, it’s zeroed from RAM.

Diagram: VaultMem's two-layer key architecture (see it in the post)
The user passphrase goes through Argon2id with a salt (64 MiB, 3 iterations) to derive the KEK, the key encryption key. The KEK lives in RAM only and decrypts the wrapped MEK with AES-GCM. The MEK, the master encryption key, stays in RAM during the session and is zeroed when it closes. The MEK encrypts each memory atom with AES-256-GCM, with a 128-bit authentication tag and the atom's UUID bound as additional authenticated data. On disk there is only the salt with the wrapped MEK, and the encrypted memory atoms.
Only ciphertext ever reaches disk. Because the MEK is wrapped separately from the data, a passphrase change re-wraps one key instead of re-encrypting every atom.

A platform developer who integrates VaultMem receives encrypted blobs from their users. They can store them, back them up, sync them — but they cannot read them. The math makes this true regardless of what their privacy policy says.

Two things I’m particularly happy with:

Passphrase changes are O(1). Because the MEK is wrapped separately from the data, changing a passphrase means re-wrapping one key — not re-encrypting every memory atom. A vault with 10,000 atoms changes passphrase in milliseconds.

Per-atom tamper detection. Each atom is encrypted with AES-256-GCM, which includes a 128-bit authentication tag. Each atom also has its UUID bound as Additional Authenticated Data, so you can’t transplant a valid ciphertext from one slot to another. Any tampering is detected before the data surfaces.

The memory model

The Memory-as-Asset paper defines memory as a 5-tuple: content, context, owner, permissions, versioning state. VaultMem’s MemoryObject implements this directly.

Four memory types: EPISODIC (things that happened), SEMANTIC (facts), PERSONA (preferences and habits), PROCEDURAL (how-to knowledge). Type is assigned by a deterministic classifier — no LLM call, no network request.

The more interesting part is the granularity hierarchy:

ATOM — a single memory statement. Immutable once written.
COMPOSITE — a type-homogeneous grouping of related atoms.
AFFINITY — a derived atom representing a recurring pattern. “User drinks coffee every morning” synthesized from twenty individual coffee mentions.

AFFINITY atoms carry a significance score:

σ = (1 − e^(−κ × freq)) × e^(−λ × days_since_last)

Frequency drives the first factor (with diminishing returns), recency drives the second. The parameters vary by data class — medical memories decay slower than general ones.

Three-tier retrieval — and why it matters

The retrieval model searches AFFINITY → COMPOSITE → ATOM and re-ranks results globally. On factual queries this behaves identically to flat cosine search — no regression. On pattern queries it’s a different story.

I ran a benchmark to verify this. Three topic clusters (coffee habits, Python usage, exercise routine), each with one AFFINITY atom and five ATOM-granularity memories including a deliberately tricky decoy: an episodic memory containing the exact query keywords, designed to fool flat cosine.

For the query “my morning coffee habits”, flat cosine ranks the decoy atom first (it contains the words “coffee habits” and “morning”) and the AFFINITY second. Three-tier flips this — the AFFINITY’s significance score (~0.92 for a well-established habit) outweighs the cosine advantage of the decoy.

Diagram: VaultMem's three-tier retrieval (see it in the post)
A query is searched against three tiers in order: AFFINITY atoms, which are derived recurring patterns that carry a significance score; COMPOSITE groupings of related atoms; and individual ATOM statements. All results are then re-ranked globally using cosine similarity plus significance. For the query "my morning coffee habits", flat cosine ranks a decoy atom first and the AFFINITY second, while three-tier retrieval ranks the AFFINITY first.
The decoy repeats the query's words, so flat cosine puts it first. The AFFINITY's significance score lifts the real habit to the top; specific queries rank the same as flat cosine.

Metric	Flat cosine	Three-tier	Δ
MRR (pattern queries)	0.50	1.00	+0.50
MRR (specific queries)	1.00	1.00	0.00

Pattern queries: AFFINITY surfaced first every time. Specific queries: no regression. The significance weight isn’t a heuristic — it’s a correction for a systematic bias in pure cosine retrieval.

The `.vmem` format

Vaults are stored as two files:

my_vault/
├── current.vmem      # encrypted index (48-byte self-describing header)
└── current.atoms     # append-only encrypted atom blocks

The 48-byte header is the only plaintext in the file. It carries the magic bytes (VMEM), format version, encryption algorithm, KDF, and file type — enough for any conforming implementation to decrypt the vault given the passphrase. No prior coordination with the writing implementation needed.

Atoms are append-only and immutable. Checkpoints update only the index file, atomically via fsync + os.rename(). Crash-safe.

The format is designed to be the interoperability layer for a future where encrypted memory vaults move between applications and agents — the same way a PDF can be opened by any conforming reader regardless of which app produced it.

What’s in the library

About 2,250 lines of Python, 39 unit tests, three core dependencies (cryptography, argon2-cffi, numpy). Sentence-transformers or a local Ollama instance for embeddings — or bring your own.

# LocalEmbedder — no network, runs on CPU
from vaultmem import LocalEmbedder
embedder = LocalEmbedder()

# OllamaEmbedder — GPU-accelerated, localhost or Tailscale
from vaultmem import OllamaEmbedder
embedder = OllamaEmbedder("http://localhost:11434", model="all-minilm")

# Custom — implement .embed(text) -> list[float]
class MyEmbedder:
    def embed(self, text: str) -> list[float]: ...
    @property
    def dimension(self) -> int: return 384

The research angle

The work turned into a paper. VaultMem is the first implementation of Layer 1 of the Memory-as-Asset framework — the paper I stumbled on during the initial research that ended up shaping the entire design. The paper formalizes the threat model, scores ten existing AI memory systems against a four-dimension privacy taxonomy (no prior system achieves all four properties simultaneously), and specifies the .vmem format as a portable standard.

The preprint is live on Zenodo and I’m targeting ACM CCS 2026 for the full paper.

Try it

pip install vaultmem                   # core
pip install "vaultmem[local]"          # + local embeddings (sentence-transformers)

The library is intentionally small — the code is the specification. If you’re building an AI agent and want to give your users memory that’s actually theirs, this is the starting point.

github.com/aag1091-alt/vaultmem-sdk

Part 2 covers what was built on top of this foundation in v0.2.0 and v0.2.1: multi-modal memory ingestion, temporal search, pluggable storage backends, HNSW vector indexing (how HNSW works), and a query normalizer Protocol — including a live interactive demo.

VaultMem Part 2 — Multi-Modal, Temporal Search, and Pluggable Backends →

New to terms like zero-knowledge storage, client-side encryption or episodic memory? The agent memory glossary explains them in plain language. For the non-technical case behind all of this, see Your AI Agent Shouldn’t Be Able to Read Your Diary.