![\"VaultMem:](\"/assets/images/vaultmem-explainer-720.webp)
\n Paper: Yu. A. Malkov & D. A. Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (arXiv, first posted March 2016; later published in IEEE Transactions on Pattern Analysis and Machine Intelligence, DOI 10.1109/TPAMI.2018.2889473)
\n\nOne-line TL;DR: Stack a few neighbor graphs on top of each other — sparse ones with long jumps at the top, a dense one with every item at the bottom — and you can find the closest vectors to a query by hopping down the stack instead of checking everything.
\n\nEvery memory or RAG system eventually asks the same thing: which stored embeddings are closest to this query vector? Checking every vector works, but the cost grows linearly with the collection. Exact shortcuts like k-d trees stop helping once vectors have hundreds of dimensions. So practical systems use approximate nearest neighbor (ANN) search: accept that you’ll occasionally miss a true neighbor, and measure how often you don’t with recall.
\n\nOne family of ANN methods builds a proximity graph — each item linked to its near neighbors — and searches by greedily walking toward the query. The authors’ own earlier method, Navigable Small World (NSW), did this but had two problems: search cost grew faster than logarithmically as the data grew, and on low-dimensional or tightly clustered data the greedy walk could get stuck in a “false” local minimum. HNSW is their fix.
\n\nThe paper describes the idea as a skip list where the linked lists are replaced by proximity graphs.
\n\nef) and return the best k. A bigger ef means higher recall and more work.M neighbors found by a wider search (efConstruction). If a neighbor ends up with too many links, its list is pruned back to a cap.The knobs the paper discusses are few: M (links per item; it suggests roughly 5–48, higher for high-recall or high-dimensional data), efConstruction (build quality) and ef (search effort). The graph links cost roughly 60–450 bytes per item on top of the vectors themselves.
HNSWVectorIndex that wraps the hnswlib library.ef trades speed for recall at query time, without rebuilding. When a memory system “forgets” something it stored, the index settings are one more place to look — alongside the embeddings and the re-ranking step.Earlier in the series: RAG (the retrieve-then-generate loop this index sits inside) and Sentence-BERT (the embeddings it searches over).
\n\nTerms like HNSW, embedding, semantic search, top-k retrieval and Recall@k are defined in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-retrieval-augmented-generation/", "url": "https://www.avinashgosavi.com/post/paper-tldr-retrieval-augmented-generation/", "title": "Paper TL;DR: RAG — The Paper That Named Retrieval-Augmented Generation", "summary": "A TL;DR of the RAG paper (Lewis et al., 2020): a language model paired with a searchable Wikipedia index, so swapping the index updates what it knows.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-retrieval-augmented-generation.png", "date_published": "2026-10-20T15:45:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel & Douwe Kiela (Facebook AI Research, University College London, New York University), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, 2020; revised 2021)
\n\nOne-line TL;DR: Instead of hoping a language model memorized the right fact, give it a search step: look up a few relevant Wikipedia passages, generate the answer from them, and train the search and the writer together.
\n\nBy 2020, large language models clearly stored a lot of facts in their weights. But the paper points out the downsides: that knowledge is hard to expand or revise, the model can’t show where an answer came from, and it can hallucinate. Earlier hybrid models that paired a language model with a retriever (REALM and ORQA) had only been tried on extractive question answering — pulling answer spans out of passages rather than writing new text. This paper asks whether one general recipe can combine the two: a model that retrieves documents and generates free-form answers, fine-tuned end to end for many different tasks.
\n\nThe paper splits knowledge into two kinds of memory:
\n\nAt question time, a second BERT encoder turns the input into a query vector, the index returns the top-k passages, and BART generates an answer conditioned on the input plus each passage. Nobody labels which passage is “right”: the retriever learns only from whether the final answer comes out well.
\n\nTwo variants decide how the passages get combined:
\n\nOne practical shortcut: only the query encoder and the generator are trained. The document encoder — and so the whole 21-million-vector index — stays frozen, because re-embedding Wikipedia during training is expensive. The authors report they didn’t need that step for strong results.
\n\nEarlier in the series: Sentence-BERT (the kind of embeddings that make the search step work), Lost in the Middle (where retrieved passages land in the prompt) and MemGPT (an agent that pages its own memory in and out). Next in the series: HNSW, the graph index behind fast nearest-neighbor lookups like this one.
\n\nTerms like retrieval-augmented generation, embedding, semantic search, top-k retrieval and HNSW are defined in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-lost-in-the-middle/", "url": "https://www.avinashgosavi.com/post/paper-tldr-lost-in-the-middle/", "title": "Paper TL;DR: Lost in the Middle — Where You Put Context Matters", "summary": "A TL;DR of Lost in the Middle (Liu et al., 2023): models use facts at the start and end of a long prompt far better than the middle, and what it means for RAG.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-lost-in-the-middle.png", "date_published": "2026-10-20T10:45:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni & Percy Liang (Stanford, UC Berkeley, Samaya AI), Lost in the Middle: How Language Models Use Long Contexts (arXiv, 2023)
\n\nOne-line TL;DR: Give a model a long prompt with one useful passage in it, and how well it answers depends on where that passage sits: best at the very start or very end, noticeably worse in the middle — even for models built for long contexts.
\n\nRetrieval-augmented generation and agent memory both work the same way at the last step: fetch some passages, paste them into the prompt, and hope the model reads them. Bigger context windows make it tempting to paste in more. This paper asks a simple question first: does the model actually use everything in a long prompt equally well?
\n\nTwo tasks, each designed so the authors could move the one relevant piece of information around while keeping everything else the same.
\n\nThey ran open models (MPT-30B-Instruct, LongChat-13B) and API models (GPT-3.5-Turbo, Claude-1.3, and their longer-context versions, plus GPT-4 on a subset).
\n\nEarlier in the series: MemGPT (paging what goes into the context window), Generative Agents (ranking memories by recency, importance and relevance) and Sentence-BERT (the embeddings behind most of the retrieval step). Next in the series: RAG, the 2020 paper that named the retrieve-then-generate setup this paper stress-tests.
\n\nTerms like context window, position bias, re-ranking and top-k retrieval are defined in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-generative-agents/", "url": "https://www.avinashgosavi.com/post/paper-tldr-generative-agents/", "title": "Paper TL;DR: Generative Agents — Memory Streams, Reflection and Plans", "summary": "A TL;DR of Generative Agents (Park et al., 2023): how 25 LLM characters in a sandbox town remember, reflect and plan, and which parts builders still copy.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-generative-agents.png", "date_published": "2026-10-19T21:40:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang & Michael S. Bernstein (Stanford, Google Research, Google DeepMind), Generative Agents: Interactive Simulacra of Human Behavior (UIST 2023)
\n\nOne-line TL;DR: Give an LLM a running log of everything it observes, a way to score which memories matter right now, and a habit of periodically writing down higher-level conclusions — and simulated characters start to behave believably over days, not just single replies.
\n\nThe authors built Smallville, a small sandbox town populated by 25 characters. Each one is driven by an LLM (gpt-3.5-turbo in the paper) and starts from a single paragraph describing who they are: job, relationships, habits. From there the characters wake up, go to work, chat, form opinions of each other and make plans — all generated, step by step, by the model.
\n\nThe interesting part isn’t the town. It’s the memory design that keeps each character consistent over time, which the paper splits into three pieces: a memory stream, reflection and planning.
\n\nEvery character keeps a memory stream: a plain list of memories, each written in natural language with a creation time and a “last accessed” time. The basic memories are observations — things the character perceived, such as (to give an illustrative example) “a neighbor is watering the garden.”
\n\nThe stream grows far too large to fit into a prompt, so each time a character needs to act, it retrieves only the most useful memories. Each memory gets three scores:
\n\nEach score is scaled to 0–1 and the three are added up (the paper weights them equally). The top-scoring memories that fit into the context window go into the prompt.
\n\nRaw observations are too low-level to generalize from. Fifty separate observations of a neighbor reading in the library don’t, by themselves, tell a character “she cares a lot about her research.”
\n\nSo characters reflect. When the summed importance of recent memories passes a threshold (150 in the paper — roughly two or three times per simulated day), the model looks at the 100 most recent memories, asks itself three high-level questions about them, retrieves memories relevant to each question, and writes down insights that cite the memories they came from.
\n\nThose insights go back into the same memory stream, where they can be retrieved — and reflected on again. The result is a reflection tree: raw observations at the leaves, more abstract conclusions higher up.
\n\nTo keep behavior coherent across a day, each character drafts a rough plan (about five to eight chunks), then breaks each chunk into hour-long actions, then into 5–15 minute steps. Plans are stored in the memory stream too. At every step, the model decides whether something it just noticed is worth reacting to; if so, it rewrites the plan from that point.
\n\nInterviews. The authors “interviewed” characters with questions about themselves, their memories, plans, reactions and reflections. 100 human evaluators ranked answers from the full architecture, versions with pieces removed, and human crowdworkers role-playing the same characters. The full architecture ranked most believable; removing reflection, then planning, then memory each made answers noticeably worse. The crowdworker answers landed below every version that still had memory.
\n\nEmergent behavior over two game days. One character was seeded with the idea of throwing a Valentine’s Day party. Without further scripting, word spread from 1 character to 13 of 25, and 5 of the 12 who were invited showed up. A mayoral candidacy spread from 1 character to 8 the same way.
\n\nEarlier in the series: MemGPT (an OS-style alternative where the model pages its own memory), Complementary Learning Systems (the brain’s version of turning episodes into general knowledge — close in spirit to reflection), and Sentence-BERT (the kind of embeddings behind the relevance score). Next in the series: Lost in the Middle, on what happens to retrieved text once it lands in a long prompt.
\n\nTerms like reflection, consolidation and decay are defined in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-memgpt/", "url": "https://www.avinashgosavi.com/post/paper-tldr-memgpt/", "title": "Paper TL;DR: MemGPT — Treating the Context Window Like RAM", "summary": "A TL;DR of MemGPT (Packer et al., 2023): how an LLM can page information in and out of a small context window with function calls, like an operating system.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-memgpt.png", "date_published": "2026-10-19T10:40:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica & Joseph E. Gonzalez (UC Berkeley), MemGPT: Towards LLMs as Operating Systems (arXiv, 2023)
\n\nOne-line TL;DR: Instead of waiting for bigger context windows, MemGPT treats the prompt like a computer’s RAM and lets the model itself move information between that small fast space and larger external storage, using function calls.
\n\nAn LLM can only “see” what fits in its context window. Long chats and big document collections don’t fit, so something has to decide what stays in view. Operating systems solved a version of this decades ago: programs act as if they have lots of memory, while the OS quietly pages data between fast RAM and slower disk. The paper calls its version virtual context management.
\n\nMemGPT splits memory into two places.
\n\nMain context — what’s actually in the prompt — has three parts:
\n\nExternal context lives outside the prompt:
\n\nThe clever part is who moves things around: the model does. Its output is parsed as function calls (search recall, write to archival, edit working context), and the results are fed back in. A few mechanics make this work:
\n\nrequest_heartbeat), so the model can chain several lookups before replying.Two settings, each with a “can it really remember?” task:
\n\nEarlier in the series: LoCoMo (a benchmark for exactly this kind of long-term memory), Sentence-BERT (the embeddings behind most semantic search), and Complementary Learning Systems (a brain-inspired split between fast and slow memory). Next in the series: Generative Agents, which ranks a long memory stream by recency, importance and relevance.
\n\nTerms like virtual context, context window, episodic memory and top-k retrieval are defined in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/agent-memory-glossary/", "url": "https://www.avinashgosavi.com/post/agent-memory-glossary/", "title": "An Agent Memory Glossary: Terms in Plain Language", "summary": "Plain-language definitions of the terms that come up when building memory for AI agents — episodic vs. semantic memory, embeddings, HNSW, re-ranking and more.", "image": "https://www.avinashgosavi.com/assets/og/agent-memory-glossary.png", "date_published": "2026-10-19T02:45:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Agent memory","Explainers"], "content_html": "The vocabulary around AI agent memory borrows from neuroscience, databases, search and cryptography all at once, which makes it easy to talk past each other. This is a working glossary — short, plain definitions, with links to where each idea shows up in my own projects. I’ll keep adding to it.
\n\nContext window (working memory). What the model can see right now: the prompt, the recent conversation and anything retrieved for this turn. Fast and precise, but limited in size and gone when the session ends. Everything else on this list exists because of that limit.
\n\nEpisodic memory. Specific events: what happened, when, with whom. “On Tuesday you said the demo slipped to Friday.” In VaultMem these are one of the memory types the agent can store and retrieve.
\n\nSemantic memory. General facts and knowledge, detached from when they were learned. “Flash Attention reduces memory traffic by tiling.” Often distilled from many episodes.
\n\nProcedural memory. How to do things: workflows, preferences about process, steps that worked before. “When debugging, check the logs before the config.”
\n\nPersona (profile) memory. Stable facts about the person the agent works with — name, role, standing preferences. Small, high-value and read on almost every turn.
\n\nVirtual context (paging). Treating the context window like a computer’s RAM: the agent moves information in and out of the prompt — saving facts to external storage, searching old messages, summarizing what gets pushed out — usually through function calls. Popularized by MemGPT.
\n\nEmbedding. A list of numbers that represents the meaning of a piece of text, so that similar meanings land close together. Most memory systems store an embedding next to each memory. See the Sentence-BERT TL;DR for where modern sentence embeddings come from.
\n\nSemantic search. Finding memories by meaning instead of exact words: embed the query, then return the stored memories whose embeddings are closest.
\n\nTop-k retrieval. Returning the k closest matches (say, the top 10) and putting them into the context window. Simple and effective; also where most retrieval mistakes start.
\n\nRetrieval-augmented generation (RAG). Search first, then write: fetch the passages most relevant to the input and have the model generate its answer from them, instead of relying only on what’s stored in its weights. The term comes from a 2020 paper that also showed you can update what the system knows by swapping the search index — see the RAG TL;DR.
\n\nVector index (HNSW). A data structure that makes “find the closest vectors” fast at scale. HNSW (Hierarchical Navigable Small World) builds layered graphs of neighbors so a search can hop quickly toward the right region instead of comparing against every vector. (Paper TL;DR)
\n\nRe-ranking. A second, slower and more careful pass over the top results — often a cross-encoder that reads the query and each candidate together — to put the truly relevant ones first.
\n\nPosition bias (lost in the middle). Models tend to use information at the start and end of a long prompt better than information buried in the middle, so where retrieved memories land in the prompt affects whether they get used. Putting the strongest matches first, and keeping top-k small, helps. See the Lost in the Middle TL;DR.
\n\nHybrid search. Combining meaning-based search with keyword search (or filters like dates and tags), because embeddings are bad at exact names, numbers and IDs.
\n\nContext frame. The situation a memory was formed in — topic, people involved, task, session — stored alongside it so retrieval can filter by context, not just content. This is the core idea in Attractor.
\n\nSufficient evidence. The idea that a memory system’s job isn’t just to return relevant memories but a set that is enough to answer. Ranking memories one at a time can miss this: several records may say the same thing (so the set gets bloated), or two weak-looking records may only make sense together (so both get dropped). One recent paper defines a sufficient set as a small subset that leads the agent to the same decision it would make with its full memory (Beyond Marginal Influence).
\n\nConsolidation. Turning many specific episodes into durable, general knowledge — summarizing a month of conversations into “prefers short answers,” for example. The brain’s version is covered in the Complementary Learning Systems TL;DR.
\n\nReflection. Having the agent periodically read its recent memories and write down higher-level conclusions (“she cares a lot about her research”), stored back as memories that point to the evidence they came from. Introduced for LLM agents in Generative Agents, which also ranks memories by recency, importance and relevance together.
\n\nDecay (forgetting). Letting memories that are never used fade in importance, so old noise doesn’t crowd out what matters. Forgetting is a feature.
\n\nEvent time vs. write time. Two different timestamps for one memory: when the thing happened, and when it was saved. A photo from 2019 imported today should come back for “what did I do in 2019?”, not for “what did I add today?”. VaultMem Part 2 keeps them as separate fields (captured_at and created_at) for exactly this reason.
Versioned facts (temporal state). Keeping every value a fact has had — “lives in Boston”, then “lives in Seattle” — as a list of versions linked to where each came from, instead of overwriting the old one. The newest version answers “where does she live?”, while the older ones still answer “where did she live before?”. The TEM paper adds one more step: older versions are marked as historical before the model sees them, and the authors report that this label alone gave large gains on questions about repeatedly updated facts.
\n\nAssociative (content-addressable) memory. Retrieving a whole memory from a partial or noisy piece of it, rather than by key or address. Hopfield networks are the classic example — see Hopfield (1982) and the modern version.
\n\nZero-knowledge storage. The service storing your data cannot read it, because it’s encrypted with a key only you hold. The operator sees ciphertext. This is VaultMem’s core design (why it matters, in plain language).
\n\nClient-side encryption. Encrypting data before it leaves your device (or your session), so the server only ever handles encrypted bytes. The usual way to get zero-knowledge storage.
\n\nPII sanitization. Detecting and masking personal details — names, emails, phone numbers — before text is sent to a model provider, then restoring them in the response. It closes a gap encryption at rest can’t: the model still has to read the text. See VaultMem Part 3.
\n\nMemory unlearning (forget requests). Deleting a specific fact when the user asks — and only that fact. Done well, the fact can no longer be recalled or pieced together from other memories, while everything unrelated survives. The Bemula benchmark tests exactly this and reports two common failures: the fact stays extractable with simple prompts, or the system deletes far more than it was asked to.
\n\nRecall@k. Of the questions asked, how often the right memory appears somewhere in the top k results. Measures “did we find it at all.”
\n\nMRR (mean reciprocal rank). Rewards putting the right memory near the top: 1 if it’s first, ½ if second, ⅓ if third, averaged over all questions.
\n\nLoCoMo. A benchmark of very long, multi-session conversations with questions across single-hop, multi-hop, temporal, open-domain and adversarial categories. See the LoCoMo TL;DR.
\n\nMissing a term, or think one of these is off? I’d like to hear it — I’ll keep this page up to date.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-hopfield-1982/", "url": "https://www.avinashgosavi.com/post/paper-tldr-hopfield-1982/", "title": "Paper TL;DR: Hopfield (1982) — Memory as a Valley You Roll Into", "summary": "A TL;DR of John Hopfield's 1982 paper on associative memory: energy landscapes, Hebbian storage, recall from partial cues and its limits — with a NumPy demo.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-hopfield-1982.png", "date_published": "2026-10-18T22:00:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, PNAS 79(8), 1982
\n\nOne-line TL;DR: connect lots of simple on/off neurons with symmetric weights and the network gets an energy that only goes down as it runs — so stored memories become stable valleys, and a partial or noisy cue rolls downhill into the full memory.
\n\nAlmost everything I’ve built around memory started here. It’s also one of the most readable classic papers I know: five pages, very little math, a physicist explaining computation.
\n\nHopfield’s move was to define an energy for the whole network’s state and show that, with symmetric weights and one-at-a-time updates, every update either lowers the energy or leaves it unchanged. The network can’t cycle forever; it has to settle.
\n\nStored patterns sit at the bottoms of valleys in that energy landscape. Start anywhere nearby — a corrupted photo, half a phone number — and the network slides down into the nearest stored memory. That’s content-addressable memory: you retrieve by what you remember, not by where it’s stored. It’s also robust: knock out some neurons or connections and recall still mostly works.
\n\nStore five random patterns in 100 neurons, flip 30% of the bits of one, and let it settle:
\n\nimport numpy as np\n\nrng = np.random.default_rng(1)\nN, P = 100, 5\npatterns = rng.choice([-1, 1], size=(P, N))\nW = (patterns.T @ patterns) / N # Hebbian storage\nnp.fill_diagonal(W, 0) # no self-connections\n\ndef energy(s):\n return -0.5 * s @ W @ s\n\ndef recall(s, sweeps=5):\n s = s.copy()\n for _ in range(sweeps):\n for i in rng.permutation(N): # one neuron at a time\n s[i] = 1 if W[i] @ s >= 0 else -1\n return s\n\ntarget = patterns[0]\nnoisy = target.copy()\nnoisy[rng.choice(N, 30, replace=False)] *= -1\n\nout = recall(noisy)\nprint((noisy == target).mean(), (out == target).mean()) # 0.7 -> 1.0\nprint(round(energy(noisy), 1), round(energy(out), 1)) # -8.6 -> -50.5\n70% correct goes to 100% correct, and the energy drops as it does.
\n\nThose limits are exactly what the 2020 follow-up fixes: modern Hopfield networks sharpen the valleys so capacity grows exponentially — and their update turns out to be transformer attention.
\n\nDatabases retrieve by key. Vector stores retrieve by nearest neighbor. Hopfield networks retrieve by settling — pattern completion from a partial cue, which is much closer to how remembering feels. That idea is the reason Attractor is called Attractor.
\n\nThat completes the six papers from Part 1. The whole list, with short TL;DRs, lives on my papershelf. The series keeps going with MemGPT, which treats an LLM’s context window like an operating system’s memory.
\n\nFor a one-paragraph definition of associative (content-addressable) memory, see the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-sentence-bert/", "url": "https://www.avinashgosavi.com/post/paper-tldr-sentence-bert/", "title": "Paper TL;DR: Sentence-BERT — The Quiet Workhorse Behind Semantic Search", "summary": "A TL;DR of Sentence-BERT (Reimers & Gurevych, 2019): why BERT is too slow for search, how siamese training fixes it, and why it still powers semantic search.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-sentence-bert.png", "date_published": "2026-10-18T21:00:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: Nils Reimers & Iryna Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)
\n\nOne-line TL;DR: fine-tune BERT so that every sentence gets its own fixed-size embedding you can compare with cosine similarity — turning semantic search over thousands of sentences from hours into seconds.
\n\nIt’s the least flashy paper in this series and the one my code depends on most. Every time Attractor, VaultMem or daylog “searches by meaning,” it’s using a model descended from this paper.
\n\nIn 2019, BERT was the best tool for deciding whether two sentences meant the same thing — but only if you fed it both sentences together. That works for scoring one pair. It falls apart for search.
\n\nThe paper’s example: to find the most similar pair in a collection of 10,000 sentences, BERT would need around 50 million pairwise passes — roughly 65 hours on a GPU. And you can’t precompute anything, because every score depends on both sentences at once.
\n\nThe obvious shortcut — run each sentence through BERT alone and average its outputs (or take the special [CLS] token) — produced embeddings that were often worse than old-fashioned averaged word vectors.
Sentence-BERT puts two copies of the same BERT side by side (a siamese network — shared weights), runs one sentence through each, pools each output into a single vector, and trains the whole thing so that:
\n\nThey trained on natural-language-inference data (pairs labeled entailment / contradiction / neutral) and on similarity-scored pairs. Afterward you throw away the “pair” setup: each sentence goes through the model once, you store its vector, and comparing two sentences is a cheap cosine similarity.
\n\nThat same 10,000-sentence search drops to about 5 seconds.
\n\nsentence-transformers package that came out of this work is still how most people get embedding models running in a few lines.Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems, LoCoMo and Memory as Asset. Next in the series: Hopfield (1982), the original associative memory that recalls from a partial cue.
\n\nEmbeddings, HNSW and re-ranking each get a short definition in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-memory-as-asset/", "url": "https://www.avinashgosavi.com/post/paper-tldr-memory-as-asset/", "title": "Paper TL;DR: Memory as Asset — Whose Memory Is It Anyway?", "summary": "A TL;DR of Memory as Asset (Pan, Huang & Yang, 2026): moving AI memory from agent-centric to human-centric, its three principles, and how it shaped VaultMem.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-memory-as-asset.png", "date_published": "2026-10-18T20:00:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory","Privacy"], "content_html": "Paper: Yanqi Pan, Qinghao Huang & Weihao Yang, Memory as Asset: From Agent-centric to Human-centric Memory Management (arXiv, March 2026)
\n\nOne-line TL;DR: the memory an AI keeps about you is designed today as the agent’s internal resource; this paper argues it should be your asset — owned by you, shared only under your permissions, and able to grow over time.
\n\nThis is the paper that reshaped VaultMem. I found it while I was stuck on a vague feeling that something about agent memory was backwards, and it gave that feeling a name.
\n\nMost memory systems treat what an agent remembers about you as part of the agent — stored by the platform, shaped for the platform’s use, and stuck there. The authors call this agent-centric memory. The consequences are familiar: you can’t take your history with you, you can’t decide who sees it, and every product rebuilds its own isolated “memory island” about the same person.
\n\nThey propose flipping the default, built on three principles:
\n\nEach memory object carries its content, context, an explicit owner, an access policy and a version. Ownership and permissions are part of the data itself, not an afterthought in some admin panel.
\n\nNotably, the paper doesn’t insist everything stays on your device. If a provider hosts your memory, it should be encrypted so only you can read it.
\n\nThat last point is where VaultMem lives. I treat it as an implementation of the first layer: the platform stores your memories, but they’re encrypted with a key only you hold, so the operator cryptographically can’t read them. Everything in the VaultMem series — client-side encryption, encrypted search, keeping PII away from the model provider — follows from taking “memory in hand” literally.
\n\nI still think it’s the right question to ask before building any memory feature: whose memory is this?
\n\nEarlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems and LoCoMo. Next in the series: Sentence-BERT, the sentence embeddings behind most semantic search.
\n\nZero-knowledge storage and client-side encryption are defined in the privacy section of the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-locomo-long-term-memory-benchmark/", "url": "https://www.avinashgosavi.com/post/paper-tldr-locomo-long-term-memory-benchmark/", "title": "Paper TL;DR: LoCoMo — How Good Is Your Agent's Memory, Really?", "summary": "A TL;DR of the LoCoMo benchmark (Maharana et al., 2024) for very long-term conversational memory — what it tests, what it found, and how I used it on VaultMem.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-locomo-long-term-memory-benchmark.png", "date_published": "2026-10-18T19:00:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents (ACL 2024)
\n\nOne-line TL;DR: LoCoMo is a benchmark of very long, multi-session conversations with questions that probe whether a system really remembers — and even strong long-context models and retrieval setups fall well short of people on it.
\n\nIf you’re building memory for an agent, this is the paper that keeps you honest.
\n\nMost chat benchmarks are a few turns long. LoCoMo conversations run for dozens of sessions and hundreds of turns, spread over simulated weeks and months. The authors generated them with LLM agents that each had a persona and a timeline of life events, then had people review and fix them so the conversations stay consistent. Some turns include shared images, too.
\n\nOn top of the conversations they built three tasks: answering questions, summarizing events, and generating responses. The question-answering part is the one I care about, and its categories are what make it useful:
\n\nEven with long context windows or retrieval-augmented generation, models trail human performance by a wide margin. Retrieval helps, but the hard categories stay hard — especially reasoning about time and resisting the urge to make up answers to adversarial questions. Bigger context alone doesn’t fix it.
\n\nWhen I built VaultMem, my first benchmark was 18 synthetic memories I wrote myself. It looked great — which was the problem. LoCoMo gave me a test I didn’t design. On a sample of its questions, retrieval was strong for single-hop facts and weakest for multi-hop, which needs pieces that weren’t individually asked about. That’s a retrieval problem before it’s a model problem, and it changed what I worked on next.
\n\nEarlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, and Complementary Learning Systems. Next in the series: Memory as Asset, on who should own an agent’s memory.
\n\nRecall@k and MRR, the usual ways to score retrieval, are explained in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-complementary-learning-systems/", "url": "https://www.avinashgosavi.com/post/paper-tldr-complementary-learning-systems/", "title": "Paper TL;DR: Complementary Learning Systems — Why Agents Need Two Kinds of Memory", "summary": "A TL;DR of McClelland, McNaughton & O'Reilly (1995): the brain's fast and slow learners, catastrophic interference, and what it suggests for AI agent memory.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-complementary-learning-systems.png", "date_published": "2026-10-18T18:00:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: McClelland, McNaughton & O’Reilly, Why there are complementary learning systems in the hippocampus and neocortex, Psychological Review (1995)
\n\nOne-line TL;DR: the brain learns with two systems on purpose — a fast one that grabs specific experiences and a slow one that gradually extracts general knowledge — because a single system can’t do both without wrecking what it already knows.
\n\nIt’s a 1995 psychology paper, and it’s one of the most useful things I’ve read about designing memory for AI agents.
\n\nThe paper starts from a puzzle that anyone who has fine-tuned a neural network will recognize. If you teach a network something new by adjusting the same shared weights that hold everything else, the new learning can overwrite the old. This is catastrophic interference: learn B, forget A.
\n\nThe fix that works in artificial networks is to learn slowly and interleave — mix new examples with old ones so the weights find a setting that serves both. But that’s slow. You can’t wait for hundreds of repetitions to remember where you parked this morning.
\n\nThe authors argue the brain splits the job:
\n\nThe bridge between them is replay. Over time — notably during sleep — the hippocampus reactivates recent episodes, and the neocortex learns from those replays interleaved with older knowledge. That’s consolidation: specific memories slowly become general knowledge without overwriting what came before.
\n\nThe paper backs this with evidence from people with hippocampal damage, who struggle to form new episodic memories yet keep older knowledge, and with network simulations showing why interleaved learning avoids interference.
\n\nRead with agents in mind, the paper is basically a design doc:
\n\nThis is the model I borrowed for Attractor: memories carry the context they were formed in, track how often they’re used, and are meant to strengthen or decay rather than sit frozen.
\n\nNext in the series: the benchmark that told me how good my own memory system actually was. Earlier parts: six papers on AI memory and Hopfield Networks Is All You Need.
\n\nQuick definitions of consolidation and decay are in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-hopfield-networks-is-all-you-need/", "url": "https://www.avinashgosavi.com/post/paper-tldr-hopfield-networks-is-all-you-need/", "title": "Paper TL;DR: Hopfield Networks Is All You Need — Attention Is a Memory Lookup", "summary": "A walkthrough of Ramsauer et al. (2020): how modern Hopfield networks store exponentially many memories and why their update rule is transformer attention.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-hopfield-networks-is-all-you-need.png", "date_published": "2026-10-18T17:00:00+00:00", "date_modified": "2026-10-23T01:35:48+00:00", "tags": ["Research","Agent memory"], "content_html": "Paper: Ramsauer, Schäfl, Lehner et al., Hopfield Networks is All You Need (2020)
\n\nOne-line TL;DR: a continuous version of the 1982 Hopfield network can store exponentially many memories, usually retrieves one in a single step — and that single step is exactly the attention operation inside every transformer.
\n\nThis is the paper that made me stop thinking of Hopfield networks as history. Here’s the idea in plain language, plus a tiny demo you can run.
\n\nIn Hopfield’s 1982 network, you store patterns (say, strings of +1 and −1) in the connection weights. The network has an “energy,” and each stored pattern sits at the bottom of a valley. Start from a corrupted pattern, let the network update, and it rolls downhill into the nearest stored memory. That’s content-addressable memory: you find a memory by giving a piece of it, not by knowing its address.
\n\nThe catch: the classic network only holds about 0.14 patterns per neuron. Store more than that and the valleys blur together into spurious memories.
\n\nRamsauer et al. swap the old quadratic energy for one built on log-sum-exp — a smooth version of “max.” Intuitively, each stored pattern now carves a much sharper, deeper valley, so many more of them fit without interfering. The paper shows the capacity grows exponentially with the dimension of the patterns, and that retrieval typically converges after one update.
\n\nThat one update is:
\n\nnew_state = X · softmax(β · Xᵀ · state)\nwhere the columns of X are the stored patterns and β controls how sharp the valleys are.
Now look at transformer attention:
\n\noutput = V · softmax(Kᵀ · q / √d)\nSet the query q to the current state, use the stored patterns as both keys K and values V, and pick β = 1/√d. It’s the same equation. Every attention head in an LLM is doing a one-step associative memory lookup: compare the query against everything stored, softmax the similarities, return a weighted blend that’s usually dominated by the best match.
The paper goes further, showing how different heads and layers behave like different retrieval regimes — some settle on one memory, some average over many — but the core equivalence is the part that stuck with me.
\n\nStore five random ±1 patterns, flip 40% of the bits in one of them, and do a single update:
\n\nimport numpy as np\n\nrng = np.random.default_rng(0)\nd, n = 64, 5\nX = rng.choice([-1.0, 1.0], size=(d, n)) # stored memories (columns)\n\ndef softmax(z):\n e = np.exp(z - z.max())\n return e / e.sum()\n\ndef retrieve(xi, beta=0.5):\n return X @ softmax(beta * X.T @ xi) # one modern-Hopfield update\n\ntarget = X[:, 2]\nnoisy = target.copy()\nnoisy[rng.choice(d, size=int(0.4 * d), replace=False)] *= -1\n\nout = retrieve(noisy)\nprint(\"overlap before:\", int(noisy @ target), \"/\", d) # 14 / 64\nprint(\"overlap after: \", int(np.sign(out) @ target), \"/\", d) # 64 / 64\nOne step takes a pattern that’s only loosely related to the target back to a perfect copy. The softmax weights tell the story: about 0.97 on the right memory and crumbs on the rest — exactly what an attention map looks like.
\n\nMost agent memory today is “embed everything, return the top-k nearest neighbors” (usually through an index like HNSW). This paper reframes that: nearest-neighbor search is a crude, non-differentiable cousin of what attention already does well. It nudged me to build Attractor around retrieval by partial cues and context, and to think of memory as something that settles rather than something you look up.
\n\nIf you read only one modern paper about associative memory, make it this one. Next in the series: Complementary Learning Systems, the brain’s split between a fast and a slow learner. The rest of what I’m reading is on my papershelf, and Part 1 of this series covers the other papers behind my memory projects.
\n\nTerms like top-k retrieval and associative memory are defined in plain language in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/paper-tldr-six-papers-on-memory/", "url": "https://www.avinashgosavi.com/post/paper-tldr-six-papers-on-memory/", "title": "Paper TL;DR: Six Papers That Shaped How I Think About AI Memory", "summary": "Short, plain-language summaries of six papers behind my AI memory side projects — from Hopfield's 1982 associative memory to user-owned memory for agents.", "image": "https://www.avinashgosavi.com/assets/og/paper-tldr-six-papers-on-memory.png", "date_published": "2026-10-18T16:00:00+00:00", "date_modified": "2026-10-22T11:35:39+00:00", "tags": ["Research","Agent memory"], "content_html": "Most of my side projects over the past year have circled one question: how should an AI system remember things? Not just “where do we put the vectors,” but how memory gets formed, found again, strengthened or forgotten — and who gets to read it.
\n\nI didn’t start with papers. I started building, got stuck, and went looking for how other people had thought about the problem. These six are the ones that actually changed what I built. Here’s each one in a few sentences, in plain language, plus what I took from it. If you only read one, read the first.
\n\nNeural networks and physical systems with emergent collective computational abilities · PNAS
\n\nTL;DR: Wire up simple on/off neurons with symmetric connections and the network gets an “energy” that can only go down as it updates. Stored memories sit at the bottoms of valleys in that landscape. Give it a partial or noisy pattern and it slides downhill into the full memory. You don’t look memories up by key — you fall into them.
\n\nWhat I took from it: this was the “oh” moment behind Attractor. Recall from partial cues feels much closer to how I actually remember things than a database lookup does.
\n\nHopfield Networks is All You Need · arXiv
\n\nTL;DR: A “modern” Hopfield network with continuous states can store exponentially many patterns and usually retrieves one in a single step. And that single update step is mathematically the same operation as transformer attention.
\n\nWhat I took from it: the 1982 idea isn’t a museum piece — it’s sitting inside every LLM. That made me take attractor-style retrieval seriously as a design for agent memory, not just a curiosity.
\n\nWhy there are complementary learning systems in the hippocampus and neocortex · Psychological Review
\n\nTL;DR: The brain uses a fast learner that grabs specific episodes (hippocampus) and a slow learner that gradually pulls out general structure (neocortex). Replaying episodes into the slow system lets it learn without overwriting what it already knows.
\n\nWhat I took from it: a vocabulary for something agents get wrong — treating every memory the same forever. It’s why Attractor tracks access and why I think memories should strengthen with use and fade without it.
\n\nSentence-BERT: Sentence Embeddings using Siamese BERT-Networks · arXiv
\n\nTL;DR: Plain BERT has to compare sentences two at a time, which is painfully slow for search. Training it in a “siamese” setup gives every sentence its own embedding you can compare with cosine similarity, so finding the closest match among thousands takes seconds instead of hours.
\n\nWhat I took from it: the unglamorous workhorse. The sentence-transformers models behind semantic search in Attractor, VaultMem and daylog all come from this line of work.
\n\nEvaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo) · arXiv
\n\nTL;DR: A benchmark of very long, multi-session conversations with questions that need single facts, multi-step reasoning, time awareness, or the sense to say “that never happened.” Even strong long-context models and RAG setups struggle with it.
\n\nWhat I took from it: my own little synthetic test was flattering me. Running VaultMem against LoCoMo gave me numbers I could actually trust — and showed me where retrieval was weak.
\n\nMemory as Asset: From Agent-centric to Human-centric Memory Management · arXiv
\n\nTL;DR: Argues that agent memory today is built for the agent rather than the person, and proposes treating the memory an AI keeps about you as an asset you own, share under your own permissions, and grow over time.
\n\nWhat I took from it: this one reshaped VaultMem entirely. It gave words to the feeling that an agent’s memory of your life shouldn’t be readable by whoever runs the agent. VaultMem ended up implementing its first layer.
\n\nReading these in a row, I keep landing on the same picture: good memory is found by meaning and context, changes over time, and belongs to the person it’s about. Most systems today nail the first, ignore the second, and get the third backwards.
\n\nAll six live on my papershelf with the rest of what I’m reading. If any of the vocabulary here is new, the agent memory glossary has short definitions. If there’s a paper on memory or agents you think I should read next, I’d genuinely love to hear about it.
\n\nNext in the series: Hopfield Networks Is All You Need, how modern Hopfield networks store exponentially many memories and why their update rule is transformer attention. Every part is listed on the Paper TL;DR page.
\n" }, { "id": "https://www.avinashgosavi.com/post/doing-the-right-thing-fast/", "url": "https://www.avinashgosavi.com/post/doing-the-right-thing-fast/", "title": "Doing the Right Thing, Fast", "summary": "Cleaning up after deprecating the first Catalyst Cloud Monitoring: splitting it into focused work streams and shipping it all in two weeks with AI help.", "image": "https://www.avinashgosavi.com/assets/og/doing-the-right-thing-fast.png", "date_published": "2026-04-25T00:00:00+00:00", "date_modified": "2026-10-22T00:35:18+00:00", "tags": ["Engineering at work","AI tools"], "content_html": "We recently deprecated the initial version of Cisco Catalyst Cloud Monitoring. That means the code that drove it — onboarding flows, device-specific paths, integrations that only existed for those devices — needed to go.
\n\nIn the old world, this kind of cleanup almost never happens on schedule. It’s not that people don’t want to do it. It’s that the cost is high: you’re touching interconnected parts, there’s real risk of breaking something unrelated, and the work has to fit around ongoing feature development. So it gets triaged below the line. Unused code paths don’t disappear — they just quietly accumulate. Every engineer knows the feeling of working around code that should have been deleted six months ago.
\n\nThis time was different.
\n\nI split the cleanup into 20-25 focused work streams. A few people helped with specific areas, but most of it was mine. Over two weeks, every stream went through the same cycle: identify what to remove, make the change, review it carefully, test, and move on.
\n\nThe Meraki developer tools team’s jira-to-PR skill made the pipeline from ticket to code review almost frictionless. I wasn’t fighting process — I was just doing the work.
\n\nThat work is now fully deployed to production. No errors. The legacy load is gone.
\n\nCodex handled the mechanical side of each stream — finding the right call sites, generating the deletions, updating the ripple effects across the codebase. I’ve been here since inception, so I know this code well. That matters. A new person would have needed more time to build the mental model before they could trust what was being removed. My 4.5 years gave me the confidence to move fast without second-guessing.
\n\nThe way I verify is consistent: I do all my visual inspection in VS Code, diff by diff. When I wasn’t sure what Codex had generated, I asked it to explain the change in plain terms. That became a reliable part of my review loop — not a crutch, just a check.
\n\nTakeaway: experience and AI compound. The codebase knowledge lets you set the right scope. Codex handles the execution. The combination is what made 2 weeks possible instead of 2 quarters.
\n\nWhat I used to put off, I now just do.
\n\nNot because the work got trivial — the cleanup was real work, done the right way, with proper reviews and manual testing. But the overhead that used to make it not worth starting has collapsed. The cost of doing the right thing is now low enough that “we’ll do it properly later” is no longer the default answer.
\n\nThat’s the shift. You can do the right thing and do it fast. Not by cutting corners, but because the slow parts — finding call sites, writing boilerplate, generating consistent changes across files — aren’t slow anymore.
\n\nI’m writing less code than I used to. But I’m seeing more of my intentions land exactly as I wanted them to. That’s a trade I’ll take every time.
\n\nI’m genuinely excited to see where this goes next.
\n\nRead about how I approach repeatable workflows in Codex: Codex Skills: Baking Your Workflow Into the Tool
\n\nRelated: Are These AI-Generated Tests Good? — a repeatable way to review the tests an LLM writes.
\n" }, { "id": "https://www.avinashgosavi.com/post/live-audio-transcription-whisper/", "url": "https://www.avinashgosavi.com/post/live-audio-transcription-whisper/", "title": "Live Audio Transcription on a $0 Server Using Whisper", "summary": "Near real-time audio transcription on my own server with faster-whisper — no third-party APIs — and the open-source service I extracted from daylog.", "image": "https://www.avinashgosavi.com/assets/og/live-audio-transcription-whisper.png", "date_published": "2026-04-12T00:00:00+00:00", "date_modified": "2026-10-23T07:36:47+00:00", "tags": ["Local-first","Tutorials"], "content_html": "While building daylog — a private voice diary app — I needed a way to transcribe audio from a phone in near real-time, running entirely on my own server. No OpenAI API, no Google, no third-party services. Just a microphone, a chunk of audio, and text back.
\n\nHere’s how I built it, what I learned, and the open-source service I extracted from it.
\n\nThe browser’s MediaRecorder API lets you record audio in chunks. You can set a timeslice — every N milliseconds, it fires an ondataavailable event with a blob of audio. Send that blob to a server, get back a transcript, show it to the user. Simple enough in theory.
The tricky parts:
\n\nWebM chunks are not standalone files. The first chunk from MediaRecorder contains the WebM file header. Every subsequent chunk is raw audio data with no header. If you send chunk #2 to ffmpeg on its own, it can’t decode it — it has no context for what codec, sample rate, or format to expect. I spent an embarrassing amount of time debugging silent transcriptions before I found this.
Whisper is slow on CPU. The original openai-whisper package uses PyTorch and takes 15–30 seconds to transcribe a 5-second chunk on a shared CPU VM. That’s not a live transcription experience — that’s watching paint dry.
First request is always slow. Loading a 75MB model from disk takes 10–20 seconds. If you don’t warm it up at startup, the first user request times out.
\nThe fix: don’t send individual chunks. Send the full accumulated audio on each event. Every transcription request becomes a valid, self-contained WebM file:
\n\nconst chunks = [];\n\nrecorder.ondataavailable = async (e) => {\n if (e.data.size < 1000) return;\n chunks.push(e.data);\n\n // Full accumulated audio = always a valid WebM file\n const snapshot = new Blob(chunks, { type: e.data.type });\n\n const form = new FormData();\n form.append(\"file\", snapshot, \"audio.webm\");\n const res = await fetch(\"/transcribe\", { method: \"POST\", body: form });\n const { transcript } = await res.json();\n setLiveTranscript(transcript); // replace, don't append\n};\nThe trade-off: each request sends more data than the last. For a 60-second recording with 5-second chunks, the last request sends all 60 seconds. Acceptable for a personal tool, worth optimising for production.
\n\nThe original openai-whisper package runs on PyTorch. faster-whisper uses CTranslate2 as its backend — it’s 2–4x faster on CPU and uses less memory. Combined with the tiny model and int8 quantization, it transcribes a 5-second chunk in roughly 2–3 seconds on a shared-cpu-1x Fly.io machine.
from faster_whisper import WhisperModel\n\nmodel = WhisperModel(\"tiny\", device=\"cpu\", compute_type=\"int8\")\nsegments, _ = model.transcribe(\"audio.webm\", beam_size=1)\ntext = \" \".join(seg.text.strip() for seg in segments).strip()\nbeam_size=1 switches from beam search to greedy decoding. For short conversational audio it makes no meaningful difference to accuracy, but it’s noticeably faster.
Bake the model into the Docker image. It adds ~75MB to the image size but eliminates the per-deploy download:
\n\nRUN python3 -c \"from faster_whisper import WhisperModel; WhisperModel('tiny', device='cpu', compute_type='int8')\"\nI extracted this into a standalone microservice: whisper-live-transcribe.
\n\nIt’s a single FastAPI endpoint:
\n\nPOST /transcribe\nContent-Type: multipart/form-data\n\nfile: <audio file>\n→ { \"transcript\": \"what you said\" }\nThat’s it. No auth, no database, no state. Drop it behind your existing API or deploy it standalone.
\n\nThe full main.py is under 60 lines:
@app.post(\"/transcribe\")\nasync def transcribe(file: UploadFile = File(...)):\n suffix = Path(file.filename or \"audio.webm\").suffix or \".webm\"\n\n with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:\n tmp.write(await file.read())\n tmp_path = tmp.name\n\n def _run():\n try:\n model = _get_model()\n segments, _ = model.transcribe(tmp_path, beam_size=1)\n return \" \".join(seg.text.strip() for seg in segments).strip()\n finally:\n Path(tmp_path).unlink(missing_ok=True)\n\n text = await asyncio.to_thread(_run)\n return {\"transcript\": text}\nasyncio.to_thread keeps Whisper’s CPU-bound work off the event loop so the server stays responsive to other requests while transcribing.
git clone https://github.com/aag1091-alt/whisper-live-transcribe\ncd whisper-live-transcribe\nfly launch --name my-transcribe-service\nfly deploy --local-only\nTotal cost: $0/month on Fly’s free tier (1 shared-cpu-1x VM). The 1GB RAM config handles the model (~150MB for tiny + runtime overhead) with room to spare.
\n\nMeasured on shared-cpu-1x with 1GB RAM:
| Model | \nSize | \nLatency per 5s chunk | \nNotes | \n
|---|---|---|---|
| tiny | \n75MB | \n~2–3s | \nSweet spot for real-time UX | \n
| base | \n145MB | \n~5–8s | \nBetter accuracy, noticeable lag | \n
| small | \n460MB | \n~15–20s | \nNeeds 2GB RAM | \n
For a live transcription experience tiny is the right call. The accuracy is good enough for clear speech — it occasionally stumbles on names or technical terms, but for conversational audio it’s reliable.
After shipping this for daylog, I replaced it with the Web Speech API — webkitSpeechRecognition on iOS Safari uses Apple’s on-device neural engine. Results are truly instant, word-by-word as you speak, and nothing leaves the device.
But the server-side approach is still the right choice when:
\nThe service is sitting open-source on GitHub if you want to use it. Swap in base or small for better accuracy, or point it at a GPU and large-v3 for near-perfect transcription.
whisper-live-transcribe on GitHub — PRs welcome.
\n\nWhere this fits in daylog: the hobby project stack covers the rest of the setup, and daylog’s security architecture covers how entries are encrypted once they’re transcribed. For live audio sent to a hosted model instead, Rena streams 16 kHz PCM over a WebSocket to a backend that passes it to Gemini Live.
\n" }, { "id": "https://www.avinashgosavi.com/post/hobby-project-stack/", "url": "https://www.avinashgosavi.com/post/hobby-project-stack/", "title": "What Should You Actually Use to Build Your Next Hobby Project?", "summary": "What to use for your next hobby project in 2025: the zero-cost stack I chose for daylog — Cloudflare Pages, Fly.io, Neon, Upstash — and why.", "image": "https://www.avinashgosavi.com/assets/og/hobby-project-stack.png", "date_published": "2026-04-12T00:00:00+00:00", "date_modified": "2026-10-20T05:36:08+00:00", "tags": ["Tutorials","Infrastructure"], "content_html": "I recently built daylog — a personal diary app where you speak or type your thoughts and they get encrypted and stored privately, with AI transcription happening server-side using Whisper. The interesting technical core is vaultmem, a Python SDK for building encrypted memory vaults for AI apps. Daylog is essentially a live demo of it.
\n\nBut the more interesting story — the one I keep getting asked about — is the stack. Because I built this with a hard constraint: it should cost me nothing to run, and I should be able to deploy it from my laptop to a URL I can test on my phone in under 10 minutes.
\n\nHere’s everything I learned about what’s actually available in 2025, what’s free, and what I’d pick if I were starting fresh today.
\n\nBefore comparing services, the constraints matter:
\n\nBefore diving into each choice, here’s how all the pieces connect:
\n\nThe key insight: the vault service is never exposed to the internet. Rails acts as the public face — it handles auth, verifies the JWT, and only then forwards requests to the vault service over Fly’s private network. The vault service doesn’t need its own auth layer, rate limiting, or TLS. That’s what makes this setup so lean.
\n\nChose: Cloudflare Pages\nAlso considered: Vercel, Netlify, GitHub Pages
\n\nCloudflare Pages is the easy winner for hobby frontends. Free tier includes:
\nBut the real reason I chose it over Vercel: Cloudflare’s ecosystem. If you ever need edge functions, object storage (R2), or KV, they’re right there in the same platform. Vercel’s free tier is also generous, but it optimises for Next.js specifically — if you’re not using Next.js (I’m using Vite + React), Cloudflare has less lock-in.
\n\nNetlify is fine but feels like it’s slowly losing the race. Their free tier has a 100GB bandwidth cap and slower build times.
\n\n\n\n\nMy pick for your next hobby frontend: Cloudflare Pages, unless you’re building a Next.js app — then Vercel.
\n
Chose: Fly.io\nAlso considered: Railway, Render, Heroku (lol)
\n\nFor a backend that needs to actually run a persistent server process — not a serverless function — Fly.io is currently the best deal available:
\n\n.internal DNS — no API keys needed between your servicesThe private networking was the killer feature for daylog. My Rails API talks to the vault service at http://daylog-vault-service.internal:8000. Zero latency, zero egress cost, zero authentication surface area exposed to the internet.
Railway is the other serious contender. Their developer plan gives you $5 of free credit per month, which is enough to run a small app. The UX is genuinely nicer than Fly’s. But Railway doesn’t give you persistent volumes on the free tier, and their private networking story is less mature.
\n\nRender has a free tier but web services sleep after 15 minutes of inactivity. If your app gets a request after sleeping, the first one takes 30+ seconds to respond. For a production app this is a dealbreaker. Their paid plans are reasonable.
\n\nHeroku effectively killed their free tier in 2022. Don’t bother.
\n\n\n\n\nMy pick: Fly.io if you need persistent state, private networking between services, or Docker deploys. Railway if you want a nicer UI and don’t need volumes.
\n
Chose: Neon\nAlso considered: Supabase, Railway Postgres, PlanetScale
\n\nNeon is serverless Postgres. It scales to zero when idle, so you’re not paying for compute when nothing’s happening. Free tier:
\nThe branch feature is genuinely underrated. You can spin up a copy of your database for a feature branch, test a migration, and delete it — all from the CLI.
\n\nSupabase is the obvious alternative and it’s excellent — you get Postgres, auth, real-time, and file storage in one package. Their free tier is a 500MB Postgres database. The catch: free tier projects pause after 7 days of inactivity. Fine if you’re actively building, annoying if you step away for two weeks.
\n\nPlanetScale had the best free tier in the business for a while — then they removed it entirely in 2024. Avoid for hobby projects unless you’re paying.
\n\nRailway Postgres is convenient if you’re already on Railway, but pricier than Neon for the same storage.
\n\n\n\n\nMy pick: Neon for pure SQL, Supabase if you also need auth/file storage out of the box.
\n
Chose: Upstash Redis\nAlso considered: Railway Redis, Render Redis
\n\nUpstash is serverless Redis. I use it for one thing: a distributed lock that prevents two vault service instances from opening the same encrypted vault simultaneously. Free tier:
\nFor a hobby app this is effectively infinite. I’ve never come close to the limit.
\n\nThe key property: it doesn’t cost you anything when you’re not using it. Railway and Render both charge you for Redis even when idle. Upstash charges zero for zero activity.
\n\n\n\n\nMy pick: Upstash Redis for anything ephemeral, rate-limiting, locking, or session state in a hobby project.
\n
This is where daylog gets opinionated.
\n\nThe obvious path for audio transcription is the OpenAI Whisper API — $0.006 per minute, accurate, fast. For a diary app that’s maybe 5 minutes of audio a day, that’s $0.03/day — practically free.
\n\nBut I’m building a private diary. Audio of your thoughts shouldn’t route through third-party APIs, even cheap ones. So I run Whisper locally on the vault service VM.
\n\nThe practical setup:
\nThe tradeoff: it’s slower than the API (3-5s per chunk vs instant), and tiny makes occasional errors. For a private diary that’s fine. For a customer-facing product, you’d probably use the API. (The streaming setup itself is written up in Live Audio Transcription on a $0 Server Using Whisper.)
What this teaches you: The “serverless AI API” path is usually the right default for hobby projects — OpenAI, Replicate, Together AI all have generous free tiers or pay-per-use pricing that rounds to zero at hobby scale. Only run models yourself if you have a specific reason (cost at scale, privacy, customisation).
\n\nThis is the part I want to highlight because it’s why daylog exists at all.
\n\nvaultmem is a Python SDK that gives you an encrypted, embeddable memory vault for AI applications. You give it a passphrase and it gives you a session where you can store text atoms, media blobs, and run semantic search — all encrypted at rest.
\n\nFor daylog, this means:
\n.vmem file on a Fly volume — no database schema, no migrationsThe SDK handles all the hard parts: PBKDF2 key derivation, AES-256 encryption, sentence embeddings for search, flush/commit semantics. You write:
\n\nsession = vault.open(user_id, passphrase)\nsession.add(\"Had a great meeting with the team today\")\nsession.flush()\nAnd you get encrypted storage with semantic search for free. (How the encryption and the encrypted search index work is in the daylog security post; the library itself is explained in VaultMem Part 1.)
\n\nThis is what makes the whole stack work at hobby scale — instead of a Postgres schema for diary entries + a vector database for search + an encryption layer + blob storage for audio, you have one SDK and one file.
\n\nHere’s the actual bill:
\n\n| Service | \nWhat I use it for | \nMonthly cost | \n
|---|---|---|
| Cloudflare Pages | \nWeb app hosting | \n$0 | \n
| Fly.io | \nRails API + vault service | \n$0 (2 VMs, free tier) | \n
| Neon | \nPostgres (users, sessions) | \n$0 (free tier) | \n
| Upstash | \nRedis (vault locks) | \n$0 (free tier) | \n
| Total | \n\n | $0 | \n
The only way this crosses zero is if I add a persistent volume larger than 3GB (Fly charges $0.15/GB/month) or if my Neon database exceeds 512MB.
\n\nFor context: the same app on managed infrastructure — RDS, ElastiCache, EC2, CloudFront — would run $50-100/month minimum.
\n\nStatic site / frontend only:\nCloudflare Pages. Nothing else comes close on the free tier.
\n\nFull-stack app, no persistent state:\nCloudflare Pages (frontend) + Cloudflare Workers (API). Everything runs at the edge, free tier is very generous.
\n\nFull-stack app with a database:\nCloudflare Pages + Fly.io + Neon. This is the setup I’d start with today. All three have genuine free tiers with no sleeping, no credit card traps.
\n\nNeed auth out of the box:\nAdd Supabase for the database and use their auth — or keep Neon and use Clerk (generous free tier for up to 10k MAU).
\n\nNeed background jobs or scheduled tasks:\nFly.io handles this natively — just run a separate machine. Or use Upstash QStash ($0 for low volume).
\n\nAI features:\nStart with the API (OpenAI, Anthropic, Together, Replicate). Only run models yourself when you have a specific reason. CPU inference on Fly is possible (I’m doing it) but it’s slow — GPU machines are still expensive even at hobby scale.
\n\nEncrypted, private AI memory:\nvaultmem. That’s the pitch.
\n\nI’d set up a simple CI/CD pipeline earlier. Deploying from the terminal (fly deploy --local-only, wrangler pages deploy) is fine for the first week, but it gets old fast. GitHub Actions + Fly + Cloudflare deploy actions take maybe 30 minutes to set up and remove a lot of friction.
The broader point: the free tier ecosystem for hobby projects has genuinely never been better. In 2025 you can run a full-stack app with a real database, Redis, background jobs, and AI inference for exactly zero dollars a month — and none of these services are traps that will surprise you with a bill when a tweet goes viral.
\n\nThe days of “I can’t afford to run this side project” are over. The only cost now is your time.
\n\nI’m putting the finishing touches on a proper demo — the full flow of opening the app, speaking a thought, watching it transcribe in real time, and searching back through weeks of entries with a natural language query. All encrypted, all on-device passphrase, running on the stack described above.
\n\nIf you want to see what a privacy-first, AI-powered personal knowledge tool looks like when it costs nothing to run — keep an eye out. Demo coming soon.
\n\nvaultmem is on PyPI. daylog is coming.
\n\n\n\n" }, { "id": "https://www.avinashgosavi.com/post/daylog-security-architecture/", "url": "https://www.avinashgosavi.com/post/daylog-security-architecture/", "title": "How daylog Keeps Your Diary Private (Even from Itself)", "summary": "How daylog, a voice diary, keeps entries and its semantic search index encrypted, with nothing leaving the device to answer a query.", "image": "https://www.avinashgosavi.com/assets/og/daylog-security-architecture.png", "date_published": "2026-04-12T00:00:00+00:00", "date_modified": "2026-10-22T01:36:04+00:00", "tags": ["Privacy","Local-first"], "content_html": "daylog is a voice diary. You open it, tap the mic, say what’s on your mind, and it saves. You can search your entries semantically — type “frustrated” and it surfaces the day you wrote “couldn’t make progress, everything felt blocked” even though that exact word never appears.
\n\nThat search working at all, while keeping your diary private, requires a few things to be true simultaneously: the content must be encrypted at rest, the search index must also be encrypted, and nothing should leave the device to answer a query. This post explains how each of those holds.
\n\nThe diagram traces a single entry from your mouth to disk, and back to your eyes.
\n\n\n\nYour words are readable on your device and readable back in your browser. Everything in between — transit, storage — is ciphertext. Notice the saving path says “text + vector encrypted”: the search index is encrypted too, not just the content. More on that below.
\n\nThe vault service uses vaultmem — a library I built for giving AI agents a private, searchable long-term memory. In daylog it does three things every time you save an entry:
\n\n1. Classifies the entry automatically.
\n\nEvery entry is run through a fast, rule-based classifier (no LLM involved) that assigns a memory type:
\n\nMost diary entries classify as EPISODIC. The type determines how the memory is weighted and how it ages — episodic memories have a natural recency decay, semantic ones don’t.
\n\n2. Embeds the entry as a 384-dimensional vector.
\n\nBefore the entry is encrypted, it is run through sentence-transformers/all-MiniLM-L6-v2 — a 22M parameter model running locally inside the vault container. The model converts your text into a 384-float vector that captures the semantic meaning of what you wrote. (sentence-transformers is the library that came out of the Sentence-BERT paper.)
“went for a run today” → [0.312, -0.087, 0.441, ..., 0.198] (384 values)
This vector is how search finds relevant entries later. It is stored alongside the text.
\n\n3. Encrypts text + vector together and writes to disk.
\n\nThe entry object — text, classification, timestamp, and embedding vector — is serialised to JSON, compressed with zlib, and encrypted with AES-256-GCM using the master encryption key (MEK). The result is appended to a .vmem file on disk as IV + length + ciphertext + GCM tag.
The embedding is encrypted. The search index — the thing that makes semantic search work — is not sitting in a plaintext index somewhere. It is inside the same ciphertext block as your diary entry. Nobody can search your vault without your key.
\n\nWhen you type “frustrated” into the search bar:
\n\n1. The vault service embeds your query.
\n\n“frustrated” → [0.289, -0.041, 0.509, ...] (the same 384-dim model, same vector space as your stored entries).
2. It decrypts all stored embeddings from disk.
\n\nEvery .vmem entry is decrypted in RAM. The vault service now holds a matrix of shape (N entries) × 384.
3. It runs a batched dot product.
\n\nSince both vectors are unit-normalised, dot product = cosine similarity. One matrix multiply gives a relevance score for every entry simultaneously.
\n\n4. It filters, ranks, and returns.
\n\nEntries with score below 0.3 are dropped. The rest are sorted and the top 8 returned — with their decrypted text.
\n\nThe decrypted entries exist only in RAM during this step. They are never written back to disk.
\n\nWhy this finds meaning, not just keywords.
\n\nTwo sentences with similar meaning will have similar vectors even if they share no words. “couldn’t make any progress today, everything felt blocked” scores ~0.62 against “frustrated” because the model has learned that these concepts occupy the same region of the vector space.
\n\nThis is the same reason “angry” might surface an entry about a deploy failing at midnight, or “energised” might surface one about a flow state — the words you actually wrote don’t have to match the words you search for.
\n\nvaultmem stores entries at three granularity levels:
\n\nSearch queries all three tiers in priority order — AFFINITY first, then COMPOSITE, then ATOM — so a standing pattern surfaces above any individual entry that mentions the same thing.
\n\nWhen you tap Summarise this day, the browser’s already-decrypted entries for that day are sent to Groq (Llama 3.3 70B) via the Rails API. What Groq receives:
\n\n09:10 AM: Went for a run today.\n08:30 PM: Good session, finally fixed that bug.\nTime + text for one day. No name, no metadata, nothing from other days. The summary is returned and displayed — it’s never saved back into the vault.
\n\nEverything else — adding entries, browsing, searching — is vault-only. No external API. Summarise is the one feature that trades privacy for convenience, and it’s a button you tap deliberately.
\n\nIf that trade-off isn’t right for you, remove GROK_API_KEY from the Rails API config and everything else works unchanged.
| Where | \nWhat | \nStatus | \n
|---|---|---|
Fly.io volume (/data/vaults/) | \n .vmem — text + embedding vectors, all encrypted | \n AES-256-GCM | \n
| Postgres (Neon) | \nemail, bcrypt password hash | \nPasswords hashed; DB encrypted at rest | \n
| Browser sessionStorage | \nsession_id, derived passphrase | \nCleared on tab close | \n
| Browser localStorage | \nauth JWT | \nCleared on sign out | \n
| Vault service RAM | \nMEK, decrypted atoms during active session | \nNever written to disk | \n
The encrypted search index is the part most people don’t expect. Normally if you want full-text or semantic search, you need a plaintext index somewhere — Elasticsearch, pgvector, whatever. vaultmem stores the vectors encrypted alongside the content, so you get semantic search without giving the index to anyone.
\n\nThe app is live. You can log in with the demo account and poke around — all entries are pre-seeded across the last two months.
\n\n\n\nemail: demo@daylog.app\npassword: demo1234\nThings worth trying:
\n\nfrustrated, flow state, couldn't sleep, or good run — these find entries by meaning, not exact wordsThe rest of the stack (hosting, transcription, costs) is in my hobby project stack, and the open-source Whisper transcription service I extracted from daylog has its own post. Episodic, semantic, procedural and persona memory, embeddings, semantic search and decay each get a short definition in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/attractor-memory-engine/", "url": "https://www.avinashgosavi.com/post/attractor-memory-engine/", "title": "Attractor: Building a Memory Engine Inspired by How the Brain Works", "summary": "Attractor, a memory engine inspired by how brains remember — Hopfield-style retrieval by partial cues and context instead of keys or nearest neighbors.", "image": "https://www.avinashgosavi.com/assets/og/attractor-memory-engine.png", "date_published": "2026-04-05T00:00:00+00:00", "date_modified": "2026-10-21T17:36:35+00:00", "tags": ["Agent memory","Research"], "content_html": "I’ve been thinking about memory — not human memory, but how machines store and retrieve it. Most systems today are either key-value stores (fast, exact, dumb) or vector databases (semantic, but still just nearest-neighbour). Neither of them works the way a brain does.
\n\nSo I started building something different. I called it Attractor.
\n\nWhen you search a database, you give it a key and it returns a value. When you search a vector database, you give it a query and it returns the closest vectors by distance (usually via an approximate index like HNSW). Both are useful. But neither captures something important about how biological memory works — context.
\n\nThink about how you remember things. You don’t retrieve memories by key. You retrieve them by partial cues — a smell, a phrase, a situation. And the retrieval is shaped by context: what you were doing, who you were with, what problem you were trying to solve. Two memories with identical content mean different things depending on when and why they were formed.
\n\nMachines don’t do this. And for AI agents that need to reason across long conversations or accumulated knowledge, this is a real limitation.
\n\nIn 1982, John Hopfield described a type of recurrent neural network that stores memories as energy minima [1]. The idea is elegant: each memory is a stable state — a basin of attraction — in an energy landscape. When you give the network a partial or noisy input, it doesn’t search for the closest match. It converges — like a ball rolling downhill — to the nearest stable state.
\n\nThis is called content-addressable memory. You don’t need the exact key. You need something close enough to fall into the right basin.
\n\nThe name Attractor comes directly from this. In dynamical systems, an attractor is a state that a system converges to over time. Hopfield memories are attractors. Retrieval is the process of finding which attractor your input belongs to.
\n\nWhat makes this intriguing is that it mirrors something real about biological memory. Your brain isn’t doing a database lookup when you remember something. It’s doing something closer to energy minimisation — pattern completion from partial cues.
\n\nWhat’s also interesting: in 2020, researchers showed that modern Hopfield networks with continuous states have exponentially higher storage capacity than the original binary version [2] — and that the attention mechanism in transformers is mathematically equivalent to a single Hopfield update step. The architecture powering every LLM today is, at its core, a Hopfield retrieval operation.
\n\nAttractor is a standalone memory service. Right now it’s Phase 1: a working foundation that stores memories with context and retrieves them by meaning.
\n\nThe data model is built around something I call a ContextFrame — the situational state at the time a memory was formed:
class ContextFrame(BaseModel):\n topic: str | None # what was being discussed\n entities: list[str] # who or what was involved\n task: str | None # what was being done\n session_id: str | None # which session it came from\nThis isn’t metadata. It’s a retrieval key. When you search, you can filter by context — “find memories from this topic” or “find memories from this session” — and combine that with semantic search over the content itself.
\n\nThe storage layer uses PostgreSQL with pgvector [3]. Each memory stores its content, context frame, tags, timestamps, access count, and a 384-dimensional embedding from a sentence-transformers model [4]. The embedding enables semantic search — retrieving by meaning rather than exact words.
\n\nThe API is simple:
\n\nPOST /memories — store a memory with context\nGET /memories/:id — retrieve by ID\nPOST /memories/search — search by content + context filters\nDELETE /memories/:id — remove a memory\nThe CLI wraps the service:
\n\nattractor start # starts the service on port 7747\nattractor connect # verifies it's running\nPort 7747 is “loci” on a phone keypad — a nod to the method of loci, the memory palace technique [5], which was the project’s original name before I found a better one.
\n\nHere’s what makes it different from a plain vector search. Say you store these two memories:
\n\n\"auth tokens should never be stored in localStorage\"\n context: { topic: \"security review\", task: \"reviewing PR #42\" }\n\n\"switched from JWT to session tokens — stateless was causing issues\"\n context: { topic: \"auth refactor\", task: \"architecture decision\" }\nNow you search: \"why did we change the auth approach\" with context_filter: { topic: \"auth refactor\" }.
The second memory surfaces — not because the words match, but because:
\nWithout the context filter, both memories are relevant. With it, you get the one that was formed in the right situation. That’s the difference.
\n\nPhase 1 is the foundation. The more interesting work is ahead:
\n\nPhase 2 — Association graph: Memories aren’t just points in a vector space. They’re nodes in a graph with typed edges — supports, contradicts, caused_by, followed_by. Retrieval triggers spreading activation: surface the memory you asked for, and the ones connected to it.
Phase 3 — Decay and consolidation: Not all memories are equally important. The access_count field is already tracking how often each memory is retrieved. Phase 3 uses this to implement decay — memories that are never accessed fade, memories that are frequently retrieved strengthen. This mirrors how biological memory consolidation works [6].
Phase 4 — Hopfield retrieval: Replace vector similarity with a proper Hopfield weight matrix. Retrieval becomes pattern completion — give the network a partial cue and it converges to the full memory. This is the step that moves from “semantic search with context” to something genuinely brain-inspired.
\n\nHonestly — yes, in parts. Vector search with metadata filtering exists in Pinecone, Weaviate, and Qdrant. Memory layers for agents exist in LangChain and LlamaIndex. And the “Hopfield Networks is All You Need” paper already showed that transformer attention is mathematically a Hopfield update — so in a sense, every LLM is already doing content-addressable retrieval implicitly.
\n\nPhase 1 of Attractor is not novel. It’s a clean, focused implementation of something that exists in various forms.
\n\nThe bet is on Phases 3 and 4. Most agent memory systems don’t implement decay — memories don’t weaken over time unless reinforced, they don’t consolidate the way biological memory does, and retrieval is always ranking rather than convergence. That’s the gap Attractor is trying to close.
\n\nWhether that gap matters in practice is an open question — and the honest answer is that it depends entirely on whether the difference shows up in real use. That’s why I’m building memlog on top of Attractor as a daily-use testbed: log thoughts, decisions, and notes every day, and see whether context-aware retrieval with decay actually surfaces better results than a plain vector search would. If it does, the model is right. If it doesn’t, the model needs revision.
\n\nTheory is cheap. The feedback loop is the thing.
\n\nMost AI agents today are stateless between sessions or rely on crude context windows. The ones that do have memory bolt it on as an afterthought — store the last N messages, embed them, retrieve by cosine similarity.
\n\nAttractor is an attempt to do it properly. Not because the engineering is especially hard, but because the model of memory matters. If you treat memory as a lookup table, you get lookup-table behaviour. If you treat it as an energy landscape with context and decay, you get something closer to how intelligence actually works.
\n\nPhase 1 is live. pip install attractor-engine — start storing memories with context, retrieve by meaning.
The interesting phases are coming.
\n\nHopfield, J.J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558. Plain-language summary: Paper TL;DR.
\nRamsauer, H., Schäfl, B., Lehner, J., et al. (2020). Hopfield Networks is All You Need. arXiv:2008.02217. (Shows modern Hopfield networks have exponential capacity and that transformer attention is a Hopfield update.) Plain-language summary: Paper TL;DR.
\npgvector. (2021). Open-source vector similarity search for Postgres. GitHub.
\nReimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084. Plain-language summary: Paper TL;DR.
\nYates, F.A. (1966). The Art of Memory. University of Chicago Press. (The definitive account of the method of loci — the memory palace technique.)
\nMcClelland, J.L., McNaughton, B.L., & O’Reilly, R.C. (1995). Why there are complementary learning systems in the hippocampus and neocortex. Psychological Review, 102(3), 419–457. (The foundational paper on memory consolidation and why some memories strengthen while others decay.) Plain-language summary: Paper TL;DR.
\nSource: github.com/aag1091-alt/attractor
\n\nContext frames, consolidation, decay and associative memory are each defined in a sentence or two in the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/vaultmem-part-3-pii-sanitizer/", "url": "https://www.avinashgosavi.com/post/vaultmem-part-3-pii-sanitizer/", "title": "VaultMem Part 3 — Hiding PII from Your LLM Provider", "summary": "VaultMem v0.3: a PII sanitizer that keeps personal details away from the LLM provider — the gap that encryption at rest doesn't close.", "image": "https://www.avinashgosavi.com/assets/og/vaultmem-part-3-pii-sanitizer.png", "date_published": "2026-04-01T00:00:00+00:00", "date_modified": "2026-10-22T17:36:32+00:00", "tags": ["Agent memory","Privacy"], "content_html": "Part 1 covered the core encryption model. Part 2 covered multi-modal ingestion and pluggable backends. This post covers v0.3.x — a new layer of privacy that addresses a problem encryption at rest cannot solve.
\n\n\n\n\nInstall:
\npip install 'vaultmem[spacy]'· Code: github.com/aag1091-alt/vaultmem-sdk · Live demo: vaultmem-demo.streamlit.app
VaultMem stores every memory atom encrypted with AES-256-GCM. Your passphrase never leaves your device. The platform operator cryptographically cannot read your vault. The threat model comes from the Memory-as-Asset framework (Pan, Huang & Yang, arXiv:2603.14212; plain-language TL;DR): memory is a personal asset, not a platform resource.
\n\nBut there is a second exposure point I hadn’t fully addressed: the moment you decrypt a memory and inject it into an LLM prompt.
\n\nSay your vault contains:
\n\nI met Sarah Chen at the NeurIPS conference. She works on ML infra at Google.\nMy accountant James Park can be reached at james@parkaccounting.com.\nWhen you ask your assistant “what do I know about Sarah?”, VaultMem retrieves those memories, decrypts them, and sends them as context to the LLM provider — Groq, OpenAI, Anthropic, whoever you’re using. At that point, Sarah’s name, her employer, and James’s email address are all sitting in plaintext in an HTTP request to a third-party cloud.
\n\nEncryption at rest doesn’t help here. The data is decrypted before it goes out the door.
\n\nVaultMem v0.3.0 ships a Sanitizer class that sits between your vault and your LLM call.
from vaultmem import VaultSession, Sanitizer\n\nsan = Sanitizer(backend=\"spacy\") # one instance per session\n\nwith VaultSession.open(\"./my_vault\", passphrase=\"...\") as s:\n memories = s.search(\"sarah chen\", top_k=5)\n context = \"\\n\".join(f\"- {m.atom.content}\" for m in memories)\n\n # Sanitize context AND query with the same instance\n sanitized_context, rmap = san.sanitize(context)\n sanitized_query, rmap = san.sanitize(user_query)\n\n raw_reply = llm.chat(sanitized_query, context=sanitized_context)\n\n # Restore real values in the LLM's response\n final_reply = san.restore(raw_reply, rmap)\nThe LLM provider receives:
\n\nI met Jordan at the NeurIPS conference. She works on ML infra at Globex.\nMy accountant Casey can be reached at [EMAIL_1].\nIt reasons about this correctly, responds naturally, and VaultMem restores the real values client-side before the answer is shown to the user.
\n\nThe provider never sees the real names, companies, or contact details.
\n\nPerson names are replaced with natural-sounding pseudonyms — Jordan, Casey, Morgan, Riley, Avery — rather than opaque tokens like [PERSON_1].
The reason is fluency. If the LLM sees [PERSON_1] in context and addresses the user with it, the response reads awkwardly. With a pseudonym, it can write “Based on what Jordan told you at NeurIPS…” and after restoration the user reads “Based on what Sarah Chen told you at NeurIPS…” — completely natural.
Organisations get fictional company names (Acme, Globex, Initech, Pied Piper). Structured PII — emails, phone numbers, SSNs, credit cards, IPs — gets typed tokens ([EMAIL_1], [PHONE_2]) because there’s no natural-sounding substitute for those.
The mapping is session-scoped and stable: if “Sarah Chen” appears in five different memories, she is always Jordan in that session. The LLM can reason about her as a single consistent person across all memories.
\n\nPresidio supports different NLP backends. VaultMem exposes both:
\n\n# Lightweight — ~12 MB spaCy model, no PyTorch dependency\n# Works on Streamlit Cloud, CI, anywhere torch is unavailable\nsan = Sanitizer(backend=\"spacy\")\n\n# High-accuracy — ~400 MB transformers model, requires PyTorch\n# Best for local deployments or privacy-critical production use\nsan = Sanitizer(backend=\"transformers\")\nThe spaCy backend (en_core_web_sm) is the default. It handles the common cases — person names, organisations, locations, emails, phone numbers — and is small enough to deploy anywhere without a GPU or a multi-gigabyte dependency install.
# spaCy (default)\npip install 'vaultmem[spacy]'\npython -m spacy download en_core_web_sm\n\n# transformers (high accuracy)\npip install 'vaultmem[presidio]'\nNot everything Presidio detects is PII. Temporal expressions (DATE_TIME), cardinal numbers, percentages, monetary values, and quantities are identifying in context but not on their own — and redacting them breaks the coherence of the context significantly.
VaultMem v0.3.2 introduced a skip list:
\n\n_SKIP_ENTITY_TYPES = frozenset({\n \"DATE_TIME\", \"CARDINAL\", \"ORDINAL\",\n \"PERCENT\", \"MONEY\", \"QUANTITY\", \"LANGUAGE\",\n})\nBefore this, a memory like “Every Friday at 4pm I do a weekly review — takes 20 minutes” was returned to the LLM as “Every [DATE_TIME_3] at [DATE_TIME_4] I do a weekly review — takes [DATE_TIME_5]”. Correct in theory, unreadable in practice. The skip list fixes this.
\n\nThe same release fixed two spaCy-specific label mismatches: spaCy reports organisations as ORGANIZATION (not ORG) and geopolitical entities as GPE (not LOCATION). Before the fix these fell through to typed tokens instead of receiving pseudonyms from the pools.
During testing I hit a bug that’s easy to miss until you actually run the system with real queries.
\n\nThe memory stored \"Sarah Chen\" (title case). The user asked \"who is sarah chen?\" (lowercase). Presidio detected \"sarah chen\" as a PERSON — but the forward map looked it up with an exact string match. \"sarah chen\" != \"Sarah Chen\", so it was treated as a new entity and assigned a fresh pseudonym: Casey.
The LLM then received context about Jordan but a query about Casey. Its response: “There is no information about Casey.” Technically accurate. Completely useless.
\n\nThe fix in v0.3.3: all forward-map keys are normalised to lowercase. A separate canonical map preserves the first-seen capitalisation so restoration returns \"Sarah Chen\", not \"sarah chen\". The variant seen first wins on casing; all subsequent variants map to the same pseudonym.
def _assign(self, real: str, entity_type: str) -> str:\n key = real.lower()\n if key in self._forward:\n return self._forward[key] # \"sarah chen\" hits \"Sarah Chen\"'s entry\n self._canonical[key] = real # first-seen casing for restoration\n ...\nThe Sanitizer addresses what leaves your device. It doesn’t address what the LLM infers or reconstructs. If your memory says “I grew up two blocks from the Eiffel Tower”, stripping names doesn’t prevent the LLM from inferring something about where you’re from.
\n\nThe complete solution is TEE-based inference — running the model inside a trusted execution environment so the provider’s infrastructure never sees plaintext at all. That’s the direction the research is heading (see the arXiv paper above), but it’s not production-ready for general-purpose models yet.
\n\nPII stripping is the practical near-term answer. It’s not perfect, but it meaningfully shrinks the surface area of personal data your LLM provider sees on every query.
\n\nSanitizer class, transformers backend, pseudonym pools, restoration mapen_core_web_sm), new [spacy] optional extra, backend= parameterDATE_TIME skip list, ORGANIZATION/GPE label fixes for spaCypip install 'vaultmem[spacy]' gets you the latest.
If you’re building an AI agent that touches personal data, I’d love to hear what you’re working on.
\n\nFor short definitions of PII sanitization, zero-knowledge storage and the other terms in this series, see the agent memory glossary.
\n" }, { "id": "https://www.avinashgosavi.com/post/vaultmem-your-ai-agent-shouldnt-read-your-diary/", "url": "https://www.avinashgosavi.com/post/vaultmem-your-ai-agent-shouldnt-read-your-diary/", "title": "Your AI Agent Shouldn't Be Able to Read Your Diary", "summary": "A plain-language case for AI memory the platform can't read: if your assistant remembers your life, who else gets to see it?", "image": "https://www.avinashgosavi.com/assets/og/vaultmem-your-ai-agent-shouldnt-read-your-diary.png", "date_published": "2026-03-28T00:00:00+00:00", "date_modified": "2026-10-21T21:35:32+00:00", "tags": ["Agent memory","Privacy"], "content_html": "Imagine you have a personal AI assistant. Over months, you tell it everything — your health worries, your relationship friction, your career anxieties, the habits you’re trying to break. It remembers. It helps.
\n\nNow ask yourself: who else can read that?
\n\nThe honest answer, with every AI memory product that exists today, is: the company running it.
\n\nWhen you use an AI agent with persistent memory — whether that’s a health companion, a journaling assistant, a personal coach, or a productivity tool — your conversations get stored somewhere. That “somewhere” is a server owned by the platform. And the platform can read it.
\n\nThis is true even when there’s a privacy policy. A privacy policy is a promise. It can be broken, subpoenaed, acquired, or quietly changed. A cryptographic guarantee cannot.
\n\nI spent a few weeks building something that fixes this at the foundation:
\n\nVaultMem — a Python library that gives AI agents encrypted, persistent, portable memory where the platform operator mathematically cannot read what users store.
\n\nNot “we promise not to look.” Not “we anonymize your data.” The key never leaves your session. The math makes it true regardless of what the policy says.
\n\nSay you’re building a personal health AI — the kind of agent a person with a chronic condition might use to track symptoms, medications, and how they’re feeling day to day.
\n\nWithout VaultMem, your architecture probably looks like this:
\n\nWith VaultMem, it looks like this:
\n\nfrom vaultmem import VaultSession, LocalEmbedder\n\nwith VaultSession.open(\"./alice_vault\", passphrase=\"alice_secret\",\n embedder=LocalEmbedder()) as s:\n s.add(\"Migraines worse since starting new medication on March 15th\")\n s.add(\"Blood pressure 120/80 at checkup — within normal range\")\n s.add(\"Feeling anxious about the scan results, haven't slept well\")\n\n results = s.search(\"recent health concerns\", top_k=3)\nYou — the developer — receive an encrypted blob you cannot open. Alice holds the key. Her memories are hers.
\n\nThe same principle applies to a therapist AI, a relationship coach, a financial planner, a personal journal. Anywhere the content is deeply personal and the user has good reason not to trust a third party with it.
\n\n\n The library handles the full memory lifecycle — not just encryption:
\n\n.vmem format is open and self-describing; your vault can move between appsThree core dependencies. No mandatory cloud service. No phone-home.
\n\nThis is a work in progress. The library is usable and the core is solid, but I’m still building toward the vision.
\n\nTry the interactive demo (no install needed): vaultmem-demo.streamlit.app
\n\nIt runs a pre-seeded vault with 36 example memories and shows you both sides — what you see (meaningful memories retrieved by your question) and what the platform sees (encrypted bytes that reveal nothing).
\n\nInstall the SDK:
\npip install vaultmem\nRead the technical writeups:
\nThe research: VaultMem implements Layer 1 of the Memory-as-Asset framework — a formal model for user-owned AI memory (plain-language TL;DR). The preprint is on Zenodo.
\n\nIf you’re building an AI agent and want to give your users memory that’s genuinely theirs — or if you have thoughts on where this should go — I’d love to hear from you.
\n" } ] }