Privacy · Avinash Gosavi

Paper TL;DR: Memory as Asset — Whose Memory Is It Anyway?

2026-10-18T20:00:00+00:00

Paper: Yanqi Pan, Qinghao Huang & Weihao Yang, Memory as Asset: From Agent-centric to Human-centric Memory Management (arXiv, March 2026)

One-line TL;DR: the memory an AI keeps about you is designed today as the agent’s internal resource; this paper argues it should be your asset — owned by you, shared only under your permissions, and able to grow over time.

This is the paper that reshaped VaultMem. I found it while I was stuck on a vague feeling that something about agent memory was backwards, and it gave that feeling a name.

The problem: agent-centric memory

Most memory systems treat what an agent remembers about you as part of the agent — stored by the platform, shaped for the platform’s use, and stuck there. The authors call this agent-centric memory. The consequences are familiar: you can’t take your history with you, you can’t decide who sees it, and every product rebuilds its own isolated “memory island” about the same person.

The proposal: memory as a personal asset

They propose flipping the default, built on three principles:

Memory in hand — individuals own and control their memories, not centralized platforms.
Memory groups — memories can be shared and combined, but only through permissions the owner sets.
Collective memory evolution — memory isn’t frozen: it’s recorded, consolidated and abstracted into higher-level knowledge over time, and can be exchanged between groups under control.

Each memory object carries its content, context, an explicit owner, an access policy and a version. Ownership and permissions are part of the data itself, not an afterthought in some admin panel.

The three layers

Personal storage — fast storage of each person’s private memory objects.
Evolution — updates memory from interactions: recording experiences, merging related memories, abstracting knowledge. It can run locally for privacy or remotely for compute.
Memory exchange network — a decentralized way to discover, authenticate and exchange memory assets.

Notably, the paper doesn’t insist everything stays on your device. If a provider hosts your memory, it should be encrypted so only you can read it.

Diagram: Memory as Asset: memory objects and the three layers (see it in the post)
Each memory object carries its content, its context and a version, plus an explicit owner and an access policy, so ownership travels with the data. Memory objects are kept in layer 1, personal storage: each person's private memory objects. VaultMem implements this layer, encrypted with a key only the user holds. Layer 2, evolution, updates memory from interactions: it records experiences, merges related memories and abstracts knowledge, and can run locally or remotely. Layer 3, the memory exchange network, discovers, authenticates and exchanges memory assets; the paper sketches it rather than solving it. Overall: owned by you, shared only through your permissions.
The paper's design in one picture: owner and access policy live inside every memory object, and three layers store, evolve and exchange those objects. VaultMem is the first layer.

How it shaped what I built

That last point is where VaultMem lives. I treat it as an implementation of the first layer: the platform stores your memories, but they’re encrypted with a key only you hold, so the operator cryptographically can’t read them. Everything in the VaultMem series — client-side encryption, encrypted search, keeping PII away from the model provider — follows from taking “memory in hand” literally.

Caveats worth knowing

The authors are upfront that this is an early draft: the terms and design are still evolving.
It’s a conceptual and architectural paper — there are no benchmarks or experiments yet. The claim that existing systems meet at most two of the three principles is an argument, not a measurement.
The exchange network raises hard questions (trust, incentives, revocation) that the paper sketches rather than solves.

I still think it’s the right question to ask before building any memory feature: whose memory is this?

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems and LoCoMo. Next in the series: Sentence-BERT, the sentence embeddings behind most semantic search.

Zero-knowledge storage and client-side encryption are defined in the privacy section of the agent memory glossary.

How daylog Keeps Your Diary Private (Even from Itself)

2026-04-12T00:00:00+00:00

daylog is a voice diary. You open it, tap the mic, say what’s on your mind, and it saves. You can search your entries semantically — type “frustrated” and it surfaces the day you wrote “couldn’t make progress, everything felt blocked” even though that exact word never appears.

That search working at all, while keeping your diary private, requires a few things to be true simultaneously: the content must be encrypted at rest, the search index must also be encrypted, and nothing should leave the device to answer a query. This post explains how each of those holds.

How your words travel

The diagram traces a single entry from your mouth to disk, and back to your eyes.

Diagram: How a daylog entry travels to disk and back (see it in the post)
Saving goes down five layers and reading comes back up the same layers. You speak, for example "went for a run today". The browser transcribes it on-device, so nothing is sent yet. HTTPS carries it encrypted in transit by TLS. The vault embeds the entry and encrypts text and vector together with AES-256-GCM using your key; when reading, it decrypts with your key only, in memory, not logged. On disk the entry is stored as ciphertext such as 7f3a a4c8 e12d, unreadable. Reading reverses the path: the vault decrypts, HTTPS carries it back, the browser shows plain text and you see it.
Readable only at the top two layers. In transit and on disk it's ciphertext, and only the vault, with your key, turns it back into words.

Your words are readable on your device and readable back in your browser. Everything in between — transit, storage — is ciphertext. Notice the saving path says “text + vector encrypted”: the search index is encrypted too, not just the content. More on that below.

What vaultmem actually does

The vault service uses vaultmem — a library I built for giving AI agents a private, searchable long-term memory. In daylog it does three things every time you save an entry:

1. Classifies the entry automatically.

Every entry is run through a fast, rule-based classifier (no LLM involved) that assigns a memory type:

EPISODIC — a time-anchored personal experience. “Yesterday I went for a run.” Has a temporal marker + first-person reference.
PERSONA — a habitual trait or preference. “I usually start work before 8am.” Has a frequency word + first-person framing.
PROCEDURAL — a how-to or sequence. “First compile, then run tests.” Has sequential markers.
SEMANTIC — a timeless fact. Everything else.

Most diary entries classify as EPISODIC. The type determines how the memory is weighted and how it ages — episodic memories have a natural recency decay, semantic ones don’t.

2. Embeds the entry as a 384-dimensional vector.

Before the entry is encrypted, it is run through sentence-transformers/all-MiniLM-L6-v2 — a 22M parameter model running locally inside the vault container. The model converts your text into a 384-float vector that captures the semantic meaning of what you wrote. (sentence-transformers is the library that came out of the Sentence-BERT paper.)

“went for a run today” → [0.312, -0.087, 0.441, ..., 0.198] (384 values)

This vector is how search finds relevant entries later. It is stored alongside the text.

3. Encrypts text + vector together and writes to disk.

The entry object — text, classification, timestamp, and embedding vector — is serialised to JSON, compressed with zlib, and encrypted with AES-256-GCM using the master encryption key (MEK). The result is appended to a .vmem file on disk as IV + length + ciphertext + GCM tag.

The embedding is encrypted. The search index — the thing that makes semantic search work — is not sitting in a plaintext index somewhere. It is inside the same ciphertext block as your diary entry. Nobody can search your vault without your key.

How search works end to end

When you type “frustrated” into the search bar:

1. The vault service embeds your query.

“frustrated” → [0.289, -0.041, 0.509, ...] (the same 384-dim model, same vector space as your stored entries).

2. It decrypts all stored embeddings from disk.

Every .vmem entry is decrypted in RAM. The vault service now holds a matrix of shape (N entries) × 384.

3. It runs a batched dot product.

Since both vectors are unit-normalised, dot product = cosine similarity. One matrix multiply gives a relevance score for every entry simultaneously.

4. It filters, ranks, and returns.

Entries with score below 0.3 are dropped. The rest are sorted and the top 8 returned — with their decrypted text.

The decrypted entries exist only in RAM during this step. They are never written back to disk.

Why this finds meaning, not just keywords.

Two sentences with similar meaning will have similar vectors even if they share no words. “couldn’t make any progress today, everything felt blocked” scores ~0.62 against “frustrated” because the model has learned that these concepts occupy the same region of the vector space.

This is the same reason “angry” might surface an entry about a deploy failing at midnight, or “energised” might surface one about a flow state — the words you actually wrote don’t have to match the words you search for.

The memory hierarchy

vaultmem stores entries at three granularity levels:

ATOM — a single diary entry. This is what you write.
COMPOSITE — an auto-generated aggregation of related atoms. A weekly summary, or a cluster of entries about the same topic. Not yet used in daylog but the foundation is there.
AFFINITY — a detected standing pattern. If the vault notices you mention sleep problems every week, it could surface that as a persistent affinity. This is where it starts to feel less like a diary and more like a memory that knows you.

Search queries all three tiers in priority order — AFFINITY first, then COMPOSITE, then ATOM — so a standing pattern surfaces above any individual entry that mentions the same thing.

The one trade-off: AI summarise

When you tap Summarise this day, the browser’s already-decrypted entries for that day are sent to Groq (Llama 3.3 70B) via the Rails API. What Groq receives:

09:10 AM: Went for a run today.
08:30 PM: Good session, finally fixed that bug.

Time + text for one day. No name, no metadata, nothing from other days. The summary is returned and displayed — it’s never saved back into the vault.

Everything else — adding entries, browsing, searching — is vault-only. No external API. Summarise is the one feature that trades privacy for convenience, and it’s a button you tap deliberately.

If that trade-off isn’t right for you, remove GROK_API_KEY from the Rails API config and everything else works unchanged.

Data at rest

Where	What	Status
Fly.io volume (`/data/vaults/`)	`.vmem` — text + embedding vectors, all encrypted	AES-256-GCM
Postgres (Neon)	email, bcrypt password hash	Passwords hashed; DB encrypted at rest
Browser sessionStorage	session_id, derived passphrase	Cleared on tab close
Browser localStorage	auth JWT	Cleared on sign out
Vault service RAM	MEK, decrypted atoms during active session	Never written to disk

The short version

Voice input: transcribed on-device by Apple’s neural engine. Nothing leaves your phone at this step.
Saving: entry is classified, embedded, then encrypted. Text and search vector hit disk together, both as ciphertext.
Searching: vault decrypts embeddings in RAM, runs cosine similarity, returns matching plaintext. No external API.
Summarising: sends that day’s entries to Groq. Optional, explicit, one day at a time.

The encrypted search index is the part most people don’t expect. Normally if you want full-text or semantic search, you need a plaintext index somewhere — Elasticsearch, pgvector, whatever. vaultmem stores the vectors encrypted alongside the content, so you get semantic search without giving the index to anyone.

Try it yourself

The app is live. You can log in with the demo account and poke around — all entries are pre-seeded across the last two months.

daylog-ds3.pages.dev

email:    demo@daylog.app
password: demo1234

Things worth trying:

Search (bottom nav → Search tab): type frustrated, flow state, couldn't sleep, or good run — these find entries by meaning, not exact words
Summarise: navigate to any day that has entries, scroll past the list, tap Summarize this day — it reads everything you logged that day and reflects back themes and mood
Voice: tap the mic, say something, watch the live transcription appear word by word

The rest of the stack (hosting, transcription, costs) is in my hobby project stack, and the open-source Whisper transcription service I extracted from daylog has its own post. Episodic, semantic, procedural and persona memory, embeddings, semantic search and decay each get a short definition in the agent memory glossary.

VaultMem Part 3 — Hiding PII from Your LLM Provider

2026-04-01T00:00:00+00:00

Part 1 covered the core encryption model. Part 2 covered multi-modal ingestion and pluggable backends. This post covers v0.3.x — a new layer of privacy that addresses a problem encryption at rest cannot solve.

Install: pip install 'vaultmem[spacy]' · Code: github.com/aag1091-alt/vaultmem-sdk · Live demo: vaultmem-demo.streamlit.app

The gap encryption at rest doesn’t close

VaultMem stores every memory atom encrypted with AES-256-GCM. Your passphrase never leaves your device. The platform operator cryptographically cannot read your vault. The threat model comes from the Memory-as-Asset framework (Pan, Huang & Yang, arXiv:2603.14212; plain-language TL;DR): memory is a personal asset, not a platform resource.

But there is a second exposure point I hadn’t fully addressed: the moment you decrypt a memory and inject it into an LLM prompt.

Say your vault contains:

I met Sarah Chen at the NeurIPS conference. She works on ML infra at Google.
My accountant James Park can be reached at james@parkaccounting.com.

When you ask your assistant “what do I know about Sarah?”, VaultMem retrieves those memories, decrypts them, and sends them as context to the LLM provider — Groq, OpenAI, Anthropic, whoever you’re using. At that point, Sarah’s name, her employer, and James’s email address are all sitting in plaintext in an HTTP request to a third-party cloud.

Encryption at rest doesn’t help here. The data is decrypted before it goes out the door.

The solution: strip PII before injection, restore it after

VaultMem v0.3.0 ships a Sanitizer class that sits between your vault and your LLM call.

from vaultmem import VaultSession, Sanitizer

san = Sanitizer(backend="spacy")   # one instance per session

with VaultSession.open("./my_vault", passphrase="...") as s:
    memories = s.search("sarah chen", top_k=5)
    context = "\n".join(f"- {m.atom.content}" for m in memories)

    # Sanitize context AND query with the same instance
    sanitized_context, rmap = san.sanitize(context)
    sanitized_query, rmap   = san.sanitize(user_query)

    raw_reply = llm.chat(sanitized_query, context=sanitized_context)

    # Restore real values in the LLM's response
    final_reply = san.restore(raw_reply, rmap)

The LLM provider receives:

I met Jordan at the NeurIPS conference. She works on ML infra at Globex.
My accountant Casey can be reached at [EMAIL_1].

It reasons about this correctly, responds naturally, and VaultMem restores the real values client-side before the answer is shown to the user.

The provider never sees the real names, companies, or contact details.

Diagram: How the VaultMem sanitizer round-trips a prompt (see it in the post)
On your device, memories are decrypted from the vault, for example Sarah Chen, who works at Google, and James's email address. Sanitizer.sanitize swaps them for a pseudonym, a fictional company and a typed token: Jordan, Globex and EMAIL_1, using one mapping for the whole session. Only that sanitized text goes to the LLM provider. The provider's reply mentions Jordan. On your device again, Sanitizer.restore maps Jordan back to Sarah Chen, so the answer you read uses the real names.
Real names and contact details stay on your device; the provider only ever reasons about Jordan, Globex and [EMAIL_1].

Pseudonyms, not tokens

Person names are replaced with natural-sounding pseudonyms — Jordan, Casey, Morgan, Riley, Avery — rather than opaque tokens like [PERSON_1].

The reason is fluency. If the LLM sees [PERSON_1] in context and addresses the user with it, the response reads awkwardly. With a pseudonym, it can write “Based on what Jordan told you at NeurIPS…” and after restoration the user reads “Based on what Sarah Chen told you at NeurIPS…” — completely natural.

Organisations get fictional company names (Acme, Globex, Initech, Pied Piper). Structured PII — emails, phone numbers, SSNs, credit cards, IPs — gets typed tokens ([EMAIL_1], [PHONE_2]) because there’s no natural-sounding substitute for those.

The mapping is session-scoped and stable: if “Sarah Chen” appears in five different memories, she is always Jordan in that session. The LLM can reason about her as a single consistent person across all memories.

Two backends

Presidio supports different NLP backends. VaultMem exposes both:

# Lightweight — ~12 MB spaCy model, no PyTorch dependency
# Works on Streamlit Cloud, CI, anywhere torch is unavailable
san = Sanitizer(backend="spacy")

# High-accuracy — ~400 MB transformers model, requires PyTorch
# Best for local deployments or privacy-critical production use
san = Sanitizer(backend="transformers")

The spaCy backend (en_core_web_sm) is the default. It handles the common cases — person names, organisations, locations, emails, phone numbers — and is small enough to deploy anywhere without a GPU or a multi-gigabyte dependency install.

# spaCy (default)
pip install 'vaultmem[spacy]'
python -m spacy download en_core_web_sm

# transformers (high accuracy)
pip install 'vaultmem[presidio]'

What gets skipped

Not everything Presidio detects is PII. Temporal expressions (DATE_TIME), cardinal numbers, percentages, monetary values, and quantities are identifying in context but not on their own — and redacting them breaks the coherence of the context significantly.

VaultMem v0.3.2 introduced a skip list:

_SKIP_ENTITY_TYPES = frozenset({
    "DATE_TIME", "CARDINAL", "ORDINAL",
    "PERCENT", "MONEY", "QUANTITY", "LANGUAGE",
})

Before this, a memory like “Every Friday at 4pm I do a weekly review — takes 20 minutes” was returned to the LLM as “Every [DATE_TIME_3] at [DATE_TIME_4] I do a weekly review — takes [DATE_TIME_5]”. Correct in theory, unreadable in practice. The skip list fixes this.

The same release fixed two spaCy-specific label mismatches: spaCy reports organisations as ORGANIZATION (not ORG) and geopolitical entities as GPE (not LOCATION). Before the fix these fell through to typed tokens instead of receiving pseudonyms from the pools.

A subtle bug: case-insensitive deduplication

During testing I hit a bug that’s easy to miss until you actually run the system with real queries.

The memory stored "Sarah Chen" (title case). The user asked "who is sarah chen?" (lowercase). Presidio detected "sarah chen" as a PERSON — but the forward map looked it up with an exact string match. "sarah chen" != "Sarah Chen", so it was treated as a new entity and assigned a fresh pseudonym: Casey.

The LLM then received context about Jordan but a query about Casey. Its response: “There is no information about Casey.” Technically accurate. Completely useless.

The fix in v0.3.3: all forward-map keys are normalised to lowercase. A separate canonical map preserves the first-seen capitalisation so restoration returns "Sarah Chen", not "sarah chen". The variant seen first wins on casing; all subsequent variants map to the same pseudonym.

def _assign(self, real: str, entity_type: str) -> str:
    key = real.lower()
    if key in self._forward:
        return self._forward[key]          # "sarah chen" hits "Sarah Chen"'s entry
    self._canonical[key] = real            # first-seen casing for restoration
    ...

What this doesn’t solve

The Sanitizer addresses what leaves your device. It doesn’t address what the LLM infers or reconstructs. If your memory says “I grew up two blocks from the Eiffel Tower”, stripping names doesn’t prevent the LLM from inferring something about where you’re from.

The complete solution is TEE-based inference — running the model inside a trusted execution environment so the provider’s infrastructure never sees plaintext at all. That’s the direction the research is heading (see the arXiv paper above), but it’s not production-ready for general-purpose models yet.

PII stripping is the practical near-term answer. It’s not perfect, but it meaningfully shrinks the surface area of personal data your LLM provider sees on every query.

Changelog

v0.3.0 — Sanitizer class, transformers backend, pseudonym pools, restoration map
v0.3.1 — spaCy backend (en_core_web_sm), new [spacy] optional extra, backend= parameter
v0.3.2 — DATE_TIME skip list, ORGANIZATION/GPE label fixes for spaCy
v0.3.3 — case-insensitive entity deduplication

pip install 'vaultmem[spacy]' gets you the latest.

If you’re building an AI agent that touches personal data, I’d love to hear what you’re working on.

For short definitions of PII sanitization, zero-knowledge storage and the other terms in this series, see the agent memory glossary.

Your AI Agent Shouldn't Be Able to Read Your Diary

2026-03-28T00:00:00+00:00

Imagine you have a personal AI assistant. Over months, you tell it everything — your health worries, your relationship friction, your career anxieties, the habits you’re trying to break. It remembers. It helps.

Now ask yourself: who else can read that?

The honest answer, with every AI memory product that exists today, is: the company running it.

The problem isn’t the AI. It’s where the memory lives.

When you use an AI agent with persistent memory — whether that’s a health companion, a journaling assistant, a personal coach, or a productivity tool — your conversations get stored somewhere. That “somewhere” is a server owned by the platform. And the platform can read it.

This is true even when there’s a privacy policy. A privacy policy is a promise. It can be broken, subpoenaed, acquired, or quietly changed. A cryptographic guarantee cannot.

I spent a few weeks building something that fixes this at the foundation:

VaultMem — a Python library that gives AI agents encrypted, persistent, portable memory where the platform operator mathematically cannot read what users store.

Not “we promise not to look.” Not “we anonymize your data.” The key never leaves your session. The math makes it true regardless of what the policy says.

A real-world example: the health companion

Say you’re building a personal health AI — the kind of agent a person with a chronic condition might use to track symptoms, medications, and how they’re feeling day to day.

Without VaultMem, your architecture probably looks like this:

User tells agent “my migraines have been worse since I started the new medication”
Agent stores that in a database you control
You — the developer — can query that database
So can anyone who compromises it, or a legal subpoena, or your future acquirer

With VaultMem, it looks like this:

from vaultmem import VaultSession, LocalEmbedder

with VaultSession.open("./alice_vault", passphrase="alice_secret",
                        embedder=LocalEmbedder()) as s:
    s.add("Migraines worse since starting new medication on March 15th")
    s.add("Blood pressure 120/80 at checkup — within normal range")
    s.add("Feeling anxious about the scan results, haven't slept well")

    results = s.search("recent health concerns", top_k=3)

You — the developer — receive an encrypted blob you cannot open. Alice holds the key. Her memories are hers.

Diagram: Who can read an agent's memory, with and without VaultMem (see it in the post)
Typical agent memory: what you tell the agent is stored in the platform's database, readable by the platform, so the developer, anyone who breaches it, a subpoena or a future acquirer can read every memory. With VaultMem: you hold the key, the VaultSession encrypts in your session, and the platform stores only an encrypted blob, so the developer, a breach, a subpoena or an acquirer see only encrypted bytes.
Same app, same people around it. The difference is where the key lives: with VaultMem it never leaves your session.

The same principle applies to a therapist AI, a relationship coach, a financial planner, a personal journal. Anywhere the content is deeply personal and the user has good reason not to trust a third party with it.

What it does

The library handles the full memory lifecycle — not just encryption:

Four memory types auto-classified from content: episodic (events), semantic (facts), persona (preferences), procedural (how-tos)
Smart retrieval that surfaces recurring habits and patterns above one-off mentions — so “my morning coffee” surfaces a pattern summary, not just the last time you mentioned coffee
Time-aware search — ask “what was I doing last summer?” and it searches by when events actually happened, not when you typed them in
Multi-modal — photos, voice notes, PDFs ingested as encrypted memory atoms with EXIF dates and GPS stripped from metadata
Portable — the .vmem format is open and self-describing; your vault can move between apps
Production backends — S3 for storage, Postgres for indexing, HNSW for fast search at scale

Three core dependencies. No mandatory cloud service. No phone-home.

It’s early — and I’d love feedback

This is a work in progress. The library is usable and the core is solid, but I’m still building toward the vision.

Try the interactive demo (no install needed): vaultmem-demo.streamlit.app

It runs a pre-seeded vault with 36 example memories and shows you both sides — what you see (meaningful memories retrieved by your question) and what the platform sees (encrypted bytes that reveal nothing).

Install the SDK:

pip install vaultmem

Read the technical writeups:

The research: VaultMem implements Layer 1 of the Memory-as-Asset framework — a formal model for user-owned AI memory (plain-language TL;DR). The preprint is on Zenodo.

If you’re building an AI agent and want to give your users memory that’s genuinely theirs — or if you have thoughts on where this should go — I’d love to hear from you.

VaultMem Part 2 — Multi-Modal Memory, Temporal Search, and Pluggable Backends

2026-03-28T00:00:00+00:00

Part 1 covered why I built VaultMem, how the encryption works, and the core memory model. This post covers what came next in v0.2.0 and v0.2.1 — the parts that took the library from a solid foundation to something that can handle real-world agent use cases: multi-modal memory ingestion, time-aware retrieval, production-grade storage backends, O(log N) ANN search, and a pluggable query normalizer.

Install: pip install vaultmem · Code: github.com/aag1091-alt/vaultmem-sdk · Live demo: vaultmem-demo.streamlit.app

Text-only memory is a significant limitation for a personal AI agent. The moments that matter most — a photo from a trip, a voice note recorded mid-commute, a scanned medical document — aren’t text.

VaultMem v0.2.0 added add_media(), which ingests an image, audio file, PDF, or video and stores it as a fully encrypted memory atom. The raw bytes go into vault/media/{uuid}.enc (AES-256-GCM, same key architecture as text atoms). The extracted content becomes the searchable atom.

with VaultSession.open("./vault", passphrase, embedder=embedder) as s:
    # Image — extracts EXIF date, GPS coordinates, optional OCR
    atom = s.add_media("holiday_photo.jpg", passphrase)
    print(atom.content)      # "Photo taken 2023-07-14 in Split, Croatia"
    print(atom.captured_at)  # EXIF timestamp as Unix int
    print(atom.location)     # {"lat": 43.5, "lon": 16.4, "place": "Split"}

    # Audio — Whisper transcription
    atom = s.add_media("voice_note.mp3", passphrase)

    # PDF — PyMuPDF text extraction
    atom = s.add_media("contract.pdf", passphrase)

    # Retrieve raw bytes later (decrypted on demand, not in search results)
    raw = s.get_media(atom.id, passphrase)

The media bytes are never loaded into RAM during search — only the extracted text atom participates in embedding and retrieval. This keeps the privacy model intact: even if you extract metadata, the raw file stays encrypted behind the passphrase.

Each extractor is a pluggable Protocol, so adding new file types is straightforward:

from vaultmem import MediaExtractor, MediaExtractionResult, MediaIngester

class HeicExtractor:
    MIME_TYPES = {"image/heic", "image/heif"}

    def extract(self, path, mime_type):
        return MediaExtractionResult(
            content_type=mime_type,
            transcript="...",
            captured_at=...,
        )

ingester = MediaIngester(extra_extractors=[HeicExtractor()])

Install the extras to activate the built-in extractors:

pip install "vaultmem[media]"   # Pillow, piexif, openai-whisper, PyMuPDF, ffmpeg-python

Temporal search

The second gap I wanted to close: time. A personal agent should be able to answer “what was I doing in 2019?” or “show me everything from last summer” — not just “what’s most semantically similar to this query?”

This required two new timestamp fields on MemoryObject:

created_at — when the atom was written to the vault (set automatically, immutable)
captured_at — when the real-world event happened (set manually or from EXIF/metadata)

These are deliberately separate. A photo taken in 2019 but added to the vault today has captured_at in 2019 and created_at today. The distinction matters for any use case involving retroactive journaling or importing old media.

Browse by time window

from datetime import datetime, timezone

def ts(year, month=1, day=1):
    return int(datetime(year, month, day, tzinfo=timezone.utc).timestamp())

with VaultSession.open("./vault", passphrase, embedder=embedder) as s:
    memories_2019 = s.search_by_time(ts(2019), ts(2020))
    q3_2021 = s.search_by_time(ts(2021, 7), ts(2021, 10))

    # What was written to the vault in the last 24 hours
    import time
    recent = s.diff(int(time.time()) - 86400, int(time.time()))

Natural-language time queries

I wanted to avoid the experience of asking “what happened summer 2019?” and getting results from all time. The TimeQueryParser extracts a timestamp range from the query string, strips the time phrase, and passes the remainder to the embedding — all in pure Python with no dependencies:

from vaultmem import TimeQueryParser

from_ts, to_ts, remainder = TimeQueryParser.parse("what happened summer 2019")
# from_ts  → 2019-06-01 UTC
# to_ts    → 2019-09-01 UTC
# remainder → "what happened"

Supported phrases: season + year, month + year, bare year, “last year”, “this month”, “last N days”, “yesterday”.

Pass parse_time=True to search() and it handles everything automatically:

# Strips "in 2019" → timestamp range; embeds "what did I do"
results = s.search("what did I do in 2019", top_k=10, parse_time=True)

Diagram: How VaultMem splits a natural-language time query (see it in the post)
The query "what happened summer 2019" goes to TimeQueryParser.parse, which runs in pure Python with no dependencies. It splits the query in two. The time phrase, summer 2019, becomes a timestamp range from 2019-06-01 to 2019-09-01 UTC. The remainder, what happened, is what gets embedded. search with parse_time=True uses both, so the results come from summer 2019 instead of from all time.
The time phrase becomes a timestamp range; only what's left of the query is embedded.

Pluggable storage backends

The default storage model — two files on the local filesystem — is the right starting point. But a production agent deployment might need S3 for durability, Postgres for searchability, or both. I didn’t want to bake those in as optional flags on the session object; I wanted them to be proper extension points.

v0.2.0 introduced three ABCs:

ABC	Default	Production
`BlobStore`	`FileBlobStore`	`S3BlobStore`
`SearchIndex`	`SQLiteSearchIndex`	`PostgresSearchIndex`
`VectorIndex`	— (exact O(N) cosine)	`HNSWVectorIndex`

All three are injected at session creation:

from vaultmem import (
    VaultSession, LocalEmbedder,
    S3BlobStore, PostgresSearchIndex, HNSWVectorIndex,
)

with VaultSession.open(
    "./vault", passphrase,
    embedder=LocalEmbedder(),
    blob_store=S3BlobStore(bucket="my-vault-atoms"),
    search_index=PostgresSearchIndex("postgresql://user:pass@host/db"),
    vector_index=HNSWVectorIndex(),
) as s:
    results = s.search("coffee habits", top_k=10)

The important design decision: the vault format stays the same regardless of which backends you use. FileBlobStore and S3BlobStore store the same encrypted bytes. SQLiteSearchIndex and PostgresSearchIndex store the same metadata. Migrating between backends is O(N) byte-copies, no decryption required:

from vaultmem import migrate_vault

migrate_vault(
    src_blob_store=FileBlobStore("./old"),
    dst_blob_store=S3BlobStore(bucket="new-vault"),
    passphrase=passphrase,
)

HNSW and the O(N) problem

At small scale, exact cosine search over all atoms in RAM is fine. At 10,000 atoms it becomes the bottleneck — the benchmark showed ~148ms per query, all of it in the embedding comparison loop.

HNSWVectorIndex wraps hnswlib and plugs into the same VectorIndex ABC. The HNSW graph is built incrementally as atoms are added and serialized encrypted to vault/vector_index.hnsw.enc on flush. At session open it’s loaded once into RAM. Search then runs in O(log N):

Atoms	Exact cosine	HNSW
1,000	~15ms	~0.3ms
10,000	~148ms	~0.8ms
100,000	~1,500ms	~1.2ms

Below 1,000 atoms, the implementation falls back to exact cosine automatically — the HNSW graph has overhead that isn’t worth paying at small N. (For how the HNSW graph itself works, see the Paper TL;DR on HNSW.)

pip install "vaultmem[ann]"   # hnswlib

Query normalizer

A subtler problem surfaced while building the interactive demo: the question framing users naturally type — “What do I know about Sarah Chen?” — dilutes the embedding. A bag-of-words or hash-projection embedder assigns equal weight to “what”, “do”, “i”, “know”, “about” and “Sarah”, “Chen”. The meaningful terms are swamped by question scaffolding.

Sentence-transformers handle this natively through attention — they learn that “Sarah Chen” carries the semantic weight in the sentence. But not every deployment uses a 22M-parameter ML model.

v0.2.1 introduced a QueryNormalizer Protocol:

from vaultmem import QueryNormalizer  # Protocol — implement normalize(text) -> str

The built-in RegexQueryNormalizer strips common English question preamble:

from vaultmem import RegexQueryNormalizer

# "What do I know about Sarah Chen?" → "Sarah Chen"
# "How does Flash Attention work?"   → "Flash Attention work"
# "Am I vegetarian?"                 → "vegetarian"

More interesting is the custom path — since it’s a Protocol, any object with a normalize(self, text: str) -> str method works. The interactive demo uses a Groq-backed normalizer with a server-side API key, so users get LLM-quality query extraction without needing their own key:

class GroqQueryNormalizer:
    def __init__(self, api_key: str) -> None:
        from groq import Groq
        self._client = Groq(api_key=api_key)

    def normalize(self, text: str) -> str:
        resp = self._client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            max_tokens=20,
            messages=[
                {"role": "system",
                 "content": "Extract key search terms. Return only keywords."},
                {"role": "user", "content": text},
            ],
        )
        return resp.choices[0].message.content.strip() or text

The normalizer is injected once at session creation and runs on all searches where normalize_query=True:

with VaultSession.open("./vault", passphrase,
                        query_normalizer=GroqQueryNormalizer(api_key),
                        embedder=embedder) as s:
    results = s.search("What do I know about Sarah?", normalize_query=True)

Diagram: How VaultMem's query normalizer strips question framing before search (see it in the post)
The question "What do I know about Sarah Chen?" goes to a QueryNormalizer, any object with normalize(text) returning a string. Two kinds plug in: the built-in RegexQueryNormalizer, which strips common English question preamble, or your own class, such as the demo's Groq-backed normalizer that asks an LLM to return only keywords. Either way the result is "Sarah Chen"; the question scaffolding "What do I know about" is dropped. search with normalize_query=True then embeds only those key terms.
The normalizer runs before embedding, so the question's scaffolding words don't drown out the names that matter.

Data classes

One design detail worth calling out: memory doesn’t all decay at the same rate, and not all memory deserves the same cryptographic protection.

VaultMem has three data classes, set at vault creation:

Class	Use case	Argon2id memory	Decay λ	Half-life
`GENERAL`	Standard	64 MiB	0.005	~5 months
`MEDICAL`	Health data	128 MiB + mlock	0.002	~12 months
`ARCHIVAL`	Long-term records	64 MiB	0.0007	~3 years

MEDICAL doubles the Argon2id memory cost, making brute-force twice as expensive. ARCHIVAL slows the significance decay so memories stay surfaced for years rather than months — appropriate for journals, legal records, or anything you want to last.

VaultSession.create("./health_vault", passphrase, owner="alice", data_class="MEDICAL")

The LoCoMo benchmark

I wanted a more rigorous evaluation than the 18-atom synthetic benchmark I described in Part 1. The LoCoMo dataset is a collection of long-form conversational memory — 1,982 QA pairs across five categories (single-hop factual, multi-hop reasoning, temporal, adversarial, open-domain). It’s the closest thing to a realistic test of a personal memory system. (For how the benchmark was built, see the LoCoMo Paper TL;DR.)

VaultMem’s Recall@10 and MRR scores on a random sample:

Category	Recall@10	MRR
Single-hop factual	0.91	0.87
Temporal	0.84	0.79
Multi-hop	0.73	0.68
Adversarial	0.78	0.71

Multi-hop is the hardest — it requires reasoning across atoms that weren’t individually asked about, which is a retrieval problem before it’s even an LLM problem.

The interactive demo

The fastest way to see all of this in action is the live demo. It runs a pre-seeded vault with 36 memories across all four types — enough variety to demonstrate the retrieval model without you having to type anything first.

Every retrieved result shows its relevancy score, memory type, and retrieval tier. The right panel shows the raw encrypted bytes of current.vmem and current.atoms — what the platform operator actually sees. Each browser tab gets an isolated vault; your data can’t be seen by or contaminated by other visitors.

Some things worth trying:

Question	What it exercises
What do I know about Sarah Chen?	Groq query normalizer → EPISODIC retrieval
How does Flash Attention work?	SEMANTIC retrieval
What’s my coffee order?	PERSONA retrieval
How do I debug problems?	PROCEDURAL retrieval
Tell me about my cat	Natural language → EPISODIC

What’s next

The library is at v0.2.1. The things I’m thinking about for v0.3.0:

Composite and Affinity generation — right now the tiers exist in the retrieval model but atoms are only created at ATOM granularity. Automated COMPOSITE and AFFINITY synthesis from accumulated atoms is the next big addition.
Cross-agent memory sharing — Layer 2 of the Memory-as-Asset framework. A shared encrypted memory space where multiple agents can read without the platform being able to.
LoCoMo full benchmark — running the complete 1,982-pair dataset rather than a sample.

The paper targeting ACM CCS 2026 is in progress.

VaultMem Part 3 — Hiding PII from Your LLM Provider →

pip install vaultmem · GitHub · Preprint · Live demo

VaultMem — Building Encrypted Memory for AI Agents

2026-03-21T00:00:00+00:00

While building Rena (code) — a voice AI health agent I submitted for the Google Gemini Live Agent Challenge — I needed persistent memory. Something that could build a user profile from conversations across sessions and recall it the next time you speak to the agent. Libraries like mem0 or Zep would have worked fine for that use case.

But the more I thought about it, the more a different question started bothering me: what if I wanted a truly private agent — one where the platform running it genuinely cannot read my conversations? And beyond that, how do you store raw conversations as structured memory in the first place?

I couldn’t find anything that answered both. So I spent a few days building it: a memory library for AI agents where the platform developer cryptographically cannot read what the user stores.

Install: pip install vaultmem · Code: github.com/aag1091-alt/vaultmem-sdk · Preprint: doi.org/10.5281/zenodo.19154079

The problem

Every AI memory library I looked at — mem0, Zep, LangMem, Letta — stores user memories in plaintext on the platform’s servers. The platform operator can read everything their users have ever told the agent. This isn’t a privacy policy failure; it’s a structural one. When the key lives with the operator, the guarantee lives with the operator.

This matters more than it sounds. The deepest use case for persistent AI memory isn’t task management — it’s candor. An agent that remembers across sessions only becomes genuinely useful when you can be completely honest with it. A person working through a difficult decision. Someone with early-stage dementia building a longitudinal record of their life. A user who just wants their preferences remembered without those preferences living in a corporate database.

For that to work, the privacy guarantee has to be mathematical, not a privacy policy.

Where the research led

While digging into how existing systems handle memory, I stumbled on a paper: Memory as Asset by Pan, Huang & Yang (2026) (plain-language TL;DR). It formalizes something I’d been circling around informally — that personal memory isn’t just data to be stored; it’s an asset that belongs to the user, with ownership, permissions, and portability as first-class concerns.

The paper defines a layered architecture. Layer 1 is personal storage: encrypted, user-owned, portable. Layer 2 is cross-session intelligence. Layer 3 is collaborative memory between agents. No existing library implements even Layer 1 with real cryptographic guarantees.

What the paper gave me more than anything was vocabulary. Before reading it, I was thinking vaguely about “storing conversations.” After, I was thinking about episodic memory, semantic memory, significance decay, and granularity hierarchies. That shift changed what I built — instead of a simple key-value store with embeddings, VaultMem ended up with a proper memory model grounded in the framework. The paper became the specification.

What VaultMem is

VaultMem is an embeddable Python library. A platform developer adds it as a dependency, and their users get encrypted, portable, private memory — without the developer being able to read it.

The analogy I keep coming back to: VaultMem is to AI memory what 1Password is to credentials. The library doesn’t know what it’s storing. The platform receives opaque ciphertext it can store, back up, and serve back — but cannot decrypt.

from vaultmem import VaultSession, OllamaEmbedder

embedder = OllamaEmbedder("http://localhost:11434")

# Create an encrypted vault
with VaultSession.create("./vault", passphrase="s3cr3t", owner="alice",
                          embedder=embedder) as s:
    s.add("I met Bob at the AI conference yesterday")
    s.add("I prefer concise bullet-point answers")
    s.add("My go-to language for data work is Python")

# Search across sessions
with VaultSession.open("./vault", passphrase="s3cr3t",
                        embedder=embedder) as s:
    results = s.search("Bob conference")
    for r in results:
        print(f"[{r.tier}] {r.score:.3f}  {r.atom.content}")

Output:

[ATOM] 0.847  I met Bob at the AI conference yesterday

How the encryption works

The key architecture has two layers:

User passphrase
  → Argon2id(passphrase, salt)      ← 64 MiB, 3 iterations
  → KEK (Key Encryption Key)         ← in RAM only; zeroed after open
  → AES-GCM-Dec(KEK, wrapped_MEK)
  → MEK (Master Encryption Key)      ← in RAM during session only
  → AES-256-GCM(memory atom)        ← to disk

The Master Encryption Key (MEK) is generated once at vault creation and stored only as a ciphertext wrapped with the user’s passphrase-derived key. It never touches disk in plaintext. When the session closes, it’s zeroed from RAM.

Diagram: VaultMem's two-layer key architecture (see it in the post)
The user passphrase goes through Argon2id with a salt (64 MiB, 3 iterations) to derive the KEK, the key encryption key. The KEK lives in RAM only and decrypts the wrapped MEK with AES-GCM. The MEK, the master encryption key, stays in RAM during the session and is zeroed when it closes. The MEK encrypts each memory atom with AES-256-GCM, with a 128-bit authentication tag and the atom's UUID bound as additional authenticated data. On disk there is only the salt with the wrapped MEK, and the encrypted memory atoms.
Only ciphertext ever reaches disk. Because the MEK is wrapped separately from the data, a passphrase change re-wraps one key instead of re-encrypting every atom.

A platform developer who integrates VaultMem receives encrypted blobs from their users. They can store them, back them up, sync them — but they cannot read them. The math makes this true regardless of what their privacy policy says.

Two things I’m particularly happy with:

Passphrase changes are O(1). Because the MEK is wrapped separately from the data, changing a passphrase means re-wrapping one key — not re-encrypting every memory atom. A vault with 10,000 atoms changes passphrase in milliseconds.

Per-atom tamper detection. Each atom is encrypted with AES-256-GCM, which includes a 128-bit authentication tag. Each atom also has its UUID bound as Additional Authenticated Data, so you can’t transplant a valid ciphertext from one slot to another. Any tampering is detected before the data surfaces.

The memory model

The Memory-as-Asset paper defines memory as a 5-tuple: content, context, owner, permissions, versioning state. VaultMem’s MemoryObject implements this directly.

Four memory types: EPISODIC (things that happened), SEMANTIC (facts), PERSONA (preferences and habits), PROCEDURAL (how-to knowledge). Type is assigned by a deterministic classifier — no LLM call, no network request.

The more interesting part is the granularity hierarchy:

ATOM — a single memory statement. Immutable once written.
COMPOSITE — a type-homogeneous grouping of related atoms.
AFFINITY — a derived atom representing a recurring pattern. “User drinks coffee every morning” synthesized from twenty individual coffee mentions.

AFFINITY atoms carry a significance score:

σ = (1 − e^(−κ × freq)) × e^(−λ × days_since_last)

Frequency drives the first factor (with diminishing returns), recency drives the second. The parameters vary by data class — medical memories decay slower than general ones.

Three-tier retrieval — and why it matters

The retrieval model searches AFFINITY → COMPOSITE → ATOM and re-ranks results globally. On factual queries this behaves identically to flat cosine search — no regression. On pattern queries it’s a different story.

I ran a benchmark to verify this. Three topic clusters (coffee habits, Python usage, exercise routine), each with one AFFINITY atom and five ATOM-granularity memories including a deliberately tricky decoy: an episodic memory containing the exact query keywords, designed to fool flat cosine.

For the query “my morning coffee habits”, flat cosine ranks the decoy atom first (it contains the words “coffee habits” and “morning”) and the AFFINITY second. Three-tier flips this — the AFFINITY’s significance score (~0.92 for a well-established habit) outweighs the cosine advantage of the decoy.

Diagram: VaultMem's three-tier retrieval (see it in the post)
A query is searched against three tiers in order: AFFINITY atoms, which are derived recurring patterns that carry a significance score; COMPOSITE groupings of related atoms; and individual ATOM statements. All results are then re-ranked globally using cosine similarity plus significance. For the query "my morning coffee habits", flat cosine ranks a decoy atom first and the AFFINITY second, while three-tier retrieval ranks the AFFINITY first.
The decoy repeats the query's words, so flat cosine puts it first. The AFFINITY's significance score lifts the real habit to the top; specific queries rank the same as flat cosine.

Metric	Flat cosine	Three-tier	Δ
MRR (pattern queries)	0.50	1.00	+0.50
MRR (specific queries)	1.00	1.00	0.00

Pattern queries: AFFINITY surfaced first every time. Specific queries: no regression. The significance weight isn’t a heuristic — it’s a correction for a systematic bias in pure cosine retrieval.

The `.vmem` format

Vaults are stored as two files:

my_vault/
├── current.vmem      # encrypted index (48-byte self-describing header)
└── current.atoms     # append-only encrypted atom blocks

The 48-byte header is the only plaintext in the file. It carries the magic bytes (VMEM), format version, encryption algorithm, KDF, and file type — enough for any conforming implementation to decrypt the vault given the passphrase. No prior coordination with the writing implementation needed.

Atoms are append-only and immutable. Checkpoints update only the index file, atomically via fsync + os.rename(). Crash-safe.

The format is designed to be the interoperability layer for a future where encrypted memory vaults move between applications and agents — the same way a PDF can be opened by any conforming reader regardless of which app produced it.

What’s in the library

About 2,250 lines of Python, 39 unit tests, three core dependencies (cryptography, argon2-cffi, numpy). Sentence-transformers or a local Ollama instance for embeddings — or bring your own.

# LocalEmbedder — no network, runs on CPU
from vaultmem import LocalEmbedder
embedder = LocalEmbedder()

# OllamaEmbedder — GPU-accelerated, localhost or Tailscale
from vaultmem import OllamaEmbedder
embedder = OllamaEmbedder("http://localhost:11434", model="all-minilm")

# Custom — implement .embed(text) -> list[float]
class MyEmbedder:
    def embed(self, text: str) -> list[float]: ...
    @property
    def dimension(self) -> int: return 384

The research angle

The work turned into a paper. VaultMem is the first implementation of Layer 1 of the Memory-as-Asset framework — the paper I stumbled on during the initial research that ended up shaping the entire design. The paper formalizes the threat model, scores ten existing AI memory systems against a four-dimension privacy taxonomy (no prior system achieves all four properties simultaneously), and specifies the .vmem format as a portable standard.

The preprint is live on Zenodo and I’m targeting ACM CCS 2026 for the full paper.

Try it

pip install vaultmem                   # core
pip install "vaultmem[local]"          # + local embeddings (sentence-transformers)

The library is intentionally small — the code is the specification. If you’re building an AI agent and want to give your users memory that’s actually theirs, this is the starting point.

github.com/aag1091-alt/vaultmem-sdk

Part 2 covers what was built on top of this foundation in v0.2.0 and v0.2.1: multi-modal memory ingestion, temporal search, pluggable storage backends, HNSW vector indexing (how HNSW works), and a query normalizer Protocol — including a live interactive demo.

VaultMem Part 2 — Multi-Modal, Temporal Search, and Pluggable Backends →

New to terms like zero-knowledge storage, client-side encryption or episodic memory? The agent memory glossary explains them in plain language. For the non-technical case behind all of this, see Your AI Agent Shouldn’t Be Able to Read Your Diary.