Building VaultMem · Avinash Gosavi

VaultMem Part 3 — Hiding PII from Your LLM Provider

2026-04-01T00:00:00+00:00

Part 1 covered the core encryption model. Part 2 covered multi-modal ingestion and pluggable backends. This post covers v0.3.x — a new layer of privacy that addresses a problem encryption at rest cannot solve.

Install: pip install 'vaultmem[spacy]' · Code: github.com/aag1091-alt/vaultmem-sdk · Live demo: vaultmem-demo.streamlit.app

The gap encryption at rest doesn’t close

VaultMem stores every memory atom encrypted with AES-256-GCM. Your passphrase never leaves your device. The platform operator cryptographically cannot read your vault. The threat model comes from the Memory-as-Asset framework (Pan, Huang & Yang, arXiv:2603.14212; plain-language TL;DR): memory is a personal asset, not a platform resource.

But there is a second exposure point I hadn’t fully addressed: the moment you decrypt a memory and inject it into an LLM prompt.

Say your vault contains:

I met Sarah Chen at the NeurIPS conference. She works on ML infra at Google.
My accountant James Park can be reached at james@parkaccounting.com.

When you ask your assistant “what do I know about Sarah?”, VaultMem retrieves those memories, decrypts them, and sends them as context to the LLM provider — Groq, OpenAI, Anthropic, whoever you’re using. At that point, Sarah’s name, her employer, and James’s email address are all sitting in plaintext in an HTTP request to a third-party cloud.

Encryption at rest doesn’t help here. The data is decrypted before it goes out the door.

The solution: strip PII before injection, restore it after

VaultMem v0.3.0 ships a Sanitizer class that sits between your vault and your LLM call.

from vaultmem import VaultSession, Sanitizer

san = Sanitizer(backend="spacy")   # one instance per session

with VaultSession.open("./my_vault", passphrase="...") as s:
    memories = s.search("sarah chen", top_k=5)
    context = "\n".join(f"- {m.atom.content}" for m in memories)

    # Sanitize context AND query with the same instance
    sanitized_context, rmap = san.sanitize(context)
    sanitized_query, rmap   = san.sanitize(user_query)

    raw_reply = llm.chat(sanitized_query, context=sanitized_context)

    # Restore real values in the LLM's response
    final_reply = san.restore(raw_reply, rmap)

The LLM provider receives:

I met Jordan at the NeurIPS conference. She works on ML infra at Globex.
My accountant Casey can be reached at [EMAIL_1].

It reasons about this correctly, responds naturally, and VaultMem restores the real values client-side before the answer is shown to the user.

The provider never sees the real names, companies, or contact details.

Diagram: How the VaultMem sanitizer round-trips a prompt (see it in the post)
On your device, memories are decrypted from the vault, for example Sarah Chen, who works at Google, and James's email address. Sanitizer.sanitize swaps them for a pseudonym, a fictional company and a typed token: Jordan, Globex and EMAIL_1, using one mapping for the whole session. Only that sanitized text goes to the LLM provider. The provider's reply mentions Jordan. On your device again, Sanitizer.restore maps Jordan back to Sarah Chen, so the answer you read uses the real names.
Real names and contact details stay on your device; the provider only ever reasons about Jordan, Globex and [EMAIL_1].

Pseudonyms, not tokens

Person names are replaced with natural-sounding pseudonyms — Jordan, Casey, Morgan, Riley, Avery — rather than opaque tokens like [PERSON_1].

The reason is fluency. If the LLM sees [PERSON_1] in context and addresses the user with it, the response reads awkwardly. With a pseudonym, it can write “Based on what Jordan told you at NeurIPS…” and after restoration the user reads “Based on what Sarah Chen told you at NeurIPS…” — completely natural.

Organisations get fictional company names (Acme, Globex, Initech, Pied Piper). Structured PII — emails, phone numbers, SSNs, credit cards, IPs — gets typed tokens ([EMAIL_1], [PHONE_2]) because there’s no natural-sounding substitute for those.

The mapping is session-scoped and stable: if “Sarah Chen” appears in five different memories, she is always Jordan in that session. The LLM can reason about her as a single consistent person across all memories.

Two backends

Presidio supports different NLP backends. VaultMem exposes both:

# Lightweight — ~12 MB spaCy model, no PyTorch dependency
# Works on Streamlit Cloud, CI, anywhere torch is unavailable
san = Sanitizer(backend="spacy")

# High-accuracy — ~400 MB transformers model, requires PyTorch
# Best for local deployments or privacy-critical production use
san = Sanitizer(backend="transformers")

The spaCy backend (en_core_web_sm) is the default. It handles the common cases — person names, organisations, locations, emails, phone numbers — and is small enough to deploy anywhere without a GPU or a multi-gigabyte dependency install.

# spaCy (default)
pip install 'vaultmem[spacy]'
python -m spacy download en_core_web_sm

# transformers (high accuracy)
pip install 'vaultmem[presidio]'

What gets skipped

Not everything Presidio detects is PII. Temporal expressions (DATE_TIME), cardinal numbers, percentages, monetary values, and quantities are identifying in context but not on their own — and redacting them breaks the coherence of the context significantly.

VaultMem v0.3.2 introduced a skip list:

_SKIP_ENTITY_TYPES = frozenset({
    "DATE_TIME", "CARDINAL", "ORDINAL",
    "PERCENT", "MONEY", "QUANTITY", "LANGUAGE",
})

Before this, a memory like “Every Friday at 4pm I do a weekly review — takes 20 minutes” was returned to the LLM as “Every [DATE_TIME_3] at [DATE_TIME_4] I do a weekly review — takes [DATE_TIME_5]”. Correct in theory, unreadable in practice. The skip list fixes this.

The same release fixed two spaCy-specific label mismatches: spaCy reports organisations as ORGANIZATION (not ORG) and geopolitical entities as GPE (not LOCATION). Before the fix these fell through to typed tokens instead of receiving pseudonyms from the pools.

A subtle bug: case-insensitive deduplication

During testing I hit a bug that’s easy to miss until you actually run the system with real queries.

The memory stored "Sarah Chen" (title case). The user asked "who is sarah chen?" (lowercase). Presidio detected "sarah chen" as a PERSON — but the forward map looked it up with an exact string match. "sarah chen" != "Sarah Chen", so it was treated as a new entity and assigned a fresh pseudonym: Casey.

The LLM then received context about Jordan but a query about Casey. Its response: “There is no information about Casey.” Technically accurate. Completely useless.

The fix in v0.3.3: all forward-map keys are normalised to lowercase. A separate canonical map preserves the first-seen capitalisation so restoration returns "Sarah Chen", not "sarah chen". The variant seen first wins on casing; all subsequent variants map to the same pseudonym.

def _assign(self, real: str, entity_type: str) -> str:
    key = real.lower()
    if key in self._forward:
        return self._forward[key]          # "sarah chen" hits "Sarah Chen"'s entry
    self._canonical[key] = real            # first-seen casing for restoration
    ...

What this doesn’t solve

The Sanitizer addresses what leaves your device. It doesn’t address what the LLM infers or reconstructs. If your memory says “I grew up two blocks from the Eiffel Tower”, stripping names doesn’t prevent the LLM from inferring something about where you’re from.

The complete solution is TEE-based inference — running the model inside a trusted execution environment so the provider’s infrastructure never sees plaintext at all. That’s the direction the research is heading (see the arXiv paper above), but it’s not production-ready for general-purpose models yet.

PII stripping is the practical near-term answer. It’s not perfect, but it meaningfully shrinks the surface area of personal data your LLM provider sees on every query.

Changelog

v0.3.0 — Sanitizer class, transformers backend, pseudonym pools, restoration map
v0.3.1 — spaCy backend (en_core_web_sm), new [spacy] optional extra, backend= parameter
v0.3.2 — DATE_TIME skip list, ORGANIZATION/GPE label fixes for spaCy
v0.3.3 — case-insensitive entity deduplication

pip install 'vaultmem[spacy]' gets you the latest.

If you’re building an AI agent that touches personal data, I’d love to hear what you’re working on.

For short definitions of PII sanitization, zero-knowledge storage and the other terms in this series, see the agent memory glossary.

VaultMem Part 2 — Multi-Modal Memory, Temporal Search, and Pluggable Backends

2026-03-28T00:00:00+00:00

Part 1 covered why I built VaultMem, how the encryption works, and the core memory model. This post covers what came next in v0.2.0 and v0.2.1 — the parts that took the library from a solid foundation to something that can handle real-world agent use cases: multi-modal memory ingestion, time-aware retrieval, production-grade storage backends, O(log N) ANN search, and a pluggable query normalizer.

Install: pip install vaultmem · Code: github.com/aag1091-alt/vaultmem-sdk · Live demo: vaultmem-demo.streamlit.app

Text-only memory is a significant limitation for a personal AI agent. The moments that matter most — a photo from a trip, a voice note recorded mid-commute, a scanned medical document — aren’t text.

VaultMem v0.2.0 added add_media(), which ingests an image, audio file, PDF, or video and stores it as a fully encrypted memory atom. The raw bytes go into vault/media/{uuid}.enc (AES-256-GCM, same key architecture as text atoms). The extracted content becomes the searchable atom.

with VaultSession.open("./vault", passphrase, embedder=embedder) as s:
    # Image — extracts EXIF date, GPS coordinates, optional OCR
    atom = s.add_media("holiday_photo.jpg", passphrase)
    print(atom.content)      # "Photo taken 2023-07-14 in Split, Croatia"
    print(atom.captured_at)  # EXIF timestamp as Unix int
    print(atom.location)     # {"lat": 43.5, "lon": 16.4, "place": "Split"}

    # Audio — Whisper transcription
    atom = s.add_media("voice_note.mp3", passphrase)

    # PDF — PyMuPDF text extraction
    atom = s.add_media("contract.pdf", passphrase)

    # Retrieve raw bytes later (decrypted on demand, not in search results)
    raw = s.get_media(atom.id, passphrase)

The media bytes are never loaded into RAM during search — only the extracted text atom participates in embedding and retrieval. This keeps the privacy model intact: even if you extract metadata, the raw file stays encrypted behind the passphrase.

Each extractor is a pluggable Protocol, so adding new file types is straightforward:

from vaultmem import MediaExtractor, MediaExtractionResult, MediaIngester

class HeicExtractor:
    MIME_TYPES = {"image/heic", "image/heif"}

    def extract(self, path, mime_type):
        return MediaExtractionResult(
            content_type=mime_type,
            transcript="...",
            captured_at=...,
        )

ingester = MediaIngester(extra_extractors=[HeicExtractor()])

Install the extras to activate the built-in extractors:

pip install "vaultmem[media]"   # Pillow, piexif, openai-whisper, PyMuPDF, ffmpeg-python

Temporal search

The second gap I wanted to close: time. A personal agent should be able to answer “what was I doing in 2019?” or “show me everything from last summer” — not just “what’s most semantically similar to this query?”

This required two new timestamp fields on MemoryObject:

created_at — when the atom was written to the vault (set automatically, immutable)
captured_at — when the real-world event happened (set manually or from EXIF/metadata)

These are deliberately separate. A photo taken in 2019 but added to the vault today has captured_at in 2019 and created_at today. The distinction matters for any use case involving retroactive journaling or importing old media.

Browse by time window

from datetime import datetime, timezone

def ts(year, month=1, day=1):
    return int(datetime(year, month, day, tzinfo=timezone.utc).timestamp())

with VaultSession.open("./vault", passphrase, embedder=embedder) as s:
    memories_2019 = s.search_by_time(ts(2019), ts(2020))
    q3_2021 = s.search_by_time(ts(2021, 7), ts(2021, 10))

    # What was written to the vault in the last 24 hours
    import time
    recent = s.diff(int(time.time()) - 86400, int(time.time()))

Natural-language time queries

I wanted to avoid the experience of asking “what happened summer 2019?” and getting results from all time. The TimeQueryParser extracts a timestamp range from the query string, strips the time phrase, and passes the remainder to the embedding — all in pure Python with no dependencies:

from vaultmem import TimeQueryParser

from_ts, to_ts, remainder = TimeQueryParser.parse("what happened summer 2019")
# from_ts  → 2019-06-01 UTC
# to_ts    → 2019-09-01 UTC
# remainder → "what happened"

Supported phrases: season + year, month + year, bare year, “last year”, “this month”, “last N days”, “yesterday”.

Pass parse_time=True to search() and it handles everything automatically:

# Strips "in 2019" → timestamp range; embeds "what did I do"
results = s.search("what did I do in 2019", top_k=10, parse_time=True)

Diagram: How VaultMem splits a natural-language time query (see it in the post)
The query "what happened summer 2019" goes to TimeQueryParser.parse, which runs in pure Python with no dependencies. It splits the query in two. The time phrase, summer 2019, becomes a timestamp range from 2019-06-01 to 2019-09-01 UTC. The remainder, what happened, is what gets embedded. search with parse_time=True uses both, so the results come from summer 2019 instead of from all time.
The time phrase becomes a timestamp range; only what's left of the query is embedded.

Pluggable storage backends

The default storage model — two files on the local filesystem — is the right starting point. But a production agent deployment might need S3 for durability, Postgres for searchability, or both. I didn’t want to bake those in as optional flags on the session object; I wanted them to be proper extension points.

v0.2.0 introduced three ABCs:

ABC	Default	Production
`BlobStore`	`FileBlobStore`	`S3BlobStore`
`SearchIndex`	`SQLiteSearchIndex`	`PostgresSearchIndex`
`VectorIndex`	— (exact O(N) cosine)	`HNSWVectorIndex`

All three are injected at session creation:

from vaultmem import (
    VaultSession, LocalEmbedder,
    S3BlobStore, PostgresSearchIndex, HNSWVectorIndex,
)

with VaultSession.open(
    "./vault", passphrase,
    embedder=LocalEmbedder(),
    blob_store=S3BlobStore(bucket="my-vault-atoms"),
    search_index=PostgresSearchIndex("postgresql://user:pass@host/db"),
    vector_index=HNSWVectorIndex(),
) as s:
    results = s.search("coffee habits", top_k=10)

The important design decision: the vault format stays the same regardless of which backends you use. FileBlobStore and S3BlobStore store the same encrypted bytes. SQLiteSearchIndex and PostgresSearchIndex store the same metadata. Migrating between backends is O(N) byte-copies, no decryption required:

from vaultmem import migrate_vault

migrate_vault(
    src_blob_store=FileBlobStore("./old"),
    dst_blob_store=S3BlobStore(bucket="new-vault"),
    passphrase=passphrase,
)

HNSW and the O(N) problem

At small scale, exact cosine search over all atoms in RAM is fine. At 10,000 atoms it becomes the bottleneck — the benchmark showed ~148ms per query, all of it in the embedding comparison loop.

HNSWVectorIndex wraps hnswlib and plugs into the same VectorIndex ABC. The HNSW graph is built incrementally as atoms are added and serialized encrypted to vault/vector_index.hnsw.enc on flush. At session open it’s loaded once into RAM. Search then runs in O(log N):

Atoms	Exact cosine	HNSW
1,000	~15ms	~0.3ms
10,000	~148ms	~0.8ms
100,000	~1,500ms	~1.2ms

Below 1,000 atoms, the implementation falls back to exact cosine automatically — the HNSW graph has overhead that isn’t worth paying at small N. (For how the HNSW graph itself works, see the Paper TL;DR on HNSW.)

pip install "vaultmem[ann]"   # hnswlib

Query normalizer

A subtler problem surfaced while building the interactive demo: the question framing users naturally type — “What do I know about Sarah Chen?” — dilutes the embedding. A bag-of-words or hash-projection embedder assigns equal weight to “what”, “do”, “i”, “know”, “about” and “Sarah”, “Chen”. The meaningful terms are swamped by question scaffolding.

Sentence-transformers handle this natively through attention — they learn that “Sarah Chen” carries the semantic weight in the sentence. But not every deployment uses a 22M-parameter ML model.

v0.2.1 introduced a QueryNormalizer Protocol:

from vaultmem import QueryNormalizer  # Protocol — implement normalize(text) -> str

The built-in RegexQueryNormalizer strips common English question preamble:

from vaultmem import RegexQueryNormalizer

# "What do I know about Sarah Chen?" → "Sarah Chen"
# "How does Flash Attention work?"   → "Flash Attention work"
# "Am I vegetarian?"                 → "vegetarian"

More interesting is the custom path — since it’s a Protocol, any object with a normalize(self, text: str) -> str method works. The interactive demo uses a Groq-backed normalizer with a server-side API key, so users get LLM-quality query extraction without needing their own key:

class GroqQueryNormalizer:
    def __init__(self, api_key: str) -> None:
        from groq import Groq
        self._client = Groq(api_key=api_key)

    def normalize(self, text: str) -> str:
        resp = self._client.chat.completions.create(
            model="llama-3.3-70b-versatile",
            max_tokens=20,
            messages=[
                {"role": "system",
                 "content": "Extract key search terms. Return only keywords."},
                {"role": "user", "content": text},
            ],
        )
        return resp.choices[0].message.content.strip() or text

The normalizer is injected once at session creation and runs on all searches where normalize_query=True:

with VaultSession.open("./vault", passphrase,
                        query_normalizer=GroqQueryNormalizer(api_key),
                        embedder=embedder) as s:
    results = s.search("What do I know about Sarah?", normalize_query=True)

Diagram: How VaultMem's query normalizer strips question framing before search (see it in the post)
The question "What do I know about Sarah Chen?" goes to a QueryNormalizer, any object with normalize(text) returning a string. Two kinds plug in: the built-in RegexQueryNormalizer, which strips common English question preamble, or your own class, such as the demo's Groq-backed normalizer that asks an LLM to return only keywords. Either way the result is "Sarah Chen"; the question scaffolding "What do I know about" is dropped. search with normalize_query=True then embeds only those key terms.
The normalizer runs before embedding, so the question's scaffolding words don't drown out the names that matter.

Data classes

One design detail worth calling out: memory doesn’t all decay at the same rate, and not all memory deserves the same cryptographic protection.

VaultMem has three data classes, set at vault creation:

Class	Use case	Argon2id memory	Decay λ	Half-life
`GENERAL`	Standard	64 MiB	0.005	~5 months
`MEDICAL`	Health data	128 MiB + mlock	0.002	~12 months
`ARCHIVAL`	Long-term records	64 MiB	0.0007	~3 years

MEDICAL doubles the Argon2id memory cost, making brute-force twice as expensive. ARCHIVAL slows the significance decay so memories stay surfaced for years rather than months — appropriate for journals, legal records, or anything you want to last.

VaultSession.create("./health_vault", passphrase, owner="alice", data_class="MEDICAL")

The LoCoMo benchmark

I wanted a more rigorous evaluation than the 18-atom synthetic benchmark I described in Part 1. The LoCoMo dataset is a collection of long-form conversational memory — 1,982 QA pairs across five categories (single-hop factual, multi-hop reasoning, temporal, adversarial, open-domain). It’s the closest thing to a realistic test of a personal memory system. (For how the benchmark was built, see the LoCoMo Paper TL;DR.)

VaultMem’s Recall@10 and MRR scores on a random sample:

Category	Recall@10	MRR
Single-hop factual	0.91	0.87
Temporal	0.84	0.79
Multi-hop	0.73	0.68
Adversarial	0.78	0.71

Multi-hop is the hardest — it requires reasoning across atoms that weren’t individually asked about, which is a retrieval problem before it’s even an LLM problem.

The interactive demo

The fastest way to see all of this in action is the live demo. It runs a pre-seeded vault with 36 memories across all four types — enough variety to demonstrate the retrieval model without you having to type anything first.

Every retrieved result shows its relevancy score, memory type, and retrieval tier. The right panel shows the raw encrypted bytes of current.vmem and current.atoms — what the platform operator actually sees. Each browser tab gets an isolated vault; your data can’t be seen by or contaminated by other visitors.

Some things worth trying:

Question	What it exercises
What do I know about Sarah Chen?	Groq query normalizer → EPISODIC retrieval
How does Flash Attention work?	SEMANTIC retrieval
What’s my coffee order?	PERSONA retrieval
How do I debug problems?	PROCEDURAL retrieval
Tell me about my cat	Natural language → EPISODIC

What’s next

The library is at v0.2.1. The things I’m thinking about for v0.3.0:

Composite and Affinity generation — right now the tiers exist in the retrieval model but atoms are only created at ATOM granularity. Automated COMPOSITE and AFFINITY synthesis from accumulated atoms is the next big addition.
Cross-agent memory sharing — Layer 2 of the Memory-as-Asset framework. A shared encrypted memory space where multiple agents can read without the platform being able to.
LoCoMo full benchmark — running the complete 1,982-pair dataset rather than a sample.

The paper targeting ACM CCS 2026 is in progress.

VaultMem Part 3 — Hiding PII from Your LLM Provider →

pip install vaultmem · GitHub · Preprint · Live demo

VaultMem — Building Encrypted Memory for AI Agents

2026-03-21T00:00:00+00:00

While building Rena (code) — a voice AI health agent I submitted for the Google Gemini Live Agent Challenge — I needed persistent memory. Something that could build a user profile from conversations across sessions and recall it the next time you speak to the agent. Libraries like mem0 or Zep would have worked fine for that use case.

But the more I thought about it, the more a different question started bothering me: what if I wanted a truly private agent — one where the platform running it genuinely cannot read my conversations? And beyond that, how do you store raw conversations as structured memory in the first place?

I couldn’t find anything that answered both. So I spent a few days building it: a memory library for AI agents where the platform developer cryptographically cannot read what the user stores.

Install: pip install vaultmem · Code: github.com/aag1091-alt/vaultmem-sdk · Preprint: doi.org/10.5281/zenodo.19154079

The problem

Every AI memory library I looked at — mem0, Zep, LangMem, Letta — stores user memories in plaintext on the platform’s servers. The platform operator can read everything their users have ever told the agent. This isn’t a privacy policy failure; it’s a structural one. When the key lives with the operator, the guarantee lives with the operator.

This matters more than it sounds. The deepest use case for persistent AI memory isn’t task management — it’s candor. An agent that remembers across sessions only becomes genuinely useful when you can be completely honest with it. A person working through a difficult decision. Someone with early-stage dementia building a longitudinal record of their life. A user who just wants their preferences remembered without those preferences living in a corporate database.

For that to work, the privacy guarantee has to be mathematical, not a privacy policy.

Where the research led

While digging into how existing systems handle memory, I stumbled on a paper: Memory as Asset by Pan, Huang & Yang (2026) (plain-language TL;DR). It formalizes something I’d been circling around informally — that personal memory isn’t just data to be stored; it’s an asset that belongs to the user, with ownership, permissions, and portability as first-class concerns.

The paper defines a layered architecture. Layer 1 is personal storage: encrypted, user-owned, portable. Layer 2 is cross-session intelligence. Layer 3 is collaborative memory between agents. No existing library implements even Layer 1 with real cryptographic guarantees.

What the paper gave me more than anything was vocabulary. Before reading it, I was thinking vaguely about “storing conversations.” After, I was thinking about episodic memory, semantic memory, significance decay, and granularity hierarchies. That shift changed what I built — instead of a simple key-value store with embeddings, VaultMem ended up with a proper memory model grounded in the framework. The paper became the specification.

What VaultMem is

VaultMem is an embeddable Python library. A platform developer adds it as a dependency, and their users get encrypted, portable, private memory — without the developer being able to read it.

The analogy I keep coming back to: VaultMem is to AI memory what 1Password is to credentials. The library doesn’t know what it’s storing. The platform receives opaque ciphertext it can store, back up, and serve back — but cannot decrypt.

from vaultmem import VaultSession, OllamaEmbedder

embedder = OllamaEmbedder("http://localhost:11434")

# Create an encrypted vault
with VaultSession.create("./vault", passphrase="s3cr3t", owner="alice",
                          embedder=embedder) as s:
    s.add("I met Bob at the AI conference yesterday")
    s.add("I prefer concise bullet-point answers")
    s.add("My go-to language for data work is Python")

# Search across sessions
with VaultSession.open("./vault", passphrase="s3cr3t",
                        embedder=embedder) as s:
    results = s.search("Bob conference")
    for r in results:
        print(f"[{r.tier}] {r.score:.3f}  {r.atom.content}")

Output:

[ATOM] 0.847  I met Bob at the AI conference yesterday

How the encryption works

The key architecture has two layers:

User passphrase
  → Argon2id(passphrase, salt)      ← 64 MiB, 3 iterations
  → KEK (Key Encryption Key)         ← in RAM only; zeroed after open
  → AES-GCM-Dec(KEK, wrapped_MEK)
  → MEK (Master Encryption Key)      ← in RAM during session only
  → AES-256-GCM(memory atom)        ← to disk

The Master Encryption Key (MEK) is generated once at vault creation and stored only as a ciphertext wrapped with the user’s passphrase-derived key. It never touches disk in plaintext. When the session closes, it’s zeroed from RAM.

Diagram: VaultMem's two-layer key architecture (see it in the post)
The user passphrase goes through Argon2id with a salt (64 MiB, 3 iterations) to derive the KEK, the key encryption key. The KEK lives in RAM only and decrypts the wrapped MEK with AES-GCM. The MEK, the master encryption key, stays in RAM during the session and is zeroed when it closes. The MEK encrypts each memory atom with AES-256-GCM, with a 128-bit authentication tag and the atom's UUID bound as additional authenticated data. On disk there is only the salt with the wrapped MEK, and the encrypted memory atoms.
Only ciphertext ever reaches disk. Because the MEK is wrapped separately from the data, a passphrase change re-wraps one key instead of re-encrypting every atom.

A platform developer who integrates VaultMem receives encrypted blobs from their users. They can store them, back them up, sync them — but they cannot read them. The math makes this true regardless of what their privacy policy says.

Two things I’m particularly happy with:

Passphrase changes are O(1). Because the MEK is wrapped separately from the data, changing a passphrase means re-wrapping one key — not re-encrypting every memory atom. A vault with 10,000 atoms changes passphrase in milliseconds.

Per-atom tamper detection. Each atom is encrypted with AES-256-GCM, which includes a 128-bit authentication tag. Each atom also has its UUID bound as Additional Authenticated Data, so you can’t transplant a valid ciphertext from one slot to another. Any tampering is detected before the data surfaces.

The memory model

The Memory-as-Asset paper defines memory as a 5-tuple: content, context, owner, permissions, versioning state. VaultMem’s MemoryObject implements this directly.

Four memory types: EPISODIC (things that happened), SEMANTIC (facts), PERSONA (preferences and habits), PROCEDURAL (how-to knowledge). Type is assigned by a deterministic classifier — no LLM call, no network request.

The more interesting part is the granularity hierarchy:

ATOM — a single memory statement. Immutable once written.
COMPOSITE — a type-homogeneous grouping of related atoms.
AFFINITY — a derived atom representing a recurring pattern. “User drinks coffee every morning” synthesized from twenty individual coffee mentions.

AFFINITY atoms carry a significance score:

σ = (1 − e^(−κ × freq)) × e^(−λ × days_since_last)

Frequency drives the first factor (with diminishing returns), recency drives the second. The parameters vary by data class — medical memories decay slower than general ones.

Three-tier retrieval — and why it matters

The retrieval model searches AFFINITY → COMPOSITE → ATOM and re-ranks results globally. On factual queries this behaves identically to flat cosine search — no regression. On pattern queries it’s a different story.

I ran a benchmark to verify this. Three topic clusters (coffee habits, Python usage, exercise routine), each with one AFFINITY atom and five ATOM-granularity memories including a deliberately tricky decoy: an episodic memory containing the exact query keywords, designed to fool flat cosine.

For the query “my morning coffee habits”, flat cosine ranks the decoy atom first (it contains the words “coffee habits” and “morning”) and the AFFINITY second. Three-tier flips this — the AFFINITY’s significance score (~0.92 for a well-established habit) outweighs the cosine advantage of the decoy.

Diagram: VaultMem's three-tier retrieval (see it in the post)
A query is searched against three tiers in order: AFFINITY atoms, which are derived recurring patterns that carry a significance score; COMPOSITE groupings of related atoms; and individual ATOM statements. All results are then re-ranked globally using cosine similarity plus significance. For the query "my morning coffee habits", flat cosine ranks a decoy atom first and the AFFINITY second, while three-tier retrieval ranks the AFFINITY first.
The decoy repeats the query's words, so flat cosine puts it first. The AFFINITY's significance score lifts the real habit to the top; specific queries rank the same as flat cosine.

Metric	Flat cosine	Three-tier	Δ
MRR (pattern queries)	0.50	1.00	+0.50
MRR (specific queries)	1.00	1.00	0.00

Pattern queries: AFFINITY surfaced first every time. Specific queries: no regression. The significance weight isn’t a heuristic — it’s a correction for a systematic bias in pure cosine retrieval.

The `.vmem` format

Vaults are stored as two files:

my_vault/
├── current.vmem      # encrypted index (48-byte self-describing header)
└── current.atoms     # append-only encrypted atom blocks

The 48-byte header is the only plaintext in the file. It carries the magic bytes (VMEM), format version, encryption algorithm, KDF, and file type — enough for any conforming implementation to decrypt the vault given the passphrase. No prior coordination with the writing implementation needed.

Atoms are append-only and immutable. Checkpoints update only the index file, atomically via fsync + os.rename(). Crash-safe.

The format is designed to be the interoperability layer for a future where encrypted memory vaults move between applications and agents — the same way a PDF can be opened by any conforming reader regardless of which app produced it.

What’s in the library

About 2,250 lines of Python, 39 unit tests, three core dependencies (cryptography, argon2-cffi, numpy). Sentence-transformers or a local Ollama instance for embeddings — or bring your own.

# LocalEmbedder — no network, runs on CPU
from vaultmem import LocalEmbedder
embedder = LocalEmbedder()

# OllamaEmbedder — GPU-accelerated, localhost or Tailscale
from vaultmem import OllamaEmbedder
embedder = OllamaEmbedder("http://localhost:11434", model="all-minilm")

# Custom — implement .embed(text) -> list[float]
class MyEmbedder:
    def embed(self, text: str) -> list[float]: ...
    @property
    def dimension(self) -> int: return 384

The research angle

The work turned into a paper. VaultMem is the first implementation of Layer 1 of the Memory-as-Asset framework — the paper I stumbled on during the initial research that ended up shaping the entire design. The paper formalizes the threat model, scores ten existing AI memory systems against a four-dimension privacy taxonomy (no prior system achieves all four properties simultaneously), and specifies the .vmem format as a portable standard.

The preprint is live on Zenodo and I’m targeting ACM CCS 2026 for the full paper.

Try it

pip install vaultmem                   # core
pip install "vaultmem[local]"          # + local embeddings (sentence-transformers)

The library is intentionally small — the code is the specification. If you’re building an AI agent and want to give your users memory that’s actually theirs, this is the starting point.

github.com/aag1091-alt/vaultmem-sdk

Part 2 covers what was built on top of this foundation in v0.2.0 and v0.2.1: multi-modal memory ingestion, temporal search, pluggable storage backends, HNSW vector indexing (how HNSW works), and a query normalizer Protocol — including a live interactive demo.

VaultMem Part 2 — Multi-Modal, Temporal Search, and Pluggable Backends →

New to terms like zero-knowledge storage, client-side encryption or episodic memory? The agent memory glossary explains them in plain language. For the non-technical case behind all of this, see Your AI Agent Shouldn’t Be Able to Read Your Diary.