Paper: Yanqi Pan, Qinghao Huang & Weihao Yang, Memory as Asset: From Agent-centric to Human-centric Memory Management (arXiv, March 2026)

One-line TL;DR: the memory an AI keeps about you is designed today as the agent’s internal resource; this paper argues it should be your asset — owned by you, shared only under your permissions, and able to grow over time.

This is the paper that reshaped VaultMem. I found it while I was stuck on a vague feeling that something about agent memory was backwards, and it gave that feeling a name.

The problem: agent-centric memory

Most memory systems treat what an agent remembers about you as part of the agent — stored by the platform, shaped for the platform’s use, and stuck there. The authors call this agent-centric memory. The consequences are familiar: you can’t take your history with you, you can’t decide who sees it, and every product rebuilds its own isolated “memory island” about the same person.

The proposal: memory as a personal asset

They propose flipping the default, built on three principles:

  • Memory in hand — individuals own and control their memories, not centralized platforms.
  • Memory groups — memories can be shared and combined, but only through permissions the owner sets.
  • Collective memory evolution — memory isn’t frozen: it’s recorded, consolidated and abstracted into higher-level knowledge over time, and can be exchanged between groups under control.

Each memory object carries its content, context, an explicit owner, an access policy and a version. Ownership and permissions are part of the data itself, not an afterthought in some admin panel.

The three layers

  1. Personal storage — fast storage of each person’s private memory objects.
  2. Evolution — updates memory from interactions: recording experiences, merging related memories, abstracting knowledge. It can run locally for privacy or remotely for compute.
  3. Memory exchange network — a decentralized way to discover, authenticate and exchange memory assets.

Notably, the paper doesn’t insist everything stays on your device. If a provider hosts your memory, it should be encrypted so only you can read it.

How it shaped what I built

That last point is where VaultMem lives. I treat it as an implementation of the first layer: the platform stores your memories, but they’re encrypted with a key only you hold, so the operator cryptographically can’t read them. Everything in the VaultMem series — client-side encryption, encrypted search, keeping PII away from the model provider — follows from taking “memory in hand” literally.

Caveats worth knowing

  • The authors are upfront that this is an early draft: the terms and design are still evolving.
  • It’s a conceptual and architectural paper — there are no benchmarks or experiments yet. The claim that existing systems meet at most two of the three principles is an argument, not a measurement.
  • The exchange network raises hard questions (trust, incentives, revocation) that the paper sketches rather than solves.

I still think it’s the right question to ask before building any memory feature: whose memory is this?

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems and LoCoMo.