Paper: Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica & Joseph E. Gonzalez (UC Berkeley), MemGPT: Towards LLMs as Operating Systems (arXiv, 2023)

One-line TL;DR: Instead of waiting for bigger context windows, MemGPT treats the prompt like a computer’s RAM and lets the model itself move information between that small fast space and larger external storage, using function calls.

The problem

An LLM can only “see” what fits in its context window. Long chats and big document collections don’t fit, so something has to decide what stays in view. Operating systems solved a version of this decades ago: programs act as if they have lots of memory, while the OS quietly pages data between fast RAM and slower disk. The paper calls its version virtual context management.

How it works

MemGPT splits memory into two places.

Main context — what’s actually in the prompt — has three parts:

  • System instructions: read-only rules, including how to use the memory functions.
  • Working context: a small, fixed-size scratchpad of key facts (for example, things learned about the user). The model can only change it through function calls.
  • A FIFO message queue: the recent conversation. Its first entry is a running summary of older messages that were pushed out.

External context lives outside the prompt:

  • Recall storage: the full message history, searchable on demand.
  • Archival storage: a store for arbitrary text the model chooses to save or look up later.
MemGPT's two tiers of memory Main context is what is in the prompt: read-only system instructions, a small working context of key facts edited through function calls, and a FIFO message queue that starts with a summary of older messages. External context sits outside the prompt: recall storage holds the full message history and archival storage holds any text the model chooses to save. The oldest messages are evicted from the queue into recall storage, and the model searches and saves across both tiers with function calls. Main context · in the prompt System instructions read-only rules, incl. the memory functions Working context small scratchpad of key facts FIFO message queue running summary + recent messages oldest messages evicted, not deleted function calls: search, save, edit Recall storage full message history, searchable Archival storage any text the model chooses to keep External context · outside the prompt
Main context is the prompt; external context is everything the model can page back in. The model itself moves information between them.

The clever part is who moves things around: the model does. Its output is parsed as function calls (search recall, write to archival, edit working context), and the results are fed back in. A few mechanics make this work:

  • Memory pressure warnings. When the prompt gets close to full (the paper’s example uses 70%), the system adds a warning so the model can save what matters before it’s lost.
  • Eviction with summaries. When the prompt is full, a chunk of the oldest messages is evicted and folded into the running summary. Evicted messages aren’t deleted — they stay in recall storage.
  • Heartbeats. A function call can ask to keep going (request_heartbeat), so the model can chain several lookups before replying.
  • Events. User messages, system alerts and scheduled timers all wake the agent up, so it doesn’t only act when a person types.

What they tested

Two settings, each with a “can it really remember?” task:

  • Multi-session chat. In a deep memory retrieval task, the agent is asked about something from an earlier session. MemGPT answered far more of these correctly than the same models with a fixed context. In a “conversation opener” task, its openers drew on what it knew about the person, and the authors found that saving facts to working context was key.
  • Document analysis. For question answering over many documents, MemGPT could page through more search results than fit in one prompt. In a nested key-value lookup task, where one value points to another key, fixed-context models broke down as nesting got deeper, while MemGPT with GPT-4 kept answering by looking things up repeatedly.

Why it matters for builders

  1. Memory can be a set of tools, not just a bigger prompt. Giving the model explicit read/write/search functions makes memory something you can inspect and log.
  2. Separate “always in view” from “look it up.” A small working context of key facts plus searchable history is a pattern that shows up again and again in agent memory designs.
  3. Plan for overflow. Warning before the window fills, and summarizing instead of silently truncating, is a simple idea worth copying.

Honest caveats

  • It leans on the model’s function-calling skill. The authors report that results dropped noticeably with GPT-3.5, which was weaker at calling functions.
  • Retrieval is still the bottleneck. With embedding search, the right document often wasn’t near the top, and the agent tended to stop paging before exhausting the results.
  • More model isn’t always better. On the nested lookup task, the GPT-4 Turbo version did worse than GPT-4, and some versions gave up after too few lookups.
  • Self-managed memory is a judgment call by the model. What gets saved, summarized or forgotten depends on the model’s choices, which is harder to predict than a fixed rule.

Earlier in the series: LoCoMo (a benchmark for exactly this kind of long-term memory), Sentence-BERT (the embeddings behind most semantic search), and Complementary Learning Systems (a brain-inspired split between fast and slow memory).

Terms like context window, episodic memory and top-k retrieval are defined in the agent memory glossary.