Paper: Ramsauer, Schäfl, Lehner et al., Hopfield Networks is All You Need (2020)

One-line TL;DR: a continuous version of the 1982 Hopfield network can store exponentially many memories, usually retrieves one in a single step — and that single step is exactly the attention operation inside every transformer.

This is the paper that made me stop thinking of Hopfield networks as history. Here’s the idea in plain language, plus a tiny demo you can run.

The old idea: memories as valleys

In Hopfield’s 1982 network, you store patterns (say, strings of +1 and −1) in the connection weights. The network has an “energy,” and each stored pattern sits at the bottom of a valley. Start from a corrupted pattern, let the network update, and it rolls downhill into the nearest stored memory. That’s content-addressable memory: you find a memory by giving a piece of it, not by knowing its address.

The catch: the classic network only holds about 0.14 patterns per neuron. Store more than that and the valleys blur together into spurious memories.

The new idea: sharpen the valleys

Ramsauer et al. swap the old quadratic energy for one built on log-sum-exp — a smooth version of “max.” Intuitively, each stored pattern now carves a much sharper, deeper valley, so many more of them fit without interfering. The paper shows the capacity grows exponentially with the dimension of the patterns, and that retrieval typically converges after one update.

That one update is:

new_state = X · softmax(β · Xᵀ · state)

where the columns of X are the stored patterns and β controls how sharp the valleys are.

Wait — that’s attention

Now look at transformer attention:

output = V · softmax(Kᵀ · q / √d)

Set the query q to the current state, use the stored patterns as both keys K and values V, and pick β = 1/√d. It’s the same equation. Every attention head in an LLM is doing a one-step associative memory lookup: compare the query against everything stored, softmax the similarities, return a weighted blend that’s usually dominated by the best match.

The paper goes further, showing how different heads and layers behave like different retrieval regimes — some settle on one memory, some average over many — but the core equivalence is the part that stuck with me.

Twenty lines of NumPy

Store five random ±1 patterns, flip 40% of the bits in one of them, and do a single update:

import numpy as np

rng = np.random.default_rng(0)
d, n = 64, 5
X = rng.choice([-1.0, 1.0], size=(d, n))   # stored memories (columns)

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

def retrieve(xi, beta=0.5):
    return X @ softmax(beta * X.T @ xi)      # one modern-Hopfield update

target = X[:, 2]
noisy = target.copy()
noisy[rng.choice(d, size=int(0.4 * d), replace=False)] *= -1

out = retrieve(noisy)
print("overlap before:", int(noisy @ target), "/", d)       # 14 / 64
print("overlap after: ", int(np.sign(out) @ target), "/", d) # 64 / 64

One step takes a pattern that’s only loosely related to the target back to a perfect copy. The softmax weights tell the story: about 0.97 on the right memory and crumbs on the rest — exactly what an attention map looks like.

Why I care, as someone building agent memory

Most agent memory today is “embed everything, return the top-k nearest neighbors.” This paper reframes that: nearest-neighbor search is a crude, non-differentiable cousin of what attention already does well. It nudged me to build Attractor around retrieval by partial cues and context, and to think of memory as something that settles rather than something you look up.

Caveats worth knowing

  • “Exponential capacity” is a statement about well-separated patterns in high dimensions; real embeddings are messy and correlated.
  • The one-step convergence holds when β is large enough relative to how similar the stored patterns are — similar memories still blur together.
  • Treating attention as memory is a lens, not a full explanation of what transformers learn.

If you read only one modern paper about associative memory, make it this one. The rest of what I’m reading is on my papershelf, and Part 1 of this series covers the other papers behind my memory projects.