Paper: Nils Reimers & Iryna Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)

One-line TL;DR: fine-tune BERT so that every sentence gets its own fixed-size embedding you can compare with cosine similarity — turning semantic search over thousands of sentences from hours into seconds.

It’s the least flashy paper in this series and the one my code depends on most. Every time Attractor, VaultMem or daylog “searches by meaning,” it’s using a model descended from this paper.

The problem: BERT is a slow judge

In 2019, BERT was the best tool for deciding whether two sentences meant the same thing — but only if you fed it both sentences together. That works for scoring one pair. It falls apart for search.

The paper’s example: to find the most similar pair in a collection of 10,000 sentences, BERT would need around 50 million pairwise passes — roughly 65 hours on a GPU. And you can’t precompute anything, because every score depends on both sentences at once.

The obvious shortcut — run each sentence through BERT alone and average its outputs (or take the special [CLS] token) — produced embeddings that were often worse than old-fashioned averaged word vectors.

The fix: train it to embed

Sentence-BERT puts two copies of the same BERT side by side (a siamese network — shared weights), runs one sentence through each, pools each output into a single vector, and trains the whole thing so that:

  • sentences that mean similar things land close together, and
  • sentences that don’t land far apart.

They trained on natural-language-inference data (pairs labeled entailment / contradiction / neutral) and on similarity-scored pairs. Afterward you throw away the “pair” setup: each sentence goes through the model once, you store its vector, and comparing two sentences is a cheap cosine similarity.

That same 10,000-sentence search drops to about 5 seconds.

Why it still matters

  • It made vector search practical. Embed once, store in an index (pgvector, Qdrant, HNSW), and query in milliseconds. That’s the foundation of nearly every RAG system and agent memory store.
  • The library outlived the paper. The sentence-transformers package that came out of this work is still how most people get embedding models running in a few lines.
  • Small models are often enough. The 384-dimension model I use in Attractor runs comfortably on a CPU — handy when the whole point is keeping data on hardware you own.

Caveats worth knowing

  • Bi-encoders like SBERT trade a little accuracy for speed. When precision matters, a common pattern is to retrieve with embeddings, then re-rank the top results with a slower cross-encoder.
  • “Similar meaning” is whatever the training data said it was. Domain-specific text (logs, configs, medical notes) often needs a domain-tuned model.
  • Embeddings find related text; they don’t reason. Multi-hop questions — the ones LoCoMo showed are hardest — need more than nearest neighbors.

Earlier in the series: six papers on AI memory, Hopfield Networks Is All You Need, Complementary Learning Systems, LoCoMo and Memory as Asset.