- Author
- Alex Wang
- Published
- 15 October 2026
- Kind
- Personal blog post with derivations, simulations and interactive widgets
- Original
- alexwang.ai/posts/associative-memory/
- Subject
- Language models and training
- Themes
- Getting good, What lasts
What it is
Alex Wang writes an explainer on transformers as memory systems. It starts from an outer-product key-value store. It adds fixes that reduce interference until MLP and attention layers appear.
What it says
- A plain outer-product memory fails once memories outnumber the key dimension, 64 here.
- A least-squares linear store doubles that capacity, to about 128 memories.
- A nonlinearity on the similarities gives an MLP. Squared ReLU reaches about 2,700 memories.
- Softmax attention follows from the exponential, a sum of all powers. At sharpness 8, capacity is about 1.2 million.
- Attention builds memories from context tokens, so its store grows with context length.
- Scaling sharpness with the log of context length allows essentially unbounded memories.
- The post says the original transformer paper drew this link in versions 2 and 3.
What evidence it gives
The post gives algebraic derivations and simulations with random unit keys and random values. A summary table lists the capacity of each store at key dimension 64. Interactive widgets let readers vary the settings.
Interests and limits
This is a personal blog post. Capacity counts recall of random vectors before interference equals signal. The post says it ignores training data, which shapes real keys and values. Results come from simulations at key dimension 64.