Understanding Transformer Capacity Through Memory

The post views linear layers, MLPs, and self-attention as memory stores, using interference-to-signal ratio to measure capacity: with 64-dimensional keys, capacity jumps from about 64 memories to about 2,700, then to over a million.

• EngramEdit Enables Decoupled LLM Knowledge Updates2026-10-09
• Have URMs and UTs been adopted by frontier models?2026-10-09
• 825k‑parameter model generates RP2040‑executable drawing programs2026-09-13