The Transformer is a neural network architecture built on self-attention and the basis of most current large language models. Recent coverage looks at its capacity and how it can be modified: one analysis treats linear layers, MLPs and self-attention as memory stores and measures capacity via an interference-to-signal ratio, rising from about 64 memories to about 2,700 and then over a million with 64-dimensional keys. EngramEdit edits shared n-gram embeddings without touching the backbone, and a discussion asks whether Universal Transformer variants have been adopted by frontier labs.
Understanding Transformer Capacity Through MemoryThe post views linear layers, MLPs, and self-attention as memory stores, using interference-to-signal ratio to measure capacity: with 64-dimensional keys, capacity jumps from about 64 memories to about 2,700, then to over a million.
EngramEdit Enables Decoupled LLM Knowledge UpdatesThe EngramEdit paper jointly updates shared n-gram embeddings in conditional-memory architectures, enabling factual knowledge edits without touching the Transformer backbone. It achieves near-perfect edit success and about 3x the strongest baseline's chain-of-thought accuracy on multi-hop reasoning.
Have URMs and UTs been adopted by frontier models?A Reddit discussion questions whether frontier labs at big tech have adopted Universal Reasoning Model (URM) and Universal Transformer (UT) enhancements, as UT-based small models trained from scratch outperform standard LLMs on certain tasks by a significant margin.
825k‑parameter model generates RP2040‑executable drawing programsA researcher trained an 825k‑parameter autoregressive transformer to emit ~100‑byte drawing bytecode that a fixed‑point VM executes exactly on a Raspberry Pi Pico (RP2040). All 12,670 generated traces matched the Python reference VM; the interpreter uses 1,862 bytes of flash and runs without floating‑point or tensor runtimes.
The A.I. Daily