Researcher Proposes GAP: Merged Attention and MLP Transformer Block
A researcher introduced GAP (Gated Attention Projection), merging attention and MLP into a single gated block. Tested on WikiText-103, GAP20 used 19.91% fewer parameters than a standard Transformer but had slightly lower language-model accuracy; torch.compile improved training throughput by about 19.9%.