Issue 2026-11-09 · Industry · 研究 · 开源

Researcher Proposes GAP: Merged Attention and MLP Transformer Block

A researcher introduced GAP (Gated Attention Projection), merging attention and MLP into a single gated block. Tested on WikiText-103, GAP20 used 19.91% fewer parameters than a standard Transformer but had slightly lower language-model accuracy; torch.compile improved training throughput by about 19.9%.

r/MachineLearning14 d ago
Read original ↗