MassAlloc Attention

CoWindow and MassAlloc Attention Speed Up Long-Context Models

Authors present CoWindow Attention, which spreads distant context across KV heads via complementary windows without a learned router, and MassAlloc Attention, which uses softmax statistics to skip low-contribution post-score compute. At 128K context on 8 H100s, attention-operator speeds reach 7.4x/8.6x/3.0x for forward/backward/decode, and 14B models at 32K cut training FLOPs by 28.5% and 23.1% with comparable capability to FullAttn.