CoWindow and MassAlloc Attention Speed Up Long-Context Models

Authors present CoWindow Attention, which spreads distant context across KV heads via complementary windows without a learned router, and MassAlloc Attention, which uses softmax statistics to skip low-contribution post-score compute. At 128K context on 8 H100s, attention-operator speeds reach 7.4x/8.6x/3.0x for forward/backward/decode, and 14B models at 32K cut training FLOPs by 28.5% and 23.1% with comparable capability to FullAttn.

• Ai2 replaces priority scheduler with GPU time budgets2026-10-09
• 21M model with 6.4B lookup table matches 114M dense model, table on SSD2026-10-07
• Jeeves: Reasoning Improves Jev-Like Decision Models2026-09-29
• Economics of Open-Weight Inference: Older GPUs Retain Value2026-09-22
• SupraLabs Open-Sources 100M Text-to-Image Model Supra2-IMG2026-09-22