LittleBit-2

LittleBit Achieves Sub-1-Bit LLM Compression

LittleBit (NeurIPS 2025) factorizes weight matrices into low-rank latent factors, binarizes them, and restores magnitude via lightweight learned scales, enabling 1.0 to 0.1 bits per weight while keeping the original inference architecture. The follow-up LittleBit-2 (ICML 2026) adds Joint-ITQ initialization, improving initialization without adding inference overhead.