Study: 8-bit KV cache near-lossless, 4-bit measurably worse

A community study tested 35 weight quantizations across llama.cpp, ExLlamaV3 and vLLM with 16-, 8- and 4-bit KV caches, using KL divergence to measure output drift. 8-bit caches roughly halve memory with essentially no loss, while 4-bit caches cause measurable drift that grows with context length.

• Four RTX 3060 Ti GPUs Hit 120 t/s Local Inference With 262K Context2026-09-27
• Eagle-4 Released: Faster than DFlash2 & DSpark2026-10-21
• User builds 256GB VRAM rig with 8 Radeon Pro V620s2026-10-08
• Developer releases Ninfer 4080 for 16GB GPUs with 100k context2026-10-04
• Swiss-South African lab open-sources 27B creative-writing model Hemmingway-12026-09-21