Qwen3.8-Flash-Next is an open-source large language model variant focused on long-context inference and local deployment. Recent coverage describes methods to offload most KV cache to RAM (3×3090 users reporting 1M context with ~80 tok/s short, ~60 tok/s long, and 3,701 tok/s at a 248k prefill), running on 12GB RTX 4070 with quantization and SSD offload to raise generation from ~6 to ~20 tok/s, and MLX-serve demos claiming 1M-context support on M5 Max with 8-bit KV cache and ~117GB peak memory.
Offload most of Qwen3.8 KV cache to RAMA vLLM user shared on r/LocalLLaMA a method to keep most of Qwen3.8-Flash-Next's KV cache in system RAM while using a barely-fitting VRAM quant, achieving 1M context on 3×3090 GPUs with ~80 tok/s at short context, ~60 tok/s at long context, and 3,701 tok/s at a 248k prefill; patches and the model are available on their Hugging Face page.
Running Qwen3.8-Flash-Next on 12GB VRAMOn an RTX 4070 (12GB VRAM) with 64GB RAM, the author used quantization, n-gram SSD offloading, --fit settings, and MTP patches to raise Qwen3.8-Flash-Next generation from ~6 tok/s to nearly 20 tok/s, with PP around 300–350 tok/s, and documents the trade-offs and patches used.
Qwen3.8-Flash-Next supports 1M context on MLX-serveQwen3.8-Flash-Next was shown running up to ~760k context and claimed 1M-context support on MLX-serve using M5 Max with 8-bit KV cache, sustaining ~40–75 tok/s generation and requiring ~117GB peak memory.
The A.I. Daily