vLLM is a software framework for running large language models locally and managing inference resources such as memory and KV cache. Recent coverage shows a user offloading most of Qwen3.8-Flash-Next's KV cache to RAM to reach a 1M context on 3×3090 GPUs (about 80/60 tok/s short/long, 3,701 tok/s at a 248k prefill), inspection of Deepseek V4.1 Flash safetensors (~748B total/552B base), and advice urging hobbyists not to let FOMO drive costly local LLM setups.
Offload most of Qwen3.8 KV cache to RAMA vLLM user shared on r/LocalLLaMA a method to keep most of Qwen3.8-Flash-Next's KV cache in system RAM while using a barely-fitting VRAM quant, achieving 1M context on 3×3090 GPUs with ~80 tok/s at short context, ~60 tok/s at long context, and 3,701 tok/s at a 248k prefill; patches and the model are available on their Hugging Face page.
Deepseek V4.1 Flash is 748B total/552B baseInspection of the safetensors shows Deepseek V4.1 Flash’s main model is about 551.566B parameters (40 layers), with FFN experts totaling 543.582B and other parts 7.984B; additionally an engram of ~196.929B, DSpark/MTP ~14.225B and a vision encoder ~0.485B, summing to roughly 748B. Other reported sizes (e.g., 485B, 522B) stem from differing counting methods or misreads.
Advice for hobbyists: don’t let FOMO drive local LLM spendingAdvice to hobbyists: don’t let FOMO drive costly local LLM setups—learn via APIs or small models that fit existing hardware; you can gain more by doing less rather than buying expensive GPUs.
The A.I. Daily