Unsloth is an open-source project that provides quantized GGUF builds and fine-tuning tools for running large models locally. Recent coverage centers on its quant releases: Dynamic v3.1 GGUFs for Qwen3.8-27B, claimed 5-10% more accurate than UD3 on DeepSWE and Terminal Bench, and V3.1 GGUFs for Qwen3.8-Flash-Next with a claimed 15% accuracy gain. Unsloth also disputed ISTA-DASLab's gsq-rco benchmark numbers as unreproducible, and its IQ2 quant of DeepSeek-V4.1-Flash was used in local Delta engine tests.
Unsloth released Dynamic V3.1 quantizations for Qwen3.8-27B, improving average accuracy by 5-10% at the same size across 2-bit to 6-bit quants. The 27B dense model supports vision and reasoning with a 256K context and runs locally on 17GB RAM/VRAM.
Unsloth released Dynamic v3.1 GGUFs for Qwen3.8-27B, claiming 5-10% better accuracy than UD3, GSQ-RCO and Exllama-3 on DeepSWE and Terminal Bench. Two- to six-bit quants are updated, while the 3-bit stays on UD3.
An r/LocalLLaMA post traced Qwen4's drawn-out rollout, from architecture hints and conference reveals to teaser images and early X access, predicting the launch may land near ANT's IPO. It also mocks the expected day-one inference and quant chaos, reflecting community fatigue with the slow release.
ISTA-DASLab's gsq-rco quantization has drawn criticism from Unsloth for unreproducible benchmark numbers, with Unsloth finding its own quants outperform gsq-rco. The creator of the Basalt inference engine then replaced ISTA-DASLab-based quants with similar-sized Unsloth equivalents.
Delta Inference Engine v0.93 supports 64–192GB RAM; the Unsloth IQ2 quant of DeepSeek-V4.1-Flash runs at 41 tok/s on one RTX 3090 with 192GB RAM, first token about 40% sooner, scoring 57/99 versus 59 for the full model. It adds Unsloth quants, EXL3 and basic multi-GPU support.
Unsloth released V3.1 of its GGUF quantized versions of Qwen3.8-Flash-Next, claiming a 15% accuracy improvement.
A user ran the 748B-parameter DeepSeek-V4.1-Flash-1017 at 7.2 tok/s on a used ~€2,000 PC (RTX 3090, 128GB DDR4, i9-9900X) using a 1-bit quant and llama.cpp. A 110GB Engram table stays on NVMe and is read on demand, letting it fit. The output contained a physics error and has little practical use, the user said; Qwen models work better on the setup.
Cohere's Nick Frosst posted on r/LocalLLaMA that the team merged llama.cpp support for the North Micro Vision Instruct vision encoder, making all of Cohere's latest model architectures compatible with llama.cpp. The vision model underpins several new releases, and the Cohere Transcribe speech-to-text model is now available in Unsloth and Unsloth Studio for fine-tuning.
A LocalLLaMA user says the SSD-resident PLE (n-gram table) is heavily quantized in common GGUFs, hurting performance. Packing an FP8 PLE with ple_fp8_pack.py reportedly cut errors for an iq3_s model with no speed loss, only extra disk use. The needed safetensors come from Qwen's FP8 repo, and the packed file loads via the --ple-gguf option.
Developers released Basalt, an MIT-licensed inference engine for Blackwell GPUs, tuned for Qwen3.8 Flash-Next. It claims up to 2.6x Strata's throughput, supports dual GPUs and eight concurrent streams.
A user praised Qwen 3.6 35B (Unsloth UD_Q4_K_XL quant) for consistent general agentic tasks like smart-home control and messaging, reaching ~120-140 t/s generation on Tesla P100s, though it is weak at coding and holding a persona.
An XDA writer says the local model Qwen 3.8-27B handles most of his daily tasks, so he no longer needs the $20/month ChatGPT Plus and Claude Pro subscriptions. He runs Unsloth's UD-IQ4_XS quantized build, a 13.3GB download, on a 4070 Ti Super with 16GB of VRAM, with room to spare at a 16K context window.
A user proposes a pinned section on the LocalLLaMA forum with detailed deployment guides for each model, covering the best inference engine, minimum hardware requirements, and trusted sources for buying or renting hardware. The user complains that AI search answers are often hallucinated and even make up prices, making accurate information hard to find.
A user's comparison shows Swift-1.5-Qwen3.8-Flash-Next scored 41.1% first-try pass on the Aider benchmark versus 40.2% for the base model, while cutting tokens per case from 17,646 to 6,991 and time from 1,542s to 608s, sharply reducing overthinking.
That is everything