Unsloth

14 stories

Unsloth is an open-source project that provides quantized GGUF builds and fine-tuning tools for running large models locally. Recent coverage centers on its quant releases: Dynamic v3.1 GGUFs for Qwen3.8-27B, claimed 5-10% more accurate than UD3 on DeepSWE and Terminal Bench, and V3.1 GGUFs for Qwen3.8-Flash-Next with a claimed 15% accuracy gain. Unsloth also disputed ISTA-DASLab's gsq-rco benchmark numbers as unreproducible, and its IQ2 quant of DeepSeek-V4.1-Flash was used in local Delta engine tests.

Related topics

Community mulls slow-motion Qwen4 release rollout

An r/LocalLLaMA post traced Qwen4's drawn-out rollout, from architecture hints and conference reveals to teaser images and early X access, predicting the launch may land near ANT's IPO. It also mocks the expected day-one inference and quant chaos, reflecting community fatigue with the slow release.

r/LocalLLaMA ·

User runs 748B DeepSeek-V4.1-Flash on a €2k PC

A user ran the 748B-parameter DeepSeek-V4.1-Flash-1017 at 7.2 tok/s on a used ~€2,000 PC (RTX 3090, 128GB DDR4, i9-9900X) using a 1-bit quant and llama.cpp. A 110GB Engram table stays on NVMe and is read on demand, letting it fit. The output contained a physics error and has little practical use, the user said; Qwen models work better on the setup.

r/LocalLLaMA ·

Cohere says llama.cpp now supports all its latest model architectures

Cohere's Nick Frosst posted on r/LocalLLaMA that the team merged llama.cpp support for the North Micro Vision Instruct vision encoder, making all of Cohere's latest model architectures compatible with llama.cpp. The vision model underpins several new releases, and the Cohere Transcribe speech-to-text model is now available in Unsloth and Unsloth Studio for fine-tuning.

r/LocalLLaMA ·

Community tests full n-gram table for Qwen 3.8 Flash Next

A LocalLLaMA user says the SSD-resident PLE (n-gram table) is heavily quantized in common GGUFs, hurting performance. Packing an FP8 PLE with ple_fp8_pack.py reportedly cut errors for an iq3_s model with no speed loss, only extra disk use. The needed safetensors come from Qwen's FP8 repo, and the packed file loads via the --ple-gguf option.

r/LocalLLaMA ·

User calls for pinned model deployment guides on LocalLLaMA

A user proposes a pinned section on the LocalLLaMA forum with detailed deployment guides for each model, covering the best inference engine, minimum hardware requirements, and trusted sources for buying or renting hardware. The user complains that AI search answers are often hallucinated and even make up prices, making accurate information hard to find.

r/LocalLLaMA ·
That is everything