The A.I. Daily
llama.cpp v0.6.0 Released With MTP Decoding
According to a Reddit post on r/LocalLLaMA, llama.cpp released version 0.6.0, adding MTP speculative decoding for Qwen4Exp along with many other updates.
llama.cpp
67 storiesllama.cpp is an open-source inference project for running local large language models, mainly in GGUF format, across CPUs, GPUs and other hardware. Recent coverage includes a heap out-of-bounds write flaw in its Pocket-TTS component triggered by a malicious mmproj file, fixed in build b11796; a merged pull request (#31028) that merges tensors during conversion and may require reconverting GGUF files; and a default llama server port change from 8080 to 9931. Cohere also merged support for its North Micro Vision Instruct encoder.
Developer Runs Qwen3.5 LLMs on Cheap FPGA Mining Hardware
A developer used ~$280 SQRL FK33 ex-mining FPGAs, with Claude assisting the RTL work, to run Qwen3.5-9B INT4 inference. Two cards at 75MHz produced ~3.2 tok/s generation, dropping to ~2.4 tok/s at longer context, with layer-by-layer validation against llama.cpp.
llama.cpp Outperforms Alternative Engine on Long-Form Reasoning
A user on r/LocalLLaMA found that the same IQ3_S model produces coherent output in llama.cpp but degrades into repetitive loops after roughly 10k tokens in another engine. This highlights how much the inference framework affects long-form generation stability for local LLMs.
User Shares Strata Speedup for Local LLMs but 64GB RAM Constraint
A Reddit user reported that on a dual-GPU setup (R9700, RTX 5060 Ti) with 64GB DDR5 RAM, running a quantized Qwen3.8-Flash-Next via Strata boosted inference from about 21 to 60 tokens per second, but memory usage hit 96%, preventing simultaneous ComfyUI image/video workloads, highlighting high system RAM demands of local LLMs.
176B MoE Model Runs on 16GB Laptop GPU
A developer ran Qwen3.8 Flash Next 176B on a laptop with a 16GB RTX 3080, 32GB RAM, and an SSD using his open-source engine TensorSharp. Benchmarks show 11.09 tok/s decode versus Strata's 10.24 tok/s, and whole-process time of 16.54s versus 62.15s. The result suggests quantization plus MoE-aware scheduling across VRAM, RAM, and SSD can make huge sparse models usable on consumer hardware.
Developer releases Ninfer 4080 for 16GB GPUs with 100k context
A developer released Ninfer 4080, an inference solution that runs ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on a 16GB RTX 4080, peaking at 2720 tok/s prefill and 262 tok/s generation. The project is open-sourced to bring better performance to 16GB-class GPUs than general-purpose engines.
Anyworld: self-hosted multiplayer text RPG run by a local LLM
A developer released Anyworld, a browser-based multiplayer text RPG in which a local LLM run via llama.cpp (cloud APIs also supported) acts as dungeon master. Players join by browser link with no setup, actions are resolved by the model, and real dice rolls are handled by Python RNG, enabling custom scenarios.
llama.cpp PR cuts Qwen Flash Next VRAM use
A llama.cpp pull request (#29825) halves the indexer score memory, reducing VRAM usage for Qwen Flash Next.
Strata With Qwen3.8 Next Speeds Up Slow-Memory Rigs
A user reports that Strata with Qwen3.8 Next and IQ3_XXS weights reaches roughly 45-70 tokens/s on a DDR4 and 7900XTX setup, versus about 22.5t/s with tuned Llama CPP, with similar results on 12GB and 16GB GPUs.
Developer uses iPhone as second GPU to speed up MacBook LLM inference
A developer running Qwen 3.8 27B on a 24GB M4 Pro MacBook used an iPhone 17 Pro Max over a 10Gb/s USB-C link as a second GPU to offload some layers, raising end-to-end prefill speed by 29%–44% and moving KV cache beyond 64k context to the phone.
llama.cpp Adds Decision Models
The local inference project llama.cpp added a Decision Models feature, discussed on r/LocalLLaMA. The update targets local model use and decision-making capabilities.
User builds 6-board BC-250 cluster to run Qwen locally
An enthusiast built a local AI cluster from six BC-250 mining boards, running Qwen Next Flash on four at about 28 tok/s and a 3.6 35B Q4 model on two at 60 tok/s.
Janus: Go Binary Runs GGUF Models via Vulkan
A developer released Janus, a single Go binary that runs GGUF models on AMD, Intel, or NVIDIA GPUs via Vulkan or CPU, exposing an OpenAI-compatible API. It requires no Python, Docker, or Ollama, and supports hot-swapping models and thinking models.
Pi extension lets local Qwen 27B skip reasoning and answer now
pi-llama-skip-reasoning, an extension for the Pi.dev harness, uses llama.cpp's native mechanism to force a local model to stop reasoning and answer immediately, triggered by a command or shortcut. Its author warns against skipping reasoning for important problems, since that hurts quality.
llama.cpp merges MTP support for Qwen Flash Next
A pull request adding MTP support for Qwen4Exp was merged into llama.cpp after 17 hours of work, enabling MTP with Qwen Flash Next. Corresponding GGUF quantizations have been published on Hugging Face.
User builds 64GB VRAM four-GPU local inference rig
A Reddit user showcased a workstation with four Tesla T4 16GB GPUs (64GB total VRAM), a Xeon W3225, 768GB RAM, running Ubuntu and llama.cpp, and said they want even more cards.
YC S25 Startup Magnitude Launches Self-Optimizing Inference Engine for Agents
Magnitude launched a self-optimizing inference engine for agents that runs on Mac, Linux, and Windows on any hardware and is up to 2x faster than llama.cpp. It features on-device compilation and tuning, dynamic memory allocation, and hybrid paged attention, designed for running local agents.
llama.cpp Adds GLM-5.3-Flash Support
A merged pull request adds support for the GLM-5.3-Flash (GLM5-Next) model to llama.cpp, letting users run it on their home computers.
Qwen3.8 runs locally at 51 t/s on 12GB laptop GPU
A user reports that the Strata inference engine with ISTA-DASLab's quantized Qwen3.8-Flash-Next hits 51 t/s generation and 1500 t/s prompt processing on a 12GB VRAM, 64GB RAM laptop, far outpacing stock llama.cpp. Strata currently supports only Nvidia and select GGUF models.
Community debates surge of llama.cpp hard forks
An r/LocalLLaMA user criticized the many llama.cpp hard forks optimized for specific GPUs that never submit PRs upstream, instead rebranding and promoting themselves. Consensus held that upstream must meet compatibility needs while downstream forks can be more focused, though low-effort engagement-bait forks remain criticized.
Logit penalty on Qwen models improves math accuracy
A user tested logit-bias penalties on words like "wait", "maybe" and "perhaps" across quantizations of Qwen3.5-4B in llama.cpp, running 50 random MATH-500 questions, building on a Meta paper on overthinking markers. The paper had not covered llama.cpp's quantization options.
42x faster prompt lookup drafting in llama.cpp
A llama.cpp community post shares a prompt lookup drafting optimization claimed to speed up inference by 42x. The method reuses text spans already present in the prompt as draft tokens to cut regeneration overhead, an efficiency improvement for local inference; technical details are in the linked post.
Four RTX 3060 Ti GPUs Hit 120 t/s Local Inference With 262K Context
A user built a 4-GPU rig from power-limited 8GB RTX 3060 Ti cards and, using tensor parallelism with Exl3 and HyperQwen, reached roughly 120 t/s at 150K context in bf16; quantizing the KV cache extends context to 262K with little slowdown under concurrency.
Ling 3.0 Tiny brings edge intelligence to old laptops
A user reports running the 8B-parameter (1B active) Ling 3.0 Tiny model via llama.cpp on a 2017 i5 laptop with 8GB RAM and no GPU, achieving about 10 tokens per second and iteratively writing a network-scan script. The author sees this as a new beginning for edge intelligence, enabling old devices to perform intelligent tasks without new compute.
Mica v0.1 4B Builds an Iron Pickaxe in Minecraft With Zero Generated Tokens
The open-weight Mica v0.1 4B played a live Minecraft 1.20.4 server by scoring label-token probabilities instead of generating text, needing 23 decisions to craft an iron pickaxe from an empty inventory at roughly 90-150 ms per step on an RTX 3090. Weights and code are public.
Qwengram-0.8B: n-gram memory transfer cuts perplexity 5.05%
A developer transferred Qwen3.8 Flash-Next's pretrained PLE n-gram memory into Qwen3.5-0.8B, keeping the backbone and ~51B-parameter memory frozen while training an R=1 reader; validation perplexity fell from 18.2759 to 17.3534, a 5.05% reduction. This is a language-model validation result, not a 5% benchmark accuracy gain.
Qwen3.8-Flash-Next Hits 65 Tokens/Second on 12GB VRAM
A developer built a custom inference engine, Strata, running Qwen3.8-Flash-Next on consumer hardware including a 12GB RTX 5070, achieving about 65 tokens/s output and 430 tokens/s prompt processing, a major gain over the earlier llama.cpp setup. The engine is open-sourced, one-click installable, and currently optimized for CUDA only.
Benchmark: ThinkingCap vs Swift vs Qwen 3.8-27B
A third-party Aider evaluation found that ThinkingCap and Swift fine-tunes cut Qwen3.8-27B's median reasoning tokens from 12,547 to about 7,400, a roughly 40% reduction matching their model cards, with first-try pass rates of 27.1%-30.8% and retry pass rates of 75.7%-77.6%, showing no meaningful performance loss.
Hugging Face releases VLM speculative decoding drafter
Hugging Face released an experimental DSpark draft model for its LFM2.5-VL-3B vision-language model, adding 280M parameters (8.9%) for up to 3.13x faster decoding on device and 2.66x on an H100 without changing output quality, with day-one support for llama.cpp, MLX-VLM and SGLang.
Transformers now runs llama.cpp quants
Hugging Face added GGUF support to transformers, letting users load Hub checkpoints sized for their machine via from_pretrained. To approach llama.cpp performance, it reuses ggml kernels and cuts generate overhead, initially targeting Apple Silicon and the Qwen3.5 architecture. Local tools like Ollama, LM Studio and Jan are powered by llama.cpp.