llama.cpp is an open-source inference project for running local large language models, mainly in GGUF format, across CPUs, GPUs and other hardware. Recent coverage includes a heap out-of-bounds write flaw in its Pocket-TTS component triggered by a malicious mmproj file, fixed in build b11796; a merged pull request (#31028) that merges tensors during conversion and may require reconverting GGUF files; and a default llama server port change from 8080 to 9931. Cohere also merged support for its North Micro Vision Instruct encoder.
Developer Measures Output Damage From Engine Speed FlagsA developer tested Basalt engine's routing-bias and expert-skip speed flags on RTX 5090 hardware, finding they notably alter outputs: Q8_0's KLD rose from 0.031 to 0.401 with shipped defaults and 1.616 in the worst case, with same top-1 agreement falling to 56.8%.
Developer runs Qwen 3.8 Flash Next on an RTX 3060 at 22 tok/sA developer released Betelgeuse, a llama.cpp fork, that runs a 68GB quantized Qwen 3.8 Flash Next MoE on a consumer PC with an RTX 3060 12GB and 16GB DDR4 RAM at 22 tok/s sustained, spiking to 48 tok/s, with no gate pruning and 100% bit-exact output. Stock llama.cpp managed only about 1.47 tok/s on the same setup.
DFlash2 drafter lifts Qwen3.8-Flash-Next speed by 17%A user's tests show enabling the DFlash2 drafter for Qwen3.8-Flash-Next in llama.cpp v0.8.0 raises decode speed by about 17% geometric mean: +18% on code, +39% on lists, -3% on prose. The DFlash2 draft head is only ~350MB versus 2.2GB for MTP and supports vision input.
llama.cpp discloses out-of-bounds write flawllama.cpp's Pocket-TTS component has a heap out-of-bounds write flaw triggered by a malicious mmproj file; it requires local access and a public exploit exists. Upgrading to build b11796 fixes it, but the root-cause patch is still open.
llama.cpp Adds Tensor Merge on GGUF ConversionOpen-source project llama.cpp merged a pull request (#31028) by contributor am17an that merges tensors during conversion. The change may require users to reconvert existing GGUF model files.
8GB GPU runs 262K-context Qwen quant at 70 T/sA user demonstrated running ISTA DASLab's 3.5 BPW Qwen-Flash-Next Coder quant on 8GB VRAM plus 24GB RAM, reaching 262K context with about 70 T/s prefill and 8 T/s generation via MoE offloading and disk n-gram. Half the experts are pruned, giving an effective 1.89 bits per parameter while retained weights stay at 3.5 BPW.
ISTA-DASLab releases quantized DeepSeek-V4.1-FlashISTA-DASLab published GGUF quantizations of DeepSeek-V4.1-Flash-1017 using its GSQ-RCO quantization method, available on Hugging Face. The model is not yet supported in llama.cpp because original architecture support has not been merged.
Developer runs GGUF Q4 model in HTML via WebGL2A LocalLLaMA developer says building your own inference engine is feasible: a GGUF Q4 model runs inside a webpage via WebGL2 compute buffers and GLSL, though tokenizer and vocab handling remain a debugging nightmare. The experiment shows browser-based local inference is possible but costly to engineer.
User tests LFM2.5-24B-A2B, finds local performance poorA Reddit user tested the quantized LFM2.5-24B-A2B model across several local inference frameworks, reporting looping outputs and broken, incomplete code on a Hermes plugin task, and judged its performance poor.
User runs 748B DeepSeek-V4.1-Flash on a €2k PCA user ran the 748B-parameter DeepSeek-V4.1-Flash-1017 at 7.2 tok/s on a used ~€2,000 PC (RTX 3090, 128GB DDR4, i9-9900X) using a 1-bit quant and llama.cpp. A 110GB Engram table stays on NVMe and is read on demand, letting it fit. The output contained a physics error and has little practical use, the user said; Qwen models work better on the setup.
Eagle-4 Released: Faster than DFlash2 & DSparkEagle-4 achieves 3.59x speedup on Qwen 3.8 27B, outperforming DFlash2 (2.80x) and DSpark (2.57x). Currently only supports Qwen 3.8 27B; community awaits integration into llama.cpp/vLLM.
AnyLanguageModel 1.0 ships for local models on Apple platformsAnyLanguageModel 1.0 is out as a drop-in replacement for Apple's Foundation Models framework, letting apps switch between the system model, local models via llama.cpp, MLX and Core ML, and hosted providers such as OpenAI and Anthropic. swift-huggingface 1.0 and swift-xet 1.0 ship alongside to speed model downloads.
Cohere says llama.cpp now supports all its latest model architecturesCohere's Nick Frosst posted on r/LocalLLaMA that the team merged llama.cpp support for the North Micro Vision Instruct vision encoder, making all of Cohere's latest model architectures compatible with llama.cpp. The vision model underpins several new releases, and the Cohere Transcribe speech-to-text model is now available in Unsloth and Unsloth Studio for fine-tuning.
llama server default port changes from 8080 to 9931Starting with v0.7.0, llama server's default port changes from 8080 to 9931. Users who did not explicitly set a port face a breaking change on upgrade, and tools hardcoding port 8080 will need adjustment.
Community Proposes New IQ2_K to IQ6_K Quantization TypesThe LocalLLaMA community is discussing new llama.cpp quantization types IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K. File sizes match Q-quants and KLD improves substantially, but performance drops slightly and creating IQ quants is slow. Test numbers cover only a single model.
Ling 3.1 Flash 560B MoE deployed across four DGX Sparks via llama.cppA developer loaded Ling 3.1 Flash, a 560B MoE with 25B active parameters per token, across four DGX Sparks using llama.cpp RPC and Atomic Chat's day-one Q4 GGUF, since no official vLLM FP4 build was available. He expects bandwidth and speed to be painful and wants to compare it with the 124B Ling 3.0 Flash.
Community tests full n-gram table for Qwen 3.8 Flash NextA LocalLLaMA user says the SSD-resident PLE (n-gram table) is heavily quantized in common GGUFs, hurting performance. Packing an FP8 PLE with ple_fp8_pack.py reportedly cut errors for an iq3_s model with no speed loss, only extra disk use. The needed safetensors come from Qwen's FP8 repo, and the packed file loads via the --ple-gguf option.
User recounts local LLM journey with llama.cppA Dutch user recounts running local LLMs since March 2025, starting with Mistral Nemo through SillyTavern and llama.cpp, then upgrading to dual RTX 5060 Ti 16GB cards in hopes that llama.cpp will add tensor parallel and NVFP4 support.
User shares 10 practical uses for Qwen 3.6 27BA LocalLLaMA user posted a guide listing ten things to do with Qwen 3.6 27B via llama.cpp, including building small apps, custom chat frontends, ReAct loops and MCP tools. The author calls the model their first agent and aims to help newcomers get started with open-source LLMs.
llama.cpp runs GGUF models natively on jailbroken PS5PS5LM uses llama.cpp's Vulkan backend to run GGUF models on a jailbroken PS5's RDNA 2 GPU, serving OpenAI- and Anthropic-compatible APIs as a local model server.
AnythingLLM moves from Ollama to Lemonade and llama.cppAnythingLLM is switching its backend from Ollama to AMD's Lemonade embedded SDK and llama.cpp. The move reflects consolidation in local LLM inference tooling, welcomed by an AMD developer.
AWS Deploys llama.cpp Decision Models on SageMaker AIAWS published guidance on deploying decision models with llama.cpp on Amazon SageMaker AI, letting developers run lightweight inference in its managed environment.
OpenMed 3.0 ships: Apache-2.0 clinical AI that runs fully localOpen-source medical AI toolkit OpenMed released version 3.0 under Apache-2.0, keeping patient data local and working offline. The update adds Patient Journey, which stitches clinic notes, FHIR feeds, HL7 messages and lab CSVs into one per-patient timeline, and refuses to guess on ambiguous matches, keeping conflicts flagged for human review.
Open-source Mac app DigUp runs EmbeddingGemma 2 locally for file searchA developer releases DigUp, a free MIT-licensed Mac app that runs Google DeepMind's EmbeddingGemma 2 locally via llama.cpp and Metal to search text, images, audio and video by content, using an 865MB Q8_0 GGUF model with no Python required.
Qwen 3.8 Flash runs at ~21 tok/s on RTX 3060A developer reports running a 68GB quantized Qwen 3.8 Flash Next MoE at about 21 tok/s on an RTX 3060 12GB with 16GB RAM, using expert-routing prediction to speed CPU/GPU offload with no gate pruning and bit-exact output. Warm cache reaches 24+ tok/s, versus roughly 1.4-2.1 tok/s with stock llama.cpp.
User builds 256GB VRAM rig with 8 Radeon Pro V620sA user built a roughly $2,800 inference rig with eight used Radeon Pro V620 cards totaling 256GB VRAM, running Qwen3.8-Flash-Next via a custom vLLM fork. It reached 60-100 tokens/s decode and over 3,000 tokens/s prefill, roughly 8x faster than llama.cpp.
Four open decision models benchmarked on an RTX 4090A user benchmarked four new open decision models, Laya, d1 3B, Clef-Flash 9B and Lev 4B, on a single RTX 4090, tasking them to flag centipede names word by word across 9,534 words of Wikipedia text. Laya was fastest overall at 3.9 ms per word, about 7,980 words in 32 seconds and 97.4% accuracy; Lev was slowest but caught the most names.
llama.cpp takes the stageA llama.cpp presentation took the stage; the project, led by Georgi Gerganov, is a key inference framework for running LLMs locally. Specifics were not detailed in the post.
llama.cpp Adds GPU Cache for MoE Experts in Host MemoryDeveloper am17an submitted PR #29887 to llama.cpp adding a GPU cache for MoE experts kept in host memory, potentially delivering big speedups for Mixture-of-Experts models that don't fully fit in VRAM.
llama.cpp adds local support for GLM 5 Flash MTPA llama.cpp pull request adds support for GLM 5 Flash MTP, allowing users to run the model locally. The update was submitted by contributor pwilkin.
The A.I. Daily