llama.cpp is an open-source inference project for running local large language models, mainly in GGUF format, across CPUs, GPUs and other hardware. Recent coverage includes a heap out-of-bounds write flaw in its Pocket-TTS component triggered by a malicious mmproj file, fixed in build b11796; a merged pull request (#31028) that merges tensors during conversion and may require reconverting GGUF files; and a default llama server port change from 8080 to 9931. Cohere also merged support for its North Micro Vision Instruct encoder.
Yandex Releases 80B AliceAI Foundation ModelYandex released AliceAI-Foundation-80B-A3B-Base, a Russian-developed competitor to Qwen 35B and DeepSeek V4 Flash. It uses a fully custom architecture rather than a Qwen3 finetune, has not been post-trained, and currently lacks llama.cpp support.
Qwen 27B runs autonomously for 3 weeks on one 3090A user ran a quantized Qwen 27B locally on a single RTX 3090 for about 21 days with the goal of having the agent build its own CUDA inference engine. The run consumed roughly 230M tokens, spent about 83 hours in 699 compactions, and reached about half of llama.cpp's prefill speed while producing working kernels and benchmarks.
Qwen3.8 27B IQ3_XXS vs Bonsai Ternary PQ2 quantization testA user benchmarked Qwen3.8 27B IQ3_XXS (10.18GiB) against Bonsai Ternary PQ2 (6.42GiB) on the same 16GB VRAM setup. Both completed 4/4 tasks, but Qwen ran 3.02x faster and used 3.10x fewer output tokens; Bonsai was slower with slightly worse results.
Voodoo Dynamic Quant goes MITA developer open‑sourced Voodoo Dynamic Quant under an MIT license on GitHub; the method uses gradient descent to select per‑tensor quantization levels (for formats like GGUF) to optimize filesize while preserving model quality.
Local LLM community glimpses a golden eraA Reddit post says hardware shortages have pushed the local LLM community back into low-level optimization: forks of llama.cpp and Strix Halo's halogen-flash-server boosted Qwen 3.8 Flash Next (Q38FN) to about 52 tok/s decode and 1300 tok/s prefill. The author argues scarcity spurs learning and technical growth.
OpenUI releases OUI-1 UI modelOpenUI released OUI-1, fine-tuned on DiffusionGemma using a custom DSL OpenUI‑Lang instead of HTML/Markdown/React. Fine‑tuning can save context compared with prompting a general LLM to emit OpenUI‑Lang, but may bias the model toward that format. DiffusionGemma isn't supported by llama.cpp (so Ollama isn't an option); weights are on Hugging Face, and users wonder how to run it locally on a consumer GPU like an RTX 5090.
Qwen3-0.6B drives real desktop Chrome on old phoneA developer ran Qwen3-0.6B (Q4_K_M) with llama.cpp on a Galaxy Note 8 (2017, 6GB) and used a relay to drive a real desktop Chrome. The model performed structured page-perception tasks and scored 10/10 on several tests. The experiment shows structured browser perception was far more efficient than feeding raw HTML.
The A.I. Daily