RTX 3060
4 storiesThe RTX 3060 is an Nvidia consumer graphics card, and its 12GB version is often used to run large language models locally. Recent coverage centers on developers running models larger than the card's memory: the llama.cpp fork Betelgeuse runs a 68GB quantized Qwen 3.8 Flash Next MoE on an RTX 3060 12GB with 16GB RAM at about 21–22 tok/s with bit-exact output, versus roughly 1.5 tok/s for stock llama.cpp. Another report runs DeepSeek V4.1 Flash at 19 tok/s decode, and virtio-nvgpu brings near-native GPU access to KVM guests.
Related topics
A developer reports running DeepSeek V4.1 Flash Q8_XL on a budget RTX 3060 12GB + 16GB RAM PC via Betelgeuse, hitting 19 tok/s decode and 62 tok/s prefill. Cache partitioning and keeping the Engram table on NVMe reportedly ease the prefill bottleneck for large out-of-core MoEs.
A developer released Betelgeuse, a llama.cpp fork, that runs a 68GB quantized Qwen 3.8 Flash Next MoE on a consumer PC with an RTX 3060 12GB and 16GB DDR4 RAM at 22 tok/s sustained, spiking to 48 tok/s, with no gate pruning and 100% bit-exact output. Stock llama.cpp managed only about 1.47 tok/s on the same setup.
A developer reports running a 68GB quantized Qwen 3.8 Flash Next MoE at about 21 tok/s on an RTX 3060 12GB with 16GB RAM, using expert-routing prediction to speed CPU/GPU offload with no gate pruning and bit-exact output. Warm cache reaches 24+ tok/s, versus roughly 1.4-2.1 tok/s with stock llama.cpp.
virtio-nvgpu forwards NVIDIA kernel driver ioctls at the driver ABI level, letting a Linux guest run the vendor's own unmodified user-mode drivers. On an RTX 3060, guest performance is within 2% of bare metal above about 2 ms per frame and costs the same CPU; four guests together hit about 103.7 fps, targeting headless streaming.
That is everything