Open-source Project Teaches LLM Inference from Scratch in C++/CUDA
A teaching project, samir-vllm, appeared on Hacker News, explaining LLM inference from first principles by building a small engine in C++ and CUDA. It offers over 6 hours of video and 80+ pages of handwritten notes covering weight loading, RMSNorm, RoPE, self-attention, KV cache, batching and paged attention. It already supports loading Llama 3.2 1B Instruct safetensors weights.