eaddario

Community Proposes New IQ2_K to IQ6_K Quantization Types

The LocalLLaMA community is discussing new llama.cpp quantization types IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K. File sizes match Q-quants and KLD improves substantially, but performance drops slightly and creating IQ quants is slow. Test numbers cover only a single model.