TorchAO

TKintegers K5 tops FP8 baseline in ternary inference

A developer released TKintegers, a balanced-ternary weight representation with GPU inference kernels. On Qwen2.5-0.5B-Instruct, K5 reached 3,402.85 tokens/s prefill and 26.75 tokens/s decode on an NVIDIA L4 lacking native K5 instructions, 3.21x and 3.10x the TorchAO FP8 baseline. The author noted the FP16 control was faster and results are limited to this setup.