The A.I. Daily
Understanding and Enhancing Kimi Delta Attention
A study compares the expressivity of Gated Deltanet and Kimi Delta Attention, proposing Complex KDA (CKDA). By extending gate range to [-1,1] and delta rule learning rate to [0,2], CKDA can express any orthogonal diagonal-plus-rank-one matrix and track S3, S4, and A5 groups, while being competitive with standard KDA on language modeling.