The A.I. Daily
Developer Converts Dense Models to MoE on RTX 4060
A developer converted Qwen2.5-0.5B and SmolLM2-360M into sparse MoE models using only ~40-44% of original MLP compute per token. However, WikiText-2 perplexity rose from 14.66 to 19.20 and mean accuracy fell from 0.467 to 0.408, showing a clear quality gap.