Alps

Hugging Face Releases MoE Training Handbook Built on Megatron-LM

Hugging Face's Nouamane released a Mixture-of-Experts training handbook covering Megatron-LM settings, from hardware checks and dense baselines to MoE layer details and serving/RL, with 55 figures and scripts released. Lessons draw on about 800k GH200 GPU-hours of MoE training on the Alps cluster at CSCS.