HOPE: higher-order pruning that accounts for expert cooperation in MoEs
A Hugging Face blog post presents HOPE, a pruning method for mixture-of-experts models that scores experts in pairs rather than scoring each independently. It notes that Qwen3.5-122B has 256 experts per layer, and that expert MLPs alone account for 95% of its ~250 GB footprint while only 8 experts per layer are used per token. The method needs no training and composes with quantization.