Domyn and PyTorch train 1T MoE on 384 GH200 GPUs
Domyn, working with TorchTitan and PyTorch maintainers, trained a ~1T-parameter MoE on 384 GH200 GPUs in the JUPITER supercomputer at ~1,300 tokens/s per GPU, a 14x gain over their starting point. They used 384-way FSDP plus 4-way expert parallelism kept inside nodes, skipping pipeline parallelism to keep iteration fast.