Issue 2026-11-13 · Industry · 研究 · 开源

Domyn and PyTorch train 1T MoE on 384 GH200 GPUs

Domyn, working with TorchTitan and PyTorch maintainers, trained a ~1T-parameter MoE on 384 GH200 GPUs in the JUPITER supercomputer at ~1,300 tokens/s per GPU, a 14x gain over their starting point. They used 384-way FSDP plus 4-way expert parallelism kept inside nodes, skipping pipeline parallelism to keep iteration fast.

r/MachineLearning3 d ago
Read original ↗