Simulating fault tolerance with stage skipping in pipeline-parallel training

Templar team simulated stage-skipping fault tolerance in pipeline-parallel training on the Crucible distributed pre-training platform. With a 178M model, eight replicas, and four stages per replica, validation loss stayed close to the no-failure baseline at 1% per-replica failure probability. Fixed projections shared across layers improve robustness, supporting training on unreliable compute.