Dust: Pretraining Transformers Without Backpropagation

Researchers propose Dust, a zeroth-order method that perturbs activations independently per token so one forward pass evaluates a virtual population. It approaches or even exceeds backprop with enough compute, is 10^3-10^4x more efficient than state-of-the-art ES method EGGROLL, and larger models are more population-efficient, not less.