Study: deleting under 0.05% of weights matches full fine-tuning
Researchers propose Deletion Tuning, which zeroes fewer than 0.05% of weights in transformer gate projections and matches full fine-tuning across reinforcement learning, supervised fine-tuning and on-policy distillation. The learned masks are hundreds to thousands of times smaller than full checkpoints and forget far less in continual learning.