Llama 3.2 1B

Meta trains byte-level small models, predicted to beat token models

Meta trained ~1B-parameter byte-level models distilled from Llama 3 8B; they lag token models early but keep improving. Extrapolated curves suggest they could beat Llama 3.2 1B and Gemma 3 1B with about 6x less text, though real runs still favored tokens.