LMU

Study: Byte-Level LLMs Can Overcome Subword Tokenization Limits

Researchers from LMU, the Allen Institute for AI, Cambridge, the University of Washington and Imperial College London published a Nature method that converts existing LLMs into byte-level models without retraining from scratch, using under 1% of the usual compute. The byteified models largely keep original performance, add byte-level benefits, outperform prior same-size byte-level LLMs and handle character-level tasks like spelling backwards far better.