32 Researchers Release Comprehensive Tokenization Survey

Thirty-two tokenization researchers spent about eight months producing a comprehensive survey of tokenization for modern NLP, covering algorithms, evaluation, multilinguality, encodings, and theory. It also examines replacements such as latent or visual tokenization, plus adjacent topics like constrained generation, token healing, and tokenizer security.