The A.I. Daily
Hugging Face releases tokenizers v1 with big speed gains
Hugging Face released a v1 release candidate of tokenizers focused on performance, saying it is often tens of times faster than v0.23. The team says tokenization can bottleneck large-scale training, high-concurrency serving and long inputs, leaving GPUs idle. IBM, NVIDIA and the ExecuTorch team contributed patches and cross-hardware testing.