Tokenizers
ProductHigh-performance Rust and Python library for modern text tokenization.
- Launched
- Mar 10, 2020
- Models
- —
- Modality
- —
- Open source
- Yes
- API available
- —
- Releases
- 2
Releases
Sep 21, 2026Tokenizers v1 releasedSDK release
Hugging Face released Tokenizers v1, a major milestone for its Rust-based tokenization stack.
Mar 2020Tokenizers library releasedSDK release
Hugging Face introduced its high-performance Rust-backed Tokenizers library for production-scale text preprocessing.

