Tokenizers
ProductHigh-performance Rust and Python library for modern text tokenization.
- Mar 10, 2020
- —
- —
- Yes
- —
- 2
Releases
Sep 21, 2026Tokenizers v1 releasedSDK release
Hugging Face released Tokenizers v1, a major milestone for its Rust-based tokenization stack.
Mar 2020Tokenizers library releasedSDK release
Hugging Face introduced its high-performance Rust-backed Tokenizers library for production-scale text preprocessing.