Toolchain · 2019
HF Tokenizers
Hugging Face tokenization library with a Rust reference implementation and bindings for other languages.
What is HF Tokenizers written in?
Its runtime is implemented in Rust (well-documented).
Frequently Asked Questions
What language is HF Tokenizers written in?
HF Tokenizers is primarily implemented in Rust.
When was HF Tokenizers first released?
HF Tokenizers was first released in 2019.
Evidence Sources
- https://github.com/huggingface/tokenizers/blob/main/README.md Source repository
Discover More
Spot something wrong?
Every fact on this page is meant to be checkable. If one is wrong, say so and it becomes a dataset change, not a patch to this page.