Toolchain · 2019

HF Tokenizers

RustWritten in
2019First released
Hugging FaceBuilt by
Built from HHF Tokenizers Rust
Runtime

Hugging Face tokenization library with a Rust reference implementation and bindings for other languages.

What is HF Tokenizers written in?

Its runtime is implemented in Rust (well-documented).

Frequently Asked Questions

What language is HF Tokenizers written in?
HF Tokenizers is primarily implemented in Rust.
When was HF Tokenizers first released?
HF Tokenizers was first released in 2019.

Evidence Sources

Discover More

Spot something wrong?

Every fact on this page is meant to be checkable. If one is wrong, say so and it becomes a dataset change, not a patch to this page.

Goes to a public GitHub issue for review. Nothing you write appears on this site.

Explore HF Tokenizers in Graph →