← Back to Leaderboard
Tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
⭐
10.7K
Stars
🍴
1.1K
Forks
👥
147
Contributors
💚
58
Health Score
Star History
Not enough data for chart
Health Score Breakdown
Star Momentum
0
Commit Frequency
23
Issue Health
80
Contributors
100
Release Cadence
80
Documentation
80
Community Signal
100
Project Info
- Language
- Rust
- License
- Apache-2.0
- Last Commit
- 4d ago
- Last Release
- v0.23.1 (14d ago)
- Open Issues
- 167
- Bus Factor
- 10
- Commits/Week
- 4.6
- Avg Issue Close
- 6.1 days
bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers
Project Badge
Embed a live health badge in a README or docs page.
[](https://www.ai-tools-scout.com/projects/tokenizers)Compare Tokenizers
See how this project stacks up against other framework tools.
Weekly AI open-source movers
Get the fastest-growing projects, useful MCP servers, and technical reads in one weekly email.