Tokenizer Visualizer
Upload your local tokenizer or load from HuggingFace Hub.
Upload tokenizer file (tokenizer.json / .model)
Drop File Here
- or -
Click to Upload
Or HuggingFace Hub name
Load Tokenizer
Status
Input Text
Index
⋮
Token
⋮
Token ID
⋮
Total Tokens