Tokenizer
Large language models process text using tokens, which are common sequences of characters. This tool shows how your text is tokenized.
Input Text
Enter text to see how it would be tokenized by a language model
Characters:
0
Tokens:
0
Large language models process text using tokens, which are common sequences of characters. This tool shows how your text is tokenized.
Enter text to see how it would be tokenized by a language model