Your files stay on your device

# OpenAI tokenizer playground

Count tokens exactly with the encoding a model actually uses. Every token id and slice listed, both directions, computed in a worker.
no Web Workers in this environment: exact counts need them; the page stays honest insteadCountClearText → tokensToken ids → textEncoding

Paste text (or ids), then press Count.
Count with every encoding

The same text through o200k_base, cl100k_base, p50k_base, and r50k_base. Exact BPE counts per row. The spread between vocabularies is a fact, not a ranking.

- Counts are the BPE output of the installed gpt-tokenizer encoding, a port of OpenAI's tiktoken. Exact for these encodings: what the matching OpenAI models charge for is what is counted.
- Claude, Gemini, and other vendors use different tokenizers. Counts here are NOT their counts. The context-fit tool labels those as estimates instead of pretending.
- Special tokens pasted into the text (like <|endoftext|>) count as their single special id: that is what a model actually receives.
- A token can hold partial UTF-8 (an emoji often splits); its slice may show the replacement character. That is the real boundary, not a bug.

- Plain sentence

Ordinary English: watch spaces lead most tokens.

- Emoji + accents

Astral emoji and CJK carry multi-byte tokens; one grapheme is not one token.

- Code snippet

Code tokens split on punctuation runs; why code costs more tokens than prose.

- Special token

A pasted special token counts as its single special id; what the model actually sees.

Paste, drop, or type to begin. Everything stays on this device.

## More AI tools

- [Context fit](/ai/context-fit)
- [API cost estimator](/ai/cost-estimator)
[All AI tools](/ai)

---

Canonical HTML: https://nutter.tools/ai/tokenizer/
Markdown version: https://nutter.tools/ai/tokenizer/index.md
Plain-text version: https://nutter.tools/ai/tokenizer/index.txt
Agent index: https://nutter.tools/llms.txt