Your files stay on your device

OpenAI tokenizer playground

Count tokens exactly with the encoding a model actually uses. Every token id and slice listed, both directions, computed in a worker.
no Web Workers in this environment: exact counts need them; the page stays honest insteadCountClearText → tokensToken ids → textEncoding

Paste text (or ids), then press Count.
Count with every encoding

The same text through o200kbase, cl100kbase, p50kbase, and r50kbase. Exact BPE counts per row. The spread between vocabularies is a fact, not a ranking.

- Counts are the BPE output of the installed gpt-tokenizer encoding, a port of OpenAI's tiktoken. Exact for these encodings: what the matching OpenAI models charge for is what is counted.
- Claude, Gemini, and other vendors use different tokenizers. Counts here are NOT their counts. The context-fit tool labels those as estimates instead of pretending.
- Special tokens pasted into the text (like <|endoftext|>) count as their single special id: that is what a model actually receives.
- A token can hold partial UTF-8 (an emoji often splits); its slice may show the replacement character. That is the real boundary, not a bug.

- Plain sentence

Ordinary English: watch spaces lead most tokens.

- Emoji + accents

Astral emoji and CJK carry multi-byte tokens; one grapheme is not one token.

- Code snippet

Code tokens split on punctuation runs; why code costs more tokens than prose.

- Special token

A pasted special token counts as its single special id; what the model actually sees.

Paste, drop, or type to begin. Everything stays on this device.

More AI tools

- Context fit (/ai/context-fit)
- API cost estimator (/ai/cost-estimator)
All AI tools (/ai)

Canonical HTML: https://nutter.tools/ai/tokenizer/
Markdown version: https://nutter.tools/ai/tokenizer/index.md
Plain-text version: https://nutter.tools/ai/tokenizer/index.txt
Agent index: https://nutter.tools/llms.txt