Why the Same Prompt Has Different Token Counts Across Models

Published March 12, 2026 · Updated March 14, 2026 · 4 min read

If you count the tokens in a prompt using OpenAI's tokenizer and then count with Anthropic's, you'll get different numbers — sometimes meaningfully different. This isn't a bug; it's an expected result of each model using its own tokenizer with a different vocabulary.

Why tokenizers differ

Each model family trains its own tokenizer on its own corpus using its own vocabulary size. The tokenizer learns which substrings are common enough to deserve a single token versus which should be split into multiple tokens. A vocabulary trained primarily on English code will tokenize code tokens differently from a vocabulary trained on general web text.

OpenAI's GPT models use the tiktoken library with cl100k_base or o200k_base encoding. Anthropic's Claude uses their own tokenizer. Llama models use SentencePiece with BPE. Gemini uses a different tokenizer again.

How different can counts be?

For standard English text, the difference between major tokenizers is typically 5–15%. A 1,000-token prompt in GPT-4 might be 920 tokens in Claude or 1,080 tokens in Llama. For most applications this difference is insignificant.

Differences are larger in specific cases:

Code: programming language syntax patterns tokenize differently across vocabularies. A 500-line Python file might be 1,200 tokens in one tokenizer and 1,500 in another.
Non-English text: tokenizers trained primarily on English handle other languages with variable efficiency. Mandarin Chinese tokenizes to many more tokens per character in GPT models than in models trained on more multilingual data.
Numbers and special characters: long numbers, mathematical notation, and special character sequences tokenize inconsistently.

What this means in practice

When you're estimating costs or planning for context window limits, use the tokenizer for the specific model you're targeting. Using GPT's tokenizer to estimate Claude's context window usage gives you a ballpark — often accurate enough — but not an exact count.

Build in a safety margin. When planning for context window limits, count tokens for the specific model, then build in a 10–15% buffer. This accounts for tokenizer variance if you ever switch models and for the overhead of special tokens the API adds automatically (like role markers).

Special tokens add overhead

Models add special tokens around your content — role markers like [INST] and [/INST], beginning-of-sequence tokens, end-of-sequence tokens, and system prompt delimiters. These are model-specific and usually add 5–20 tokens per message. They're not visible in your prompt but count against the context window and contribute to the API cost. Token counters that simulate the model's chat format account for these; simple character counters don't.