What Is an AI Token and Why Does It Matter?

Published March 24, 2026 · Updated March 25, 2026 · 4 min read

When you send a prompt to an AI model, the model doesn't read words the way you do. It reads tokens. Understanding what tokens are — and how they're different from words and characters — helps you work with context limits, estimate costs, and understand why the same text has different token counts in different models.

What a token actually is

A token is the unit of text that a language model processes. Tokenizers split input text into tokens using a learned vocabulary of common substrings. Common English words are typically one token. Rare words are split into multiple tokens. Whitespace, punctuation, and numbers each tokenize differently.

Some examples of how English text tokenizes (using GPT-4's tokenizer):

"Hello"        → 1 token  ["Hello"]
"hello world"  → 2 tokens ["hello", " world"]
"unbelievable" → 3 tokens ["un", "believ", "able"]
"2026"         → 1 token  ["2026"]
"$19.99"       → 3 tokens ["$", "19", ".99"]

As a rough rule: one token is approximately 4 characters or 0.75 words in English. A 1,000-word document is roughly 1,300 tokens. A 10-page PDF is roughly 3,000–5,000 tokens.

Why tokens matter in practice

Context windows. Models can only process a fixed number of tokens at a time — their "context window." GPT-4 Turbo supports 128,000 tokens; Claude 3 supports up to 200,000. If your prompt plus the expected response exceeds the context window, the request fails or the model truncates the input.

API cost. Most LLM APIs charge per token — separately for input tokens (what you send) and output tokens (what the model generates). A prompt that seems short in words can be expensive if it contains many uncommon words that tokenize into multiple tokens each.

Rate limits. API rate limits are often expressed in tokens per minute or per day, not in requests. A single large request counts heavily against token-based rate limits.

Tokenizers differ by model

Different model families use different tokenizers. The same text might be 100 tokens in GPT-4 and 90 tokens in Claude, because each model's tokenizer has a different vocabulary. This matters when you're comparing costs across providers or trying to fit content within a specific context window on a specific model.

Non-English languages tokenize differently. Most tokenizers were trained primarily on English text. Chinese, Japanese, Arabic, and other languages may produce significantly more tokens per word than English, which affects cost and context window usage for multilingual applications.

Counting tokens before sending

Counting tokens before sending a request helps you: avoid hitting context window limits, estimate API cost accurately, and compare prompt variants efficiently. An AI token counter shows you the count for the major tokenizer families side by side, without having to send a request and pay for the count.