When we read a sentence, we see words and meanings.

An AI model processes text differently. It does not necessarily treat every word as one complete unit. Instead, it divides text into smaller units called tokens.

What is a token?

A token is a unit of text that an AI model processes when reading or generating language.

A token may represent a whole word, part of a word, a punctuation mark, or a short group of characters. Spaces can also affect how the surrounding text is divided.

For example, a common word may be represented by one token, while a longer or less familiar word may be divided into several tokens. The exact division depends on the model and the tokenizer it uses.

Why not just use words?

Languages are complicated. They contain long words, rare terms, spelling variations, numbers, punctuation, and different writing systems.

Using flexible pieces of text allows a model to process words it has not encountered as complete units. Instead of treating an unfamiliar word as entirely unknown, the tokenizer can divide it into smaller and more familiar parts.

This approach also helps models work with many languages without requiring a fixed list containing every possible word.

How does this affect you?

A model’s context window is usually measured in tokens rather than words.

If a model supports a context window of 100,000 tokens, that does not mean it can process exactly 100,000 words. The number of words that fit depends on the language, wording, punctuation, and tokenizer.

The same message may produce different token counts when written in English, Arabic, or a mixture of both. Different models and tokenizers may also count the same text differently.

A context window may include your prompt, uploaded text, earlier messages, and the model’s response, depending on how the model or service defines its limits.

What do tokens have to do with cost?

Many AI APIs calculate usage according to the number and type of tokens processed.

Input tokens can include your instructions, files, and relevant conversation history. Output tokens are the tokens generated in the answer. Some services also distinguish cached tokens or reasoning tokens.

More tokens usually mean more API usage, but prices differ by model, provider, and token category. Consumer chat subscriptions are not necessarily billed directly per token.

Understanding tokens therefore helps explain why long documents require more context space and why detailed API requests may cost more.

Do you need to count them?

Usually not.

If you use ChatGPT or a similar service for ordinary conversations, you will rarely need to count tokens yourself.

The concept becomes more important when you work with long documents, build an application using an API, estimate costs, or need to stay within a model’s context limit.

Providers may offer token-counting tools, but you must use the encoding associated with the specific model to obtain an accurate estimate.

In short

A token is a unit of text that an AI model processes and counts.

People see words and sentences. A language model receives a sequence of tokens and uses them to predict and generate text.

Tokens help explain why context has limits, why the same text can have different lengths for different models, and how usage is calculated in many AI APIs.