Token
In the context of large language models, a token is the basic unit of text that the model processes — roughly corresponding to a word, part of a word, or punctuation character.
Definition¶
A token is the smallest unit of text that a language model processes. Tokens don't correspond exactly to words — they are determined by a process called tokenization that splits text into subword units. Common words like "the," "is," and "run" are typically single tokens. Longer or less common words may be split: "humanization" might become "human" + "ization" (two tokens). Punctuation marks are usually individual tokens.
Why Tokens Matter¶
Language model pricing, context limits, and processing speed are all measured in tokens, not words or characters.
As a rough approximation: - 1 token ≈ 4 characters in English - 100 tokens ≈ 75 words - 1,000 tokens ≈ 750 words
A 5,000-character text input (AI Humanizer's limit) is approximately 1,250 tokens.
Context Window and Tokens¶
The context window of a language model is the maximum number of tokens it can consider at once — both the input (your text plus any instructions) and the output (the generated response). Models with larger context windows can handle longer documents.
GPT-4's context window is 128,000 tokens (~96,000 words). Claude 3 Sonnet's is 200,000 tokens (~150,000 words).
Tokenization Differences¶
Different models use different tokenizers. The same text may produce slightly different token counts in GPT-4 vs. Claude vs. Llama. This matters when comparing API costs across providers.