What are tokens in AI?
Tokens are the small chunks of text, such as words, parts of words or punctuation, that language models read and generate. In English, one token is roughly four characters or three-quarters of a word. Tokens determine how much a model can process and how much it costs.
Example
"Resetting passwords" might be split into tokens like "Reset", "ting", " passwords".
Why tokens matter
| Area | Effect |
|---|---|
| Cost | APIs charge per input and output token |
| Limits | Context window is measured in tokens |
| Speed | More tokens take longer |
Reducing token use
Retrieve only relevant context, summarize history and keep prompts concise.
