Token

A token is the small chunk of text that a language model actually reads and writes. It is usually a word or a piece of a word. For example, "cat" might be one token, while "unbelievable" might split into "un," "believ," and "able." Common words are often single tokens; rare or long words get broken into pieces.

Tokens are the unit of measurement for almost everything in language models. Data size is counted in tokens, context length is counted in tokens, and you are typically billed per token when using a commercial model. A rough rule of thumb in English: one token is about three quarters of a word.