Tokenization

Tokenization is the step that chops raw text into tokens before the model sees it. A model cannot read letters directly; it needs a fixed vocabulary of tokens, each mapped to a number. The tokenizer is the tool that performs this split consistently. It is a small but important detail, because how text is split affects how efficiently a model can process different languages and symbols.