Text becomes tokens

Before a language model can read a single word, the text has to become numbers. The step that does that is the tokenizer, and it shapes more of a model’s behaviour than most people realise.

Why not just use letters? Too many tiny steps: a page would be thousands of them, and the model would spend its effort spelling. Why not whole words? There are too many words, new ones appear every day (skibidi, GPT-5, your username), and the model would choke on anything it had never seen.

The answer most modern models use sits in between: subword tokens, learned by an algorithm borrowed from data compression called byte-pair encoding (BPE).

1. Start with every single character as its own token.
2. Count every pair of neighbouring tokens in a big pile of text.
3. Glue the most common pair into one new token. t + h → th.
4. Repeat, thousands of times. th + e → the. + the → the.

Common words end up as one token. Rare words get split into familiar pieces. Nothing is ever unknown: in the worst case a word falls back to single characters.

Train one

This tokenizer has learned up to 400 merges from a short story about a panda learning tech. Drag the slider to see the text with no merges, then more and more. Hover a chip to see its token id, the number the model actually receives.

Try typing things the story never contained: a long made-up word, a number like 1234567, some code, another language. They break into many small tokens. That’s the tokenizer telling you this text was rare in its training data.