One token at a time

Here is the whole job of a language model, however large:

1. Look at the text so far (the context).
2. Give every token in its vocabulary a probability of coming next.
3. Pick one. Add it to the text.
4. Go back to step 1.

Every essay, poem, program and answer you have seen a chatbot write came out of that loop, one token at a time. There is no separate “plan the answer” step. Whatever planning a model does has to happen inside step 2.

A toy with the real knobs

The model below is deliberately tiny and old-fashioned. It’s a character n-gram: it counted which character followed each short run of characters in a single story about a panda. Real LLMs are transformers working over subword tokens and trained on a large slice of the internet, but the interface is the same: context in, probabilities out, pick one. Its knobs are the real knobs.

Things to try:

• Temperature 0. The model always picks the single most likely token. It’s predictable, and it soon falls into loops. Researchers who studied this found that always picking the most likely text gives output that is “bland and strangely repetitive”.
• Temperature 2. The probabilities flatten, unlikely tokens get picked, and you get nonsense.
• Context: 1 char vs 8 chars. With one character of memory it produces gibberish shaped like English. With eight it often copies whole phrases from its story, because it has seen so little text that a long context nearly always matches just one place. That’s memorisation, the problem from Memorising is not learning.
• Top-k 1. Same as temperature 0: only the best token survives the cut.