Next word: how a language model “thinks”
GPT-2 (OpenAI, 2019, 124 million parameters) runs in your browser. Your text never leaves this computer. The model knows only English.
1 · Write the start of a sentence (in English)
2 · How the model sees the text: tokens
The model reads tokens, not words: pieces of words. Letters outside English break into bytes (odd characters), so a sentence in another language costs many more tokens. Try yours.
3 · Probability of the next token (top 10)
Click a bar to add that token. Low temperature: the model almost always picks the same. High: more chance.
Loading the model…
4 · Let the model write on
Click again: at the same temperature the result is often different. The model draws from a distribution.
What follows
The model does not check facts. It picks the tokens that usually follow such a text. Hence answers that sound sure and are sometimes invented, such as the text of Article 54 of the ICCPR, which does not exist.
Model: Xenova/gpt2 (ONNX, about 130 MB, downloaded once) via transformers.js. The full inside of a model, visualised: Transformer Explainer. Global Campus of Human Rights, Ł. Szoszkiewicz.