Next word: how a language model “thinks”

GPT-2 (OpenAI, 2019, 124 million parameters) runs in your browser. Your text never leaves this computer. The model knows only English.

1 · Write the start of a sentence (in English)

2 · How the model sees the text: tokens

The model reads tokens, not words: pieces of words. Letters outside English break into bytes (odd characters), so a sentence in another language costs many more tokens. Try yours.

3 · Probability of the next token (top 10)

Click a bar to add that token. Low temperature: the model almost always picks the same. High: more chance.

Loading the model…

4 · Let the model write on

Click again: at the same temperature the result is often different. The model draws from a distribution.

What follows

The model does not check facts. It picks the tokens that usually follow such a text. Hence answers that sound sure and are sometimes invented, such as the text of Article 54 of the ICCPR, which does not exist.

Model: Xenova/gpt2 (ONNX, about 130 MB, downloaded once) via transformers.js. The full inside of a model, visualised: Transformer Explainer. Global Campus of Human Rights, Ł. Szoszkiewicz.