In this demo the numbers are hand-picked. In a real model they start out random and are learned in a phase called training. The model reads an enormous amount of text (books, websites, code) and, at every point, plays the same game you just watched: guess the next word.
Each time it guesses wrong, every one of its billions of numbers is nudged slightly so the right word would have scored a little higher. Repeat that across trillions of words, and the numbers end up capturing grammar, facts and patterns of reasoning. Nobody writes those rules in. They emerge from getting better at guessing the next word.
Chat assistants like ChatGPT and Claude then get extra training on example conversations and human feedback, so they answer questions helpfully instead of just continuing text.
This demo is based on the Transformer, the design behind today's AI language models, introduced in "Attention Is All You Need" (Vaswani et al., 2017).
To keep it small, it leaves out parts of a real model: words are split into sub-word pieces rather than whole words, the attention step is followed by a "thinking" layer and repeated dozens of times, and the final guess here gets a nudge from a short hand-written list of common word pairs that a real model would learn on its own.
