A mini GPT you fly through
A small language model, and beside it a city of everything it read. When it answers, it walks into that city and points at the lines that back it, or says ABSENT.
The model is Karpathy's nanoGPT, copied from a public Kaggle notebook and trained the same way on the same Shakespeare: the simple example, taken apart here so you can see how an LLM works. Fly with the tour below, or drag, scroll and press W A S D to go anywhere. Point at a building to read its speech.
Loading the weights and the training text...
1 · The model
2 · Take it apart
Each character goes in at the left, climbs the six blocks, and comes out as chances for the next one. Inside each block, six attention heads each look back at the characters before; the bars in the city show where the head you pick looks.
What the model has read, each character shaded by how much the chosen head looks at it now:
3 · Its priors
4 · Ask it
The model proposes, the priors check, and every line the answer rests on lights in the city.
5 · Let it write
Each character is shaded by how long a stretch ending there the training text holds:nonea few charactersa whole phrase. Point at one for its training line.
6 · It grows
7 · The numbers
8 · Its Blueprint
Every graph of the engine, lighting as it runs. Fly through all of them in the Blueprint City.
Click a node to read the line of code it was read from.
9 · How it works