← All projects

PriorGPT

starting the 3D view How it compares How it works Fly through its Blueprint
Starting the 3D view...

A mini GPT you fly through

PriorGPT

A small language model, and beside it a city of everything it read. When it answers, it walks into that city and points at the lines that back it, or says ABSENT.

The model is Karpathy's nanoGPT, copied from a public Kaggle notebook and trained the same way on the same Shakespeare: the simple example, taken apart here so you can see how an LLM works. Fly with the tour below, or drag, scroll and press W A S D to go anywhere. Point at a building to read its speech.

Loading the weights and the training text...

1 · The model

Six blocks, stacked like towers

2 · Take it apart

Feed it one character at a time

Each character goes in at the left, climbs the six blocks, and comes out as chances for the next one. Inside each block, six attention heads each look back at the characters before; the bars in the city show where the head you pick looks.

What the model has read, each character shaded by how much the chosen head looks at it now:

Its chances for the next character

3 · Its priors

Everything it read, as a city

4 · Ask it

Every answer is checked

The model proposes, the priors check, and every line the answer rests on lights in the city.

5 · Let it write

Its own words, traced

Each character is shaded by how long a stretch ending there the training text holds:nonea few charactersa whole phrase. Point at one for its training line.

6 · It grows

Learning on the fly

7 · The numbers

How it compares with the plain model

8 · Its Blueprint

The code that runs, as graphs

Every graph of the engine, lighting as it runs. Fly through all of them in the Blueprint City.

    Click a node to read the line of code it was read from.

    9 · How it works

    How it works