See how AI thinks
An instrument for watching a real GPT-2 forward pass — attention, hidden states, and, below, the model’s answer taking shape one layer at a time. Nothing here is a simulation.
The animal didn't cross the street because it was tired→ ?
An intermediate guess — the deeper layers have not settled here yet.
Measured sample captured from this project’s own backend. The logit lens decodes the residual stream into vocabulary space at every layer, so each row is what the model “would say” if asked to stop and answer at that depth. Every other view in the lab is fed live.
What you can inspect
- Transformer layers
- 12
- Heads per layer
- 12
- Attention heads
- 144
- Residual width
- 768dims
- MLP neurons / layer
- 3.1K
- Parameters
- 124.4M
- Vocabulary
- 50.3Ktokens
- Context window
- 1.0Ktokens
Stacked blocks. The lens falls through all 13 depths, embedding to output.
Independent attention circuits running side by side in every block.
Isolate one and see exactly which earlier tokens it reads from.
The channel each token's state travels through, end to end.
Post-GELU features; the strongest firings surface per token.
Every weight in the model — few enough to hold the whole thing in view.
The space the lens decodes into at each layer.
How far back attention can reach in a single pass.
Where to point it
Run any prompt and move one slider through the whole stack — tokens, attention, hidden states, and the lens all re-read at the depth you choose.
OpenPull a single head out of the 144 and watch which earlier tokens it binds to — trace how the word “it” finds the noun it refers to.
OpenFly through the 768-dimensional token space projected into 3D and measure the geometry the model has learned between words.
OpenGenerate one token at a time and watch the probability distribution collapse onto a choice at every step.
Open