Reason to Play — Representation Viewer

How to read this viewer

What is displayed

Four cards, time-locked to the same gameplay session.

• Top left, gameplay. The actual frames the participant played, reconstructed from their behavioural log. Press play, scrub the line below, or grab the yellow thumb to jump to a specific TR (one fMRI volume = 2 s). The grey thumb on the same line marks the start of the comparison window; everything between the grey and yellow thumb is what the two RDMs below summarise.
• Top right, the LLM's view. The exact text the model receives at each step: a system prompt at the start, then a per-step user message describing the current observation, plus the assistant's stated rationale and chosen action. Highlighting tracks the current TR, so you can see what the model was reading when the participant pressed the corresponding key.
• Bottom left, the human RDM. A symmetric (TR × TR) representational dissimilarity matrix for the chosen brain region of interest, masked to the current window. Dark cells = the BOLD pattern at those two TRs is similar; bright cells = dissimilar. The eight thin coloured bars on the top and left edges are the first eight principal components of the BOLD signal, colour-coded by amplitude.
• Bottom right, the model RDM. Same kind of matrix, but computed from the LLM's hidden-state activations at the chosen layer. The bottom card lets you click through all transformer layers and watch how representational geometry changes with depth; the heat-coloured strip on the side encodes how well each layer explains the human ROI (yellow = weakly explanatory, dark red = strongly explanatory).

RSA vs. encoding performance

The viewer is a representational similarity (RSA) tool by design. RSA compares geometries: two systems are aligned when the pair-wise dissimilarity structure of their state representations matches, regardless of how each system encodes an individual state. We use it here because the eye is far better at spotting structural similarity in two RDMs side by side than at parsing a regression coefficient table: if the human and model matrices light up in the same places, the alignment is doing real work.

We use a different measure for the headline statistical results in the paper: encoding-model performance, i.e. how well a regularised regression from the model's hidden states predicts each fMRI voxel's BOLD time-course. You can think of encoding performance as the correlation between the eight thin principal-component bars on top of the model RDM and the corresponding bars on top of the human RDM: if those coloured strips track each other tick-for-tick, the encoding score is high. RSA is the faster qualitative read; encoding is the rigorous quantitative claim.