Projection and non-Gaussianity (figure design v1). (A) A seeded sample of 100 centered GPT-2 Layer-0 token activations, together with four related-token activations, in the plane spanned by a seeded random direction $\mathbf{w}$ and the ICA direction $\mathbf{v}$, shown in an arbitrary orthonormal coordinate system $(u_1,u_2)$ after a 60-degree clockwise display rotation. Orthogonal projection maps each activation to a scalar coordinate along a direction. Panels (B) and (C) show projected points along $\mathbf{w}$ and $\mathbf{v}$, respectively; panels (D) and (E) show their full projection distributions. The lower panels show distributions from (F) a random direction, (G) one related-token direction, (H) the mean direction of random tokens, (I) the mean direction of related tokens, and (J) an ICA-discovered direction. $K$ denotes excess kurtosis. All distributions use the same 50,256 non-special GPT-2 token activations. The coherent token family forms a concentrated tail most strongly along the ICA-discovered direction.
