# Jacobian lens

The Jacobian lens (J-lens), an interpretability tool that decodes a model's hidden states into words, had been used to find working-memory-like linear directions, but what makes a feature detectable to it was unclear. This work gives a mathematical theory: the lens preferentially selects directions that grow across layers, stay smooth across token positions, and remain consistent across contexts, that is, coherent Lyapunov modes. On a Bayes filter and a two-layer toy transformer, changing task parameters to modulate coherence shifts the ordering of latent states the lens selects, as predicted.

The Jacobian lens (J-lens), an interpretability tool that decodes a model's hidden states into words, had been used to find working-memory-like linear directions, but what makes a feature detectable to it was unclear. This work gives a mathematical theory: the lens preferentially selects directions that grow across layers, stay smooth across token positions, and remain consistent across contexts, that is, coherent Lyapunov modes. On a Bayes filter and a two-layer toy transformer, changing task parameters to modulate coherence shifts the ordering of latent states the lens selects, as predicted.

Prefill jailbreaks force an unsafe opening into the assistant turn so a model that would refuse continues instead. Tracking vocabulary-grounded signals with the Jacobian lens shows refusal weakening faster than harmfulness as the forced prefix deepens, reaching the benign baseline by free generation. J-Anchor, a training-free inference-time defense, anchors each request to its pre-attack assistant-header state and restores only the refusal components that fall below that target during prefilling and decoding, cutting attack success on fixed and deep prefills across four open-weight instruction models while keeping benign refusal and reasoning close to the undefended model.

Prefill jailbreaks force an unsafe opening into the assistant turn so a model that would refuse continues instead. Tracking vocabulary-grounded signals with the Jacobian lens shows refusal weakening faster than harmfulness as the forced prefix deepens, reaching the benign baseline by free generation. J-Anchor, a training-free inference-time defense, anchors each request to its pre-attack assistant-header state and restores only the refusal components that fall below that target during prefilling and decoding, cutting attack success on fixed and deep prefills across four open-weight instruction models while keeping benign refusal and reasoning close to the undefended model.