Same Login Screen, Opposite Labels: LGWM Checks the Transition in 17 ms, Not the Screen

Safety checking for GUI agents shifts from inspecting the screen to inspecting the transition: a credential request that follows a tap on Sign in is expected, while the same screen after a tap on View order is an attack, and nothing on the screen distinguishes them. Earlier GUI world models output their predictions as text, screenshots or executable code, so comparing a prediction with the observed screen required a second model to judge the two, and that judge had to be trusted in turn. RSWT-Bench makes each credential screen appear both as the true outcome of a legitimate transition and, pixel-identically, as the hijacked outcome of another, so any detector that sees only the outcome screen has an expected AUC of exactly 0.5.

LGWM places its prediction of the next screen in the same representation space used by the observation encoder, so whether the outcome matched expectation reduces to one cosine comparison between predicted and observed vectors, with no decoder and no judge model. It carries over a joint-embedding predictive architecture but turns the predicted representation from a training-time pretext into the test-time product; on the action side, touch coordinates and visual patches share one two-dimensional sinusoidal positional basis, so where a tap lands enters the prediction. The model is pretrained self-supervised on 1.85 million real GUI transitions from five public sources, with a 261M-parameter training graph of which 172.86M execute at inference; the training-free consistency score reaches 0.987 AUC on RSWT-Bench, ordering 219 of the 222 donor pairs correctly.

The residual between prediction and observation is a vector whose magnitude answers whether the transition departed from expectation and whose direction answers what the departure consists of, the latter read by a linear head that separates harmful from benign violations. A push notification, a loading spinner or a server-side layout change also violates expectation without harming the user, so magnitude alone cannot support a deployable monitor. On Harm AUC between hijacked transitions and benign mismatches, the training-free consistency score reaches only 0.513 and prompted vision-language models range from 0.512 to 0.614, while the linear head on the residual direction reaches 0.953; in the other direction, the consistency score is the stronger statistic for wrong-transition detection, 0.987 versus 0.930 for the residual head.

The demonstrated scope is transition verification for GUI agents, evaluated on the 222 donor pairs of RSWT-Bench, with training on 1.85 million real transitions from five public sources and latency measured at batch size one on a single A100-80GB. The authors propose that the latency is low enough to run this gate on every step of an agent's trajectory, invoking a renderer or vision-language-model judge only on the flagged minority.

The authors note that nothing in the score is specific to attacks: the same residual also flags missed actions, layout changes or unexpected modals. They also state that an expectation-based monitor requires an adversary to forge an outcome in the agent's own representation space, a harder objective whose difficulty they have not yet quantified; the ablations use one seed at a smaller scale, with a three-seed replication confirming small cross-seed variance.