GUI agents stop rebuilding future screens: latent prediction lifts success up to 40.6% and cuts world-model latency by 98.9%

GUI world modeling moves its prediction target from full-screen reconstruction to action-induced future information in latent space, so action selection no longer needs a render-then-re-perceive pipeline. Pixel-space world models first generate a next screenshot or renderable HTML for each candidate action and then hand it to a separate scorer for re-perception, an intermediate step that adds high computational latency and requires reconstructing appearance details action selection does not need. Across offline AndroidControl and AITZ, online AndroidWorld, and out-of-distribution GUI-Odyssey, the method improves success rate over pixel-space world models by up to 31.9% and 40.6% relative, while reducing world-model inference latency by up to 98.9%.

One shared latent representation serves two goals: an action head maps each candidate action to a score, and a transition head predicts what future state the chosen action leads to. A LoRA-adapted large multimodal model encodes the goal, current screenshot, history, and candidate actions; the action head is trained with cross-entropy against the annotated action, and the transition head aligns with the embedding of the real next screenshot during training and is discarded at inference. Training uses AndroidControl and AITZ with Qwen3-VL-8B-Instruct, LoRA rank 16, three candidate actions, and six steps of history, taking eight hours on a single A6000 GPU.

To suppress the identity shortcut that preserves the current screen, transition representations are pushed away from the current state and from unrelated transitions. With point-wise matching alone, predictions can stay close to the current screen instead of capturing action-induced change; pixel-space world models show this shortcut more strongly as transition magnitude grows, and their downstream success rate drops accordingly. The action-aware contrastive objective uses the true next state as positive and the current state plus a first-in-first-out memory bank of other transitions as negatives, and the ablation reports a further online success-rate gain from adding contrast.

The work targets action selection for mobile GUI agents and is evaluated on offline AndroidControl and AITZ, online AndroidWorld, and out-of-distribution GUI-Odyssey, covering backbones including GPT-5.6, Gemini-3.6-Flash, GUI-Owl-7B, Qwen3-VL-8B, and Qwen3-VL-32B. The model only re-ranks candidate actions proposed by a frozen base agent and does not generate new actions, and it is invoked only when a confidence-based gate identifies a low-confidence decision. The authors propose expanding candidate coverage, replacing the fixed confidence threshold with a learned uncertainty-aware invocation policy, and extending single-step latent transitions to multi-step latent dynamics.

The authors note that the model cannot recover the correct action when it is absent from the candidate set, so performance is bounded by the quality and coverage of the base agent's proposals; high-confidence base-agent errors bypass latent world-model evaluation, and terminal decisions such as completing or abandoning a task are not rescored in the current protocol; the transition objective models only the immediate action-conditioned next state, does not explicitly reason over longer-horizon consequences, and provides no interpretable reconstruction of predicted future states.