Computer-use agents stop swallowing every screenshot: VTR co-trains a router to keep 2.5–5.8× fewer visual tokens at full-resolution performance
Synopsis
Computer-use agents encode each screenshot into thousands of visual tokens, and those tokens pile up into long contexts on long-horizon tasks and in real-time games. VTR trains a vision-language model jointly with a lightweight spatio-temporal router that inspects the same image region in the current and previous screenshots and decides whether that token enters the language model, using an adaptive loss to keep as few tokens as possible without losing action prediction. After only 2,000 training steps, VTR matches the full-resolution model on OSWorld, CUA-World and video-game action prediction while using 2.5–5.8× fewer visual tokens, cutting inference cost by up to 3.3× and speeding prefill by up to 2.9×.
Interpretation
A computer-use agent can process only the visual tokens it actually needs instead of sending every one of a screenshot's thousands of tokens into the language model. Earlier approaches either kept a fixed fraction of the most-attended tokens or dropped tokens unchanged across frames by a fixed threshold, so how many tokens survived was unrelated to the agent's next action. On OSWorld, VTR scores 18.1 against 9.4 for the H.264 rule and 5.3 for pixel-diff; against the dense model at half the image area, VTR scores 2.9 points higher on OSWorld while using fewer tokens.
The router decides whether to keep a region by looking at that same region in the current and previous screenshots, and dropped tokens are removed from the input sequence entirely. The router is a three-layer MLP with 23.3M parameters, about 1% of the language model; it concatenates visual tokens at the same position and outputs a keep probability, and kept tokens are scaled by that probability so the discrete gate still receives a gradient. Training mixes 4,265 computer-use examples from the CUA-World training split with 29,401 video-game examples from markov-ai/gaming-500-hours at a 1:3 ratio, on Qwen3.5-2B, with a fully kept anchor every 10 observations.
How many tokens to keep is decided by the router per input rather than fixed in advance. Training is posed as a constrained problem: minimize the kept fraction while limiting the accuracy lost relative to a reference model that sees only the anchors to 10% of the gain the full history provides, with the weight adjusted during training by dual ascent. The same router keeps about 40% of tokens on OSWorld but only 17% in video games; grouped by screen change, it keeps 2.7% in low-motion games, 5.9% in high-motion games and 26.3% on desktop tasks.
Perspective
The demonstrated setting is Qwen3.5-2B trained jointly for 2,000 steps on CUA-World training trajectories and game recordings, evaluated on 295 internet-free OSWorld tasks, 512 CUA-World tasks and about 90,000 held-out actions from 36 games; inference cost is priced per million input, cached-input and output tokens, and prefill time is measured with vLLM on one NVIDIA L40S GPU.
The reported difference between VTR and the full-resolution dense model on OSWorld and CUA-World is small and not statistically significant; cost and prefill gains are smaller than the reduction in visual tokens because the instruction, prior actions and generated action remain; the router also keeps a few isolated regions in static areas, which the material attributes to gates being sampled from their keep probabilities.