Shuo Wang

Web-GUI agents locate buttons and links from screenshots, and high-resolution screenshots plus multi-turn histories create excessive visual tokens that raise inference cost. STP-GUI replaces independent token scoring with query-guided, spatially coherent window-level selection, trained through a lightweight selector while the vision-language backbone stays frozen. On Online-Mind2Web, retaining 21% of tokens preserves 77.7% of full-token performance while cutting inference latency by 31% and FLOPs by 57%; results on Android in the Wild suggest the method transfers to mobile GUI environments.

Web-GUI agents locate buttons and links from screenshots, and high-resolution screenshots plus multi-turn histories create excessive visual tokens that raise inference cost. STP-GUI replaces independent token scoring with query-guided, spatially coherent window-level selection, trained through a lightweight selector while the vision-language backbone stays frozen. On Online-Mind2Web, retaining 21% of tokens preserves 77.7% of full-token performance while cutting inference latency by 31% and FLOPs by 57%; results on Android in the Wild suggest the method transfers to mobile GUI environments.