Skip to main content
Search

# GUI Grounding

4 related articles

Subscribe to updates
  1. arXiv

    A 4B multimodal model reads and writes mobile UIs: fine-grained perception lifts generation score to 4.92

    UI-UG 2 is a 4B-parameter multimodal large language model that trains mobile UI understanding and generation in one pipeline. It annotates over 30,000 screenshots from RICO, VINS, Android in the Wild and Android Control with an 80-category two-level taxonomy aligned to front-end component semantics, paired with more than 5,000 requirement–screenshot–JSON tuples. Generation is optimized by online render-aware GRPO: sampled DSL outputs are rendered by a Playwright server-side pipeline, then scored by an LLM judge and contrastive preference ranking. On its own 80-category benchmark it gives the strongest referring and dense-grounding results among evaluated models, and its composite generation score of 4.92 is the highest among compared models.

    Hao Yang（First author）
    UI-UG 2Mobile UI Understanding and GenerationGRPOGUI GroundingLLM-as-a-Judge
  2. arXiv

    How a Small GUI Grounding Model Should Receive the Action Type: Auxiliary Loss, Additive Embedding, and Prompt Word Each Gain 5–7 hit@0.10 Points, but Much of the Gain Comes from a Serialization Artifact

    Fine-tuning Qwen2-VL-2B with LoRA on Android in the Wild, the study compares five ways of supplying the action type to a small GUI grounding model under matched data, compute, and decoding, finding that an auxiliary loss, an additive embedding, and a prompt word each raise hit@0.10 by five to seven points, but after retraining without the degenerate class created by the serializer no mechanism beats the baseline on hit rate, and only the auxiliary loss shortens the average miss.

    Stanford University
    Aadi Chauhan（First author）
    GUI GroundingAction Type ConditioningQwen2-VL-2BLoRAGUI Agent
  3. arXiv

    ReLaViS lifts ScreenSpot-Pro grounding accuracy by 3.1 points to 56.3% via single-round recurrent latent visual search

    The work proposes ReLaViS, which formulates GUI grounding as recurrent latent visual search within a single model interaction: at each step a spatial search head uses the hidden state to query the screenshot's visual tokens, producing a spatial search distribution that aggregates those tokens into latent visual evidence fed back as the next input embedding, supervised by GUI-aware coarse-to-fine trajectories built from flat element annotations; built on Qwen2.5-VL-7B, it outperforms matched single-step baselines on all five benchmarks, raising ScreenSpot-Pro accuracy from 53.2% to 56.3% with only 3.5% more inference FLOPs.

    Kaiyu Wu（First author）
    Recurrent Latent Visual SearchGUI GroundingReLaViS-7BLatent Visual ReasoningQwen2.5-VL
  4. arXiv

    MintAct: A Unified Visual Agent for Digital Environments

    MintAct introduces a family of vision-language models at 2B, 4B, and 8B scales that unify UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use in a single set of weights through a shared screenshot-and-pixel observation space, prompt-conditioned per-domain action sets, and balanced cross-domain mixing, backed by asynchronous RL infrastructure hosting hundreds of concurrent instances, reaching 48.9 on OSWorld-Verified, 39.1 on Online-Mind2Web, and 67.0 on AndroidWorld while matching or exceeding size-matched per-domain specialists.

    Mingfei Gao（First author）
    MintActGUI GroundingVisual Tool UseAsynchronous RL

You’re all caught up