A 4B multimodal model reads and writes mobile UIs: fine-grained perception lifts generation score to 4.92
Synopsis
UI-UG 2 is a 4B-parameter multimodal large language model that trains mobile UI understanding and generation in one pipeline. It annotates over 30,000 screenshots from RICO, VINS, Android in the Wild and Android Control with an 80-category two-level taxonomy aligned to front-end component semantics, paired with more than 5,000 requirement–screenshot–JSON tuples. Generation is optimized by online render-aware GRPO: sampled DSL outputs are rendered by a Playwright server-side pipeline, then scored by an LLM judge and contrastive preference ranking. On its own 80-category benchmark it gives the strongest referring and dense-grounding results among evaluated models, and its composite generation score of 4.92 is the highest among compared models.
Interpretation
Mobile UI understanding and generation can now be carried by one 4B-parameter multimodal model, covering referring, dense grounding, GUI grounding and structured generation. The earlier unified model UI-UG used a 7B backbone, organized labels without explicit front-end component semantics, covered mainly referring and dense grounding on the understanding side, and optimized generation with offline DPO over a fixed preference dataset. On the in-house 80-category benchmark, UI-UG 2-4B reaches 0.906 referring category accuracy and 0.777 dense-grounding F1, against 0.535 and 0.471 for the strongest general baseline, Gemini 3.1 Pro; its composite generation score is 4.92, above Claude Sonnet 4.5 at 4.70 and GPT-5.2 at 4.33.
Generation quality is optimized by actually rendering model outputs and scoring them, rather than learning only from a fixed preference dataset. Offline preference optimization sees static preference samples, leaving a gap to the interfaces actually rendered at inference time. Each sampled DSL is rendered online by a Playwright server-side pipeline, then scored by Qwen3-VL-235B-A22B on color and aesthetic design, visual structure and alignment, textual and content consistency, and interface usability, with contrastive preference ranking inside each rollout group; the composite generation score rises from 4.65 for the SFT model to 4.92.
Fine-grained UI perception supervision transfers to generation, improving generation quality under a fixed training budget. Earlier transfer evidence was preliminary, and understanding and generation had long evolved along separate data pipelines and evaluation protocols. Under a fixed SFT budget of 120K samples, generation peaks at 4.760 when 20% of the data is understanding data, above 4.668 for generation-only data; understanding metrics rise monotonically with the understanding ratio, with referring accuracy from 0.789 to 0.838 and grounding F1 from 0.475 to 0.690.
Perspective
Results cover mobile interfaces: understanding is evaluated on the in-house 80-category corpus, GUI grounding on the mobile splits of ScreenSpot V1 and V2, and generation by JSON format compliance, render success rate and four MLLM-scored dimensions.
Generation rewards and reported scores share four visual-quality dimensions, so optimizing against that rubric may overstate improvements outside it; the human-reference study covers only 100 cases and is post-hoc rather than a continuous audit during GRPO. Attention similarity is correlational evidence and does not by itself establish the causal mechanism of transfer, and some isolated understanding rewards reduce other generation dimensions.