Public articles linked to the same research event.
arXiv The work proposes the UI-in-the-Loop (UILoop) paradigm, which models GUI reasoning as a cyclic Screen-UI elements-Action process in which Multimodal Large Language Models explicitly learn the localization, semantic functions, and practical usage of key UI elements, enabling precise element discovery and interpretable reasoning; it further introduces a more challenging UI Comprehension task with three evaluation metrics and contributes a 26K-sample UI Comprehension-Bench, with experiments reporting state-of-the-art UI understanding and superior results on GUI reasoning tasks.
The work proposes the UI-in-the-Loop (UILoop) paradigm, which models GUI reasoning as a cyclic Screen-UI elements-Action process in which Multimodal Large Language Models explicitly learn the localization, semantic functions, and practical usage of key UI elements, enabling precise element discovery and interpretable reasoning; it further introduces a more challenging UI Comprehension task with three evaluation metrics and contributes a 26K-sample UI Comprehension-Bench, with experiments reporting state-of-the-art UI understanding and superior results on GUI reasoning tasks.
The work proposes the UI-in-the-Loop (UILoop) paradigm, which models GUI reasoning as a cyclic Screen-UI elements-Action process in which Multimodal Large Language Models explicitly learn the localization, semantic functions, and practical usage of key UI elements, enabling precise element discovery and interpretable reasoning; it further introduces a more challenging UI Comprehension task with three evaluation metrics and contributes a 26K-sample UI Comprehension-Bench, with experiments reporting state-of-the-art UI understanding and superior results on GUI reasoning tasks.
The work proposes the UI-in-the-Loop (UILoop) paradigm, which models GUI reasoning as a cyclic Screen-UI elements-Action process in which Multimodal Large Language Models explicitly learn the localization, semantic functions, and practical usage of key UI elements, enabling precise element discovery and interpretable reasoning; it further introduces a more challenging UI Comprehension task with three evaluation metrics and contributes a 26K-sample UI Comprehension-Bench, with experiments reporting state-of-the-art UI understanding and superior results on GUI reasoning tasks.