UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis

Xinyi Liu , Xiaoyi Zhang , Ziyun Zhang , Yan Lu

🏛 Institutions: MSR Asia , PKU
📅 Date: April 15, 2025
📑 Publisher: Findings of ACL 2025
💻 Env: General GUI
🔑 Keywords: dataset benchmark instruction synthesis GUI grounding UI-E2I-Synth UI-I2E-Bench

TLDR

UI-E2I-Synth addresses the annotation bottleneck in vision-based GUI grounding by using GPT-4o to synthesize large-scale grounding instructions with varied difficulty and annotation properties. The paper also introduces the UI-I2E-Bench benchmark for evaluating GUI instruction grounding under challenges such as implicit instructions, small elements, and underrepresented element types.

Open paper Report issue