GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

Zichuan Fu , Shirong Wang , Wenlin Zhang , Guojing Li , Yimin Deng , Jingtong Gao , Junjia Qi , Hanyu Yan , Yefeng Zheng , Xiaopeng Li , Wanyu Wang , Xian Wu , Xiangyu Zhao

🏛 Institutions: CityU , Tencent Jarvis Lab , Westlake University
📅 Date: August 4, 2026
📑 Publisher: arXiv
💻 Env: General GUI
🔑 Keywords: GUI-Lens GUI grounding coarse-to-fine cropping coordinate priming visual verification

TLDR

GUI-Lens turns GUI grounding into an iterative visual-search process: OCR and detected components provide coordinate references, a general-purpose VLM selects progressively enlarged crops, and separate verification gates reject bad crop or click proposals. Across four grounding benchmarks and three VLM backends, the paper reports gains of up to 24.9 percentage points, with GPT-5.5 reaching 87.9% on ScreenSpot-Pro.

Open paper Code Report issue