Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning

Xinbin Yuan , Jian Zhang , Kaixin Li , Zhuoxuan Cai , Lujian Yao , Jie Chen , Enguang Wang , Qibin Hou , Jinwei Chen , Peng-Tao Jiang , Bo Li

🏛 Institutions: NKU , vivo Mobile Communication Co. , Ltd , NUS , Fudan , East China University of Science and Technology
📅 Date: May 18, 2025
📑 Publisher: NeurIPS 2025 (Poster)
💻 Env: General GUI
🔑 Keywords: reinforcement learning GUI grounding dense policy gradient self-evolutionary finetuning SE-GUI

TLDR

This paper targets visual grounding for GUI agents in complex, high-resolution interfaces where supervised fine-tuning often generalizes poorly. It introduces SE-GUI, an RL-based framework with seed-data curation, dense policy gradients, and self-evolutionary reinforcement finetuning driven by attention maps. With only 3k training samples, the 7B model reaches state-of-the-art results on multiple grounding benchmarks and substantially improves ScreenSpot-Pro performance.

Open paper arXiv Report issue