GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning

Weitai Kang, Bin Lei, Gaowen Liu, Caiwen Ding, Yan Yan

🏛 Institutions: University of Illinois Chicago, University of Minnesota, Cisco Research
📅 Date: August 6, 2025
📑 Publisher: ICLR 2026 (Poster)
💻 Env: Desktop Mobile Web
🔑 Keywords: GUI visual grounding reinforcement fine-tuning Adversarial KL Factor ScreenSpot GuirlVG

TLDR

GuirlVG studies how to make reinforcement fine-tuning work for GUI visual grounding instead of naively applying standard rule-based RL. It systematically tunes reward design, prediction format, and training setup, adds an Adversarial KL Factor for stabilization, and reports stronger ScreenSpot-family results with only 5.2K training samples.

Open paper arXiv Edit on GitHub Report issue