On the Robustness of GUI Grounding Models Against Image Attacks

Haoren Zhao, Tianyi Chen, Zhen Wang

🏛 Institutions: HDU, Microsoft
📅 Date: April 7, 2025
📑 Publisher: arXiv
💻 Env: Desktop Mobile Web
🔑 Keywords: benchmark robustness natural noise adversarial attacks UGround ScreenSpot-V2

TLDR

This paper benchmarks the robustness of GUI grounding models under natural noise, untargeted attacks, and targeted attacks across mobile, desktop, and web interfaces. It finds that current models such as UGround remain highly sensitive to adversarial perturbations and low-resolution conditions, exposing a major reliability gap for practical GUI use.

Open paper arXiv Edit on GitHub Report issue