Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

🏛 Institutions: East China Normal University
📅 Date: October 12, 2026
📑 Publisher: arXiv
💻 Env: Web Mobile
🔑 Keywords: Seg2Click GUI grounding action masks segmentation visual grounding

TLDR

Reformulates GUI visual grounding as executable-region prediction, training models with bounding-box supervision to output pixel-level action masks rather than single-point coordinates. Seg2Click avoids visual-token grid misalignment and achieves higher localization accuracy on fine-grained and dense interface targets.

Open paper Report issue