Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

This work shows that sentence-embedding similarity used for GUI-element grounding is confounded by lexical coupling -- recovery of the visible on-screen label -- rather than genuine semantic understanding, across both mobile and web interfaces.

• VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks
  December 18, 2025 · arXiv
• ScaleTrack: Scaling and back-tracking Automated GUI Agents
  May 1, 2025 · arXiv
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
  January 21, 2025 · arXiv
• Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  December 2, 2024 · Findings of ACL 2025
• Improved GUI Grounding via Iterative Narrowing
  November 18, 2024 · arXiv
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
  October 30, 2024 · ICLR 2025 (Spotlight)
• TinyClick: Single-Turn Agent for Empowering GUI Automation
  October 9, 2024 · INTERSPEECH 2025
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
  October 7, 2024 · ICLR 2025 (Oral)
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
  January 17, 2024 · ACL 2024
• OpeFlo: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding
  February 25, 2026 · arXiv
• WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements
  February 12, 2026 · arXiv
• AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent
  November 30, 2025 · WACV 2026
• ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search
  May 21, 2025 · arXiv
• Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining
  December 13, 2024 · arXiv
• ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
  October 9, 2024 · SIGDIAL 2025
• MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  September 22, 2024 · arXiv
• Dual-View Visual Contextualization for Web Navigation
  February 6, 2024 · CVPR 2024 (Poster)
• GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
  August 22, 2026 · arXiv
• Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
  August 10, 2026 · arXiv
• GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
  August 4, 2026 · arXiv
• MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation
  July 31, 2026 · arXiv
• Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
  July 30, 2026 · arXiv
• OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
  July 30, 2026 · arXiv
• GUI-C²: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
  May 29, 2026 · arXiv