Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

LBS leverages GUI grounding trajectories as spatial supervision to train vision-language models for active visual perception and multi-turn crop-and-inspect exploration that transfers beyond screens to complex multimodal benchmarks.

• Right Click, Wrong Box: Grading GUI Grounding by What a Click Does
  October 27, 2026 · arXiv
• AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding
  October 27, 2026 · arXiv
• PreView: Enhancing GUI Grounding with Preprocessed Views and Cues
  October 24, 2026 · arXiv
• IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints
  October 23, 2026 · arXiv
• PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding
  October 22, 2026 · arXiv
• Beyond Next-Screen Prediction: The Decision Value of GUI World Models
  October 17, 2026 · arXiv
• GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning
  October 13, 2026 · arXiv
• SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents
  September 29, 2026 · arXiv
• ScreenHaystack: Finding Blind Zones in GUI Grounding
  September 25, 2026 · EMNLP 2026
• How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation
  September 21, 2026 · EMNLP 2026
• RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
  September 16, 2026 · ACM MM 2026
• Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
  September 14, 2026 · arXiv
• GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
  August 22, 2026 · arXiv
• Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
  August 10, 2026 · arXiv
• GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
  August 4, 2026 · arXiv
• GUI-C²: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
  May 29, 2026 · arXiv
• UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
  April 15, 2026 · arXiv
• GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
  April 15, 2026 · arXiv
• See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback
  April 14, 2026 · arXiv
• What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning
  April 8, 2026 · Findings of ACL 2026
• Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
  March 27, 2026 · CVPR 2026
• AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
  March 18, 2026 · arXiv
• Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements
  March 15, 2026 · arXiv
• Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision
  February 15, 2026 · arXiv