Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision

This paper argues that sparse grounding supervision is insufficient for GUI understanding and introduces ScreenParse, a large-scale densely annotated screen-parsing dataset. It provides complete UI-element supervision across web screenshots to support richer grounding and UI understanding models.

• OmniParser for Pure Vision Based GUI Agent
  August 1, 2024 · arXiv
• GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning
  October 13, 2026 · arXiv
• Beyond Clicking: A Step Towards Generalist GUI Grounding via Text Dragging
  November 7, 2025 · arXiv
• Scaling Computer‑Use Grounding via User Interface Decomposition and Synthesis
  May 19, 2025 · NeurIPS 2025 Datasets and Benchmarks Track (Spotlight)
• UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis
  April 15, 2025 · Findings of ACL 2025
• EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
  October 25, 2024 · arXiv
• ScreenAI: A Vision-Language Model for UI and Infographics Understanding
  February 7, 2024 · IJCAI 2024
• DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents
  October 1, 2026 · arXiv
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
  October 30, 2024 · ICLR 2025 (Spotlight)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
  October 7, 2024 · ICLR 2025 (Oral)
• MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  September 22, 2024 · arXiv
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
  January 17, 2024 · ACL 2024
• Right Click, Wrong Box: Grading GUI Grounding by What a Click Does
  October 27, 2026 · arXiv
• Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding
  October 27, 2026 · arXiv
• AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding
  October 27, 2026 · arXiv
• PreView: Enhancing GUI Grounding with Preprocessed Views and Cues
  October 24, 2026 · arXiv
• IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints
  October 23, 2026 · arXiv
• PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding
  October 22, 2026 · arXiv
• Beyond Next-Screen Prediction: The Decision Value of GUI World Models
  October 17, 2026 · arXiv
• SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents
  September 29, 2026 · arXiv
• ScreenHaystack: Finding Blind Zones in GUI Grounding
  September 25, 2026 · EMNLP 2026
• How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation
  September 21, 2026 · EMNLP 2026
• RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
  September 16, 2026 · ACM MM 2026
• Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
  September 14, 2026 · arXiv