Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus

Spotlight is a vision-only mobile UI understanding model that takes a screenshot plus a region of interest instead of relying on view hierarchy input. It is pretrained on about 2.5 million mobile UI screens and then used for widget captioning, screen summarization, command grounding, and related UI modeling tasks.

• SecAgent: Efficient Mobile GUI Agent with Semantic Context
  March 9, 2026 · arXiv
• UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents
  May 27, 2025 · NeurIPS 2025 (Poster)
• SpiritSight Agent: Advanced GUI Agent with One Look
  March 5, 2025 · CVPR 2025 (Poster)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent
  November 26, 2024 · CVPR 2025 (Poster)
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
  October 30, 2024 · ICLR 2025 (Spotlight)
• MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding
  September 23, 2024 · Findings of EMNLP 2024
• CogAgent: A Visual Language Model for GUI Agents
  December 14, 2023 · CVPR 2024 (Highlight)
• DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents
  October 1, 2026 · arXiv
• HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents
  September 29, 2026 · arXiv
• CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs
  September 25, 2026 · arXiv
• MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
  April 9, 2026 · arXiv
• ShowUI-π: Flow-based Generative Models as GUI Dexterous Hands
  December 31, 2025 · arXiv
• OpenCUA: Open Foundations for Computer-Use Agents
  August 12, 2025 · NeurIPS 2025 (Spotlight)
• Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
  May 21, 2025 · NeurIPS 2025 (Spotlight)
• Efficient Agent Training for Computer Use
  May 20, 2025 · ICLR 2026 (Poster)
• STEVE: A Step Verification Pipeline for Computer-use Agent Training
  March 16, 2025 · arXiv
• Falcon-UI: Understanding GUI Before Following User Instructions
  December 12, 2024 · arXiv
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
  December 5, 2024 · ICML 2025 (Poster)
• ScreenAI: A Vision-Language Model for UI and Infographics Understanding
  February 7, 2024 · IJCAI 2024
• Multimodal Web Navigation with Instruction-Finetuned Foundation Models
  May 19, 2023 · ICLR 2024
• Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents
  September 23, 2026 · arXiv
• Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction
  September 22, 2026 · EMNLP 2026
• MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning
  September 19, 2026 · USENIX Security 2026
• RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents
  September 18, 2026 · arXiv