OmniParser for Pure Vision Based GUI Agent

Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah

🏛 Institutions: MSR, Microsoft GenAI
📅 Date: August 1, 2024
📑 Publisher: arXiv
💻 Env: General GUI
🔑 Keywords: dataset screen parsing GUI grounding icon detection icon captioning OmniParser

TLDR

OmniParser parses UI screenshots into structured screen elements by combining interactable icon detection with element captioning. The paper also curates icon-related datasets and shows that this screen parsing layer improves GPT-4V grounding on ScreenSpot, Mind2Web, and AITW.

Open paper arXiv Edit on GitHub Report issue

Related papers

Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision

February 15, 2026 · arXiv
Beyond Clicking: A Step Towards Generalist GUI Grounding via Text Dragging

November 7, 2025 · arXiv
Scaling Computer‑Use Grounding via User Interface Decomposition and Synthesis

May 19, 2025 · NeurIPS 2025 Datasets and Benchmarks Track (Spotlight)
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis

April 15, 2025 · Findings of ACL 2025
EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

October 25, 2024 · arXiv
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

October 30, 2024 · ICLR 2025 (Spotlight)