Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

Video2GUI is an automated pipeline for extracting grounded GUI interaction trajectories from unlabeled Internet videos. It constructs WildGUI, a dataset of 12 million interaction trajectories across more than 1,500 applications and websites, and uses it for generalized GUI-agent pretraining.

• VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos
  October 22, 2025 · arXiv
• OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
  December 27, 2024 · ACL 2025
• Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents
  July 2025 · Findings of ACL 2025
• Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision
  February 15, 2026 · arXiv
• GUIGuard: Toward a General Framework for Privacy-Preserving GUI Agents
  January 26, 2026 · arXiv
• Beyond Clicking: A Step Towards Generalist GUI Grounding via Text Dragging
  November 7, 2025 · arXiv
• Scaling Synthetic Task Generation for Agents via Exploration
  September 29, 2025 · ICLR 2026 (Poster)
• Scaling Computer‑Use Grounding via User Interface Decomposition and Synthesis
  May 19, 2025 · NeurIPS 2025 Datasets and Benchmarks Track (Spotlight)
• TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents
  April 17, 2025 · AAAI 2026
• UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis
  April 15, 2025 · Findings of ACL 2025
• Falcon-UI: Understanding GUI Before Following User Instructions
  December 12, 2024 · arXiv
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
  December 5, 2024 · ICML 2025 (Poster)
• EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
  October 25, 2024 · arXiv
• OmniParser for Pure Vision Based GUI Agent
  August 1, 2024 · arXiv
• Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
  June 27, 2024 · EMNLP 2024 (Poster)
• VGA: Vision GUI Assistant - Minimizing Hallucinations through Image-Centric Fine-Tuning
  June 20, 2024 · Findings of EMNLP 2024
• GUICourse: From General Vision Language Model to Versatile GUI Agent
  June 17, 2024 · ACL 2025
• ScreenAI: A Vision-Language Model for UI and Infographics Understanding
  February 7, 2024 · IJCAI 2024
• SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models
  May 30, 2023 · NeurIPS 2023
• WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark
  April 13, 2026 · arXiv
• MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
  April 9, 2026 · arXiv
• Gym-Anything: Turn any Software into an Agent Environment
  April 7, 2026 · arXiv
• WebArena-Infinity: Generating Browser Environments with Verifiable Tasks at Scale
  March 2026 · Blog Post
• PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent
  March 31, 2026 · arXiv