OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web

OmniACT introduces a dataset and benchmark for agents that must generate executable programs from a screenshot and a visually grounded natural-language task. It covers both desktop and web applications, and the paper reports that GPT-4 reaches only about 15% of human proficiency on the benchmark.

• GUI Action Narrator: Where and When Did That Action Take Place?
  June 19, 2024 · arXiv
• GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
  June 16, 2024 · ICLR 2025 (Poster)
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
  January 17, 2024 · ACL 2024
• WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark
  April 13, 2026 · arXiv
• Gym-Anything: Turn any Software into an Agent Environment
  April 7, 2026 · arXiv
• WebArena-Infinity: Generating Browser Environments with Verifiable Tasks at Scale
  March 2026 · Blog Post
• WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces
  March 5, 2026 · arXiv
• When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents
  February 9, 2026 · arXiv
• ShowUI-π: Flow-based Generative Models as GUI Dexterous Hands
  December 31, 2025 · arXiv
• GUI-360: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  November 6, 2025 · arXiv
• Web-CogReasoner: Towards Knowledge-Induced Cognitive Reasoning for Web Agents
  August 3, 2025 · ICLR 2026 (Poster)
• NaturalGAIA: Pushing the Frontiers of GUI Agents with a Challenging Benchmark and High-Quality Trajectory Dataset
  August 2, 2025 · arXiv
• Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
  May 21, 2025 · NeurIPS 2025 (Spotlight)
• Efficient Agent Training for Computer Use
  May 20, 2025 · ICLR 2026 (Poster)
• RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users
  April 14, 2025 · AAAI 2026
• AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
  April 11, 2025 · COLM 2025
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
  March 19, 2025 · ICML 2025 (Poster)
• DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents
  March 14, 2025 · arXiv
• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
  July 15, 2024 · NeurIPS 2024 Datasets and Benchmarks Track (Poster)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
  July 7, 2024 · NeurIPS 2024 Datasets and Benchmarks Track (Poster)
• WebCanvas: Benchmarking Web Agents in Online Environments
  June 18, 2024 · arXiv
• On the Multi-turn Instruction Following for Conversational Web Agents
  February 23, 2024 · ACL 2024
• WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
  February 8, 2024 · ICML 2024
• WebVLN: Vision-and-Language Navigation on Websites
  December 25, 2023 · AAAI 2024