GPT-4V(ision) is a Generalist Web Agent, if Grounded

SeeAct studies GPT-4V as a generalist web agent and adds an online evaluation setup for running agents on live websites. It shows that GPT-4V is strong when grounding is handled manually, and identifies grounding as the main remaining bottleneck.

• Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective
  March 15, 2026 · arXiv
• Enhancing Web Agents with a Hierarchical Memory Tree
  March 7, 2026 · arXiv
• OpeFlo: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding
  February 25, 2026 · arXiv
• ColorBrowserAgent: Complex Long-Horizon Browser Agent with Adaptive Knowledge Evolution
  January 12, 2026 · arXiv
• WebATLAS: An LLM Agent with Experience-Driven Memory and Action Simulation
  October 26, 2025 · NeurIPS 2025 Workshop on Language Agents and World Models
• Surfer 2: The Next Generation of Cross-Platform Computer Use Agents
  October 22, 2025 · arXiv
• PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction
  October 17, 2025 · arXiv
• CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation
  April 30, 2025 · NAACL 2025 (System Demonstrations)
• Infogent: An Agent-Based Framework for Web Information Aggregation
  April 29, 2025 · Findings of NAACL 2025
• SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills
  April 9, 2025 · arXiv
• Inducing Programmatic Skills for Agentic Tasks
  April 9, 2025 · COLM 2025
• LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applications
  March 4, 2025 · NAACL 2025 System Demonstrations
• WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point
  February 12, 2025 · arXiv
• WebWalker: Benchmarking LLMs in Web Traversal
  January 13, 2025 · arXiv
• Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery For Foundation Model Internet Agents
  December 17, 2024 · ICML 2025 (Poster)
• The BrowserGym Ecosystem for Web Agent Research
  December 6, 2024 · TMLR
• AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
  November 24, 2024 · ACL 2025
• From Grounding to Planning: Benchmarking Bottlenecks in Web Agents
  September 3, 2024 · ECAI 2025
• WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration
  August 28, 2024 · AAAI 2025
• Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
  July 17, 2024 · arXiv
• VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
  April 9, 2024 · COLM 2024
• OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
  February 12, 2024 · LLMAgents @ ICLR 2024
• Dual-View Visual Contextualization for Web Navigation
  February 6, 2024 · CVPR 2024 (Poster)
• CogAgent: A Visual Language Model for GUI Agents
  December 14, 2023 · CVPR 2024 (Highlight)