Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

Introduces Set-of-Mark prompting, where segmented image regions are overlaid with explicit marks before being passed to a multimodal model. The paper shows that simple region marking can unlock much stronger zero-shot grounding from GPT-4V without fine-tuning.

• Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
  October 12, 2026 · arXiv
• When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks
  October 9, 2026 · arXiv
• Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution
  October 7, 2026 · arXiv
• DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis
  September 26, 2026 · arXiv
• CocoaBench: Evaluating Unified Digital Agents in the Wild
  April 13, 2026 · arXiv
• Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection
  April 9, 2026 · arXiv
• ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents
  January 30, 2026 · arXiv
• How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors
  January 29, 2026 · arXiv