RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents

RiOSWorld measures misuse risk for multimodal desktop and web agents in realistic interactive settings rather than ordinary chat-style safety probes. Its 492 risky tasks score both harmful intent and harmful task completion, showing that current computer-use agents remain highly exposed to real-world misuse despite strong task-solving ability.

• The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents
  April 12, 2026 · arXiv
• When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents
  February 9, 2026 · arXiv
• When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents
  February 9, 2026 · arXiv
• WebTrap Park: An Automated Platform for Systematic Security Evaluation of Web Agents
  January 13, 2026 · arXiv
• It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents
  December 29, 2025 · arXiv
• DECEPTICON: How Dark Patterns Manipulate Web Agents
  December 28, 2025 · arXiv
• Investigating the Impact of Dark Patterns on LLM-Based Web Agents
  October 20, 2025 · IEEE S&P 2026
• macOSWorld: A Multilingual Interactive Benchmark for GUI Agents
  June 4, 2025 · NeurIPS 2025 (Poster)
• Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents
  October 11, 2024 · arXiv
• ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
  October 9, 2024 · ICLR 2026 (Poster)
• Dissecting Adversarial Robustness of Multimodal LM Agents
  June 18, 2024 · ICLR 2025 (Poster)
• CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation
  April 10, 2026 · arXiv
• LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios
  February 3, 2026 · arXiv
• VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks
  December 18, 2025 · arXiv
• OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models
  December 18, 2025 · arXiv
• RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments
  May 28, 2025 · ICLR 2026 (Oral)
• On the Robustness of GUI Grounding Models Against Image Attacks
  April 7, 2025 · arXiv
• WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point
  February 12, 2025 · arXiv
• Attacking Vision-Language Computer Agents via Pop-ups
  November 4, 2024 · ACL 2025
• MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control
  October 23, 2024 · arXiv
• GUI Action Narrator: Where and When Did That Action Take Place?
  June 19, 2024 · arXiv
• GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
  June 16, 2024 · ICLR 2025 (Poster)
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
  April 11, 2024 · NeurIPS 2024 Datasets and Benchmarks Track
• OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
  February 29, 2024 · ECCV 2024 (Poster)