ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
Joy Chen , Alejandro Castillejo Munoz , Pierluca D'Oro , Yuxuan Sun , Chloe Evans , Joseph Tighe
- 🏛 Institutions
- FAIR at Meta
- 📅 Date
- August 25, 2026
- 📑 Publisher
- arXiv
- 💻 Env
- Mobile Desktop
- 🔑 Keywords
TLDR
ADeptS-Bench is a dual-stream trustworthiness benchmark for computer-use agents on mobile and desktop: a Safety stream with paired benign/malicious tasks and threats embedded in the visual interface, and a Disambiguation stream testing whether agents seek clarification; no model exceeds 80% success while staying under 30% attack success.
Related papers (24)
- ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web AgentsOctober 9, 2024 · ICLR 2026 (Poster)
- MobileJailBench: Red Queen Jailbreak Benchmarking for Mobile AgentsOctober 16, 2026 · arXiv
- Control OSWorld: An AI Control Environment for GUI Computer Use AgentsOctober 2, 2026 · arXiv
- Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial EnvironmentsAugust 25, 2026 · arXiv
- The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use AgentsApril 12, 2026 · arXiv
- CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI AutomationApril 10, 2026 · arXiv
- When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use AgentsFebruary 9, 2026 · arXiv
- When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use AgentsFebruary 9, 2026 · arXiv
- macOSWorld: A Multilingual Interactive Benchmark for GUI AgentsJune 4, 2025 · NeurIPS 2025 (Poster)
- RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use AgentsMay 31, 2025 · NeurIPS 2025 (Poster)
- MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device ControlOctober 23, 2024 · arXiv
- MMA-SafetyBench: A Benchmark for Multimodal Agent Safety EvaluationOctober 14, 2026 · arXiv
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer EnvironmentsOctober 14, 2026 · arXiv
- RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use AgentsSeptember 18, 2026 · arXiv
- JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task CompositionSeptember 9, 2026 · arXiv
- StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt InjectionAugust 6, 2026 · arXiv
- OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsJuly 30, 2026 · arXiv
- PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation AgentsMarch 9, 2026 · arXiv
- LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial ScenariosFebruary 3, 2026 · arXiv
- WebTrap Park: An Automated Platform for Systematic Security Evaluation of Web AgentsJanuary 13, 2026 · arXiv
- It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web AgentsDecember 29, 2025 · arXiv
- DECEPTICON: How Dark Patterns Manipulate Web AgentsDecember 28, 2025 · arXiv
- VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding TasksDecember 18, 2025 · arXiv
- OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic ModelsDecember 18, 2025 · arXiv