Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection

This paper proposes Semantic-level UI Element Injection, a red-teaming method that overlays safety-aligned UI elements onto screenshots to misdirect GUI agents' visual grounding. Using a modular Editor-Overlapper-Victim pipeline with iterative search, optimized attacks improve attack success rate by up to 4.4x over random injection and transfer across models.

• When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents
  February 9, 2026 · arXiv
• AdvAgent: Controllable Blackbox Red-teaming on Web Agents
  October 22, 2024 · ICML 2025 (Poster)
• Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents
  October 11, 2024 · arXiv
• MMA-SafetyBench: A Benchmark for Multimodal Agent Safety Evaluation
  October 14, 2026 · arXiv
• AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents
  September 6, 2026 · arXiv
• SIR: Self-improving Red-teaming for Compute Use Agents
  August 31, 2026 · arXiv
• Human-Guided Harm Recovery for Computer Use Agents
  April 20, 2026 · arXiv
• CocoaBench: Evaluating Unified Digital Agents in the Wild
  April 13, 2026 · arXiv
• LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios
  February 3, 2026 · arXiv
• SafePred: A Predictive Guardrail for Computer-Using Agents via World Models
  February 2, 2026 · arXiv
• ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents
  January 30, 2026 · arXiv
• iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception
  December 26, 2025 · arXiv
• GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents
  May 19, 2025 · arXiv
• A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?
  May 16, 2025 · arXiv
• OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use
  December 20, 2024 · ACL 2025
• Visual Grounding for User Interfaces
  June 16, 2024 · NAACL 2024 Industry Track
• MobileJailBench: Red Queen Jailbreak Benchmarking for Mobile Agents
  October 16, 2026 · arXiv
• Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
  October 12, 2026 · arXiv
• When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks
  October 9, 2026 · arXiv
• Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution
  October 7, 2026 · arXiv
• Control OSWorld: An AI Control Environment for GUI Computer Use Agents
  October 2, 2026 · arXiv
• SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety
  September 28, 2026 · arXiv
• Before Acting, Change the State: Prospective State Intervention for Web Agents under Deceptive Interfaces
  September 28, 2026 · arXiv
• DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis
  September 26, 2026 · arXiv