Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution

Wanjing Han , Levi Taiji Li , Mu Zhang , Yue Jiang , Guanhong Tao

🏛 Institutions: University of Utah
📅 Date: October 7, 2026
📑 Publisher: arXiv
💻 Env: Web
🔑 Keywords: adversarial attack web agent visual grounding browser execution red-teaming

TLDR

Studies end-to-end visual adversarial attacks targeting multimodal web agents from visual grounding down to browser execution. Demonstrates that imperceptible adversarial image perturbations on webpages consistently hijack agent attention and cause targeted misclicks across Mind2Web and VisualWebArena.

Open paper Report issue