Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions

Xunjian Yin , Tianchen Guan , Jinao Wang , Weili Cao , Daisy Xinlei Lin , Royce Cheng-Yue , Keagan Long , Kyle Wong , Bhuwan Dhingra , Xiangjun Wang , Shuyan Zhou

🏛 Institutions: Duke University , Amazon
📅 Date: September 20, 2026
📑 Publisher: arXiv
💻 Env: Web
🔑 Keywords: BreakingWeb benchmark browser-use agent robustness failure analysis

TLDR

BreakingWeb builds harder browser-use tasks from tasks that agents already solve, by pairing each base task with a deterministic environment intervention at one web stack layer that keeps the instruction and the backend success criterion unchanged. Its 519 clean/intervention task pairs over seven self-hosted websites lower agent pass rates by 22.9% on average, and 75% of the failures of six browser-use agents end with a declared success although the required change never happened.

Open paper arXiv Report issue