ClawBench: Can AI Agents Complete Everyday Online Tasks?

Yuxuan Zhang , Yubo Wang , Yipeng Zhu , Penghui Du , Junwen Miao , Xuan Lu , Wendong Xu , Yunzhuo Hao , Songcheng Cai , Xiaochen Wang , Huaisong Zhang , Xian Wu , Yi Lu , Minyi Lei , Kai Zou , Huifeng Yin , Ping Nie , Liang Chen , Dongfu Jiang , Wenhu Chen , Kelsey R. Allen

🏛 Institutions: UBC , Vector Institute , CMU , UWaterloo , SJTU , ZJU , HKUST , Tsinghua
📅 Date: April 9, 2026
📑 Publisher: arXiv
💻 Env: Web
🔑 Keywords: benchmark realistic website long-horizon tasks ClawBench

TLDR

ClawBench evaluates browser agents on 283 everyday online tasks (V1 153 + V2 130) across 163 live production websites spanning purchases, bookings, and job applications. A lightweight interception layer blocks final submissions for safe evaluation while preserving end-to-end interaction. Its two-stage interception and judge protocol exposes a persistent gap in real-world web automation.

Open paper arXiv Code Dataset Report issue