Are CAPTCHAs Still Bot-hard? Generalized Visual CAPTCHA Solving with Agentic Vision Language Model

Xiwen Teoh , Yun Lin , Siqi Li , Ruofan Liu , Avi Sollomoni , Yaniv Harel , Jin Song Dong

🏛 Institutions: SJTU , NUS , Tel Aviv University
📅 Date: August 2025
📑 Publisher: USENIX Security 2025
💻 Env: Web
🔑 Keywords: Halligan CAPTCHA benchmark visual reasoning

TLDR

Halligan reduces visual CAPTCHA solving to a search problem over a standardized metamodel of frames, elements, and keypoints, letting one agentic vision-language model generalize across CAPTCHA types without per-challenge adaptation. It solves 60.7% of 2,600 challenges spanning 26 visual CAPTCHAs, and 70.6% of previously unseen CAPTCHAs in a 30-day field study, arguing that current visual CAPTCHA defenses are no longer reliably bot-hard.

Open paper Publisher Code Report issue