Probe to Act: Elevating Browser-Use Agent via Active Visual Probing

Keliang Li , Heng Wang , Chen Hu , Daxin Jiang , Hong Chang , Shiguang Shan

🏛 Institutions: Institute of Computing Technology, CAS , University of Chinese Academy of Sciences , StepFun
📅 Date: September 27, 2026
📑 Publisher: Findings of EMNLP 2026
💻 Env: Web
🔑 Keywords: P2A Set-of-Marks memory VisualWebArena long-horizon tasks

TLDR

Probe to Act (P2A) lets a browser-use agent issue lightweight probes at decision time that translate DOM handles into pixel evidence and map screen regions back to DOM candidates, keeping only probed or committed observations as memory. It works as a prompting strategy for proprietary models and can be distilled into open-weight models, improving results on three browser-use benchmarks.

Open paper Report issue