VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents

Xiao Liu , Tianjie Zhang , Yu Gu , Iat Long Iong , Yifan Xu , Xixuan Song , Shudan Zhang , Hanyu Lai , Xinyi Liu , Hanlin Zhao , Jiadai Sun , Xinyue Yang , Yu Yang , Zehan Qi , Shuntian Yao , Xueqiao Sun , Siyi Cheng , Qinkai Zheng , Hao Yu , Hanchen Zhang , Wenyi Hong , Ming Ding , Lihang Pan , Xiaotao Gu , Aohan Zeng , Zhengxiao Du , Chan Hee Song , Yu Su , Yuxiao Dong , Jie Tang

🏛 Institutions: Tsinghua University , Zhejiang University , Peking University , The Ohio State University
📅 Date: August 12, 2024
📑 Publisher: ICLR 2025
💻 Env
🔑 Keywords: benchmark dataset visual foundation agents embodied tasks visual design VisualAgentBench VAB

TLDR

VisualAgentBench benchmarks large multimodal models as general visual foundation agents across embodied tasks, GUI tasks, and visual design rather than focusing only on GUI interaction. It also releases trajectory data for behavior cloning, making it relevant to GUI work as a broader visual-agent benchmark rather than a direct GUI paper.

Open paper Report issue