PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

Dheeraj Varghese , Anna Vettoruzzo , Walter Simoncini , Michelle Lorena Acevedo Callejas , Mohammad Mahdi Derakhshani , Kristof Meding , Joaquin Vanschoren , Cees G. M. Snoek

🏛 Institutions: University of Amsterdam , Eindhoven University of Technology
📅 Date: October 5, 2026
📑 Publisher: arXiv
💻 Env: Desktop Web
🔑 Keywords: PlaySuite benchmark interactive visual intelligence games computer-use agents VLM

TLDR

Introduces PlaySuite, a benchmark evaluating interactive visual agents across 5K+ open-source video games (Pygame, HTML5, Godot, Unity) using a closed-loop execution harness and Video-LLM judge. Evaluation of 14 vision-language models and computer-use agents reveals a substantial perception-action gap, with agents failing consistently on dynamic spatial grounding and real-time self-correction.

Open paper Report issue