OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software

Dingyuan Dai , Heli Qi , Lei Liu , Yinxi Li , Baiding Chen , Zijun Dou , Qingcheng Zeng , Qi Kang , Oliver Sun , Eric Wang , Bo Zhou , Haixin Wang , Yufan Du , Shi Bo , Ruihan Lin , Mengqi Yuan , Dunjie Lu , Steven Dillmann , Yiming Shi , Tina Su , Amy Xin , Minghao Liu , Xi Wang , Xu Huang , Ge Zhang , Pengyu Nie , Zhen Yang , Jie Tang , Juanzi Li , Weihao Xuan , Tianyu Liu

🏛 Institutions: Tsinghua , UCLA , RIKEN AIP , University of Waterloo , CMU , Yale University , Northwestern University , ZJU , UC Berkeley , University of Illinois Chicago , Boston University , HKU , Stanford , New York University , TokenWave.AI , University of Tokyo
📅 Date: September 30, 2026
📑 Publisher: arXiv
💻 Env: Desktop
🔑 Keywords: OSWorld-Science benchmark scientific software OSWorld

TLDR

OSWorld-Science is a benchmark of 146 expert-proposed tasks for computer-use agents on scientific software, covering workflows such as molecular drawing, pathology image analysis, statistical computing, and physical simulation. Execution-based evaluators inspect application states and generated artifacts with partial credit, and 12 vision-language models still struggle even with a strong harness.

Open paper arXiv Report issue