MatToolBench: Benchmarking Multimodal Agents in Real-World Materials Science Workflows
Mei Wu , Rui Xie , Runyu Zhang , Yuqiang Li , Tianfan Fu , Bo Chen , Kai Yu , Xin Chen , Lu Chen
- 🏛 Institutions
- SJTU , Suzhou Laboratory , Shanghai AI Laboratory , BIGAI , Shanghai Innovation Institute
- 📅 Date
- September 29, 2026
- 📑 Publisher
- arXiv
- 💻 Env
- Desktop
- 🔑 Keywords
TLDR
MatToolBench is a real-environment benchmark of 204 tasks across 10 professional materials science tools, covering GUI operation, OriginPro scripting, and database queries in a Windows 11 VM, with expert-defined sub-criteria for partial credit. The best model reaches 25% on GUI tasks, and failures stem from missing domain-specific operational knowledge rather than visual grounding alone.
Related papers (24)
- OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific SoftwareSeptember 30, 2026 · arXiv
- CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use AgentsOctober 20, 2026 · arXiv
- CUA-Lite: An Open Platform for Computer Use AgentsOctober 19, 2026 · arXiv
- PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse PersonasOctober 16, 2026 · arXiv
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer EnvironmentsOctober 14, 2026 · arXiv
- A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software EvolutionOctober 11, 2026 · arXiv
- AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use TasksOctober 8, 2026 · arXiv
- PlaySuite: A Large-Scale Benchmark for Interactive Visual IntelligenceOctober 5, 2026 · arXiv
- MedImageOSWorld: Benchmarking GUI Agents for Medical Image ConsolesOctober 3, 2026 · arXiv
- Control OSWorld: An AI Control Environment for GUI Computer Use AgentsOctober 2, 2026 · arXiv
- Talk2Agent: Benchmarking Voice Interfaces for Text AgentsSeptember 30, 2026 · arXiv
- EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?September 29, 2026 · arXiv
- CUA-SWE: When Computer-Use Agents Meet Visual Software EngineeringSeptember 26, 2026 · arXiv
- OSWorld-Pro: Process-based Evaluation for Computer Use AgentsSeptember 21, 2026 · arXiv
- RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use AgentsSeptember 18, 2026 · arXiv
- CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided DesignSeptember 14, 2026 · arXiv
- CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real TimeSeptember 13, 2026 · arXiv
- JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task CompositionSeptember 9, 2026 · arXiv
- FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?September 7, 2026 · arXiv
- ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across DevicesAugust 25, 2026 · arXiv
- CUADebug: Diagnosing and Repairing Computer-Use Agent FailuresJuly 31, 2026 · arXiv
- OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsJuly 30, 2026 · arXiv
- OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic GroundingJuly 29, 2026 · arXiv
- Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional FieldsJune 9, 2026 · arXiv