You Don’t Know Until You Click: Automated GUI Testing for Production-Ready Software Evaluation
Yutong Bian , Xianhao Lin , Yupeng Xie , Tianyang Liu , Mingchen Zhuge , Siyuan Lu , Haoming Tang , Jinlin Wang , Jiayi Zhang , Jiaqi Chen , Xiangru Tang , Yongxin Ni , Sirui Hong , Chenglin Wu
- 🏛 Institutions
- DeepWisdom , Fudan , HKUST(GZ) , UC San Diego , KAUST , Westlake University , Stanford , Yale University , NUS
- 📅 Date
- August 17, 2025
- 📑 Publisher
- SEA @ NeurIPS 2025 (Poster)
- 💻 Env
- General GUI
- 🔑 Keywords
TLDR
RealDevWorld is an evaluation framework for repository-scale software generation that judges whether produced applications actually work when interacted with through their GUIs. It pairs a 194-task benchmark, RealDevBench, with AppEvalPilot, an agent-as-a-judge system for functional, visual, and runtime evaluation, and reports strong alignment with expert human assessments.
Related papers (24)
- SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use SafetySeptember 28, 2026 · arXiv
- CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use AgentsMarch 11, 2026 · HEAL @ CHI 2026 Workshop
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-JudgeSeptember 18, 2025 · NeurIPS 2025 Datasets & Benchmarks Track (Poster)
- Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI AgentsOctober 29, 2026 · arXiv
- Right Click, Wrong Box: Grading GUI Grounding by What a Click DoesOctober 27, 2026 · arXiv
- Learning to Look Beyond Screens: Transferable Visual Observation from GUI GroundingOctober 27, 2026 · arXiv
- From Interaction to Capability: Task Advancement as State and Supervision for GUI AgentsOctober 27, 2026 · arXiv
- AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI GroundingOctober 27, 2026 · arXiv
- GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI AgentsOctober 26, 2026 · arXiv
- PreView: Enhancing GUI Grounding with Preprocessed Views and CuesOctober 24, 2026 · arXiv
- Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI TestingOctober 24, 2026 · arXiv
- IGG: A Benchmark for Interactive GUI Grounding under Visibility ConstraintsOctober 23, 2026 · arXiv
- Correct Answers, Closed Gates: SolveBind for GUI Agents at the Verification BarrierOctober 23, 2026 · arXiv
- Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent TrainingOctober 22, 2026 · arXiv
- PointerBench: A Diagnostic Benchmark for Fine-Grained GUI GroundingOctober 22, 2026 · arXiv
- Who Verifies the Agent's Success Claim? Self-Report Leakage in GUI-Agent Completion JudgingOctober 21, 2026 · arXiv
- Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action ReplayOctober 20, 2026 · arXiv
- ScopeMem: Latent Memory That Learns Where to Revise Frozen GUI AgentsOctober 19, 2026 · arXiv
- Focused Omni-Modal GUI AgentsOctober 19, 2026 · arXiv
- Comparing Computer Use Behavior Across Multiple Levels of AbstractionOctober 19, 2026 · arXiv
- Agent World Model in the Wild: Computer-Use Agent Environment SynthesisOctober 19, 2026 · arXiv
- Beyond Next-Screen Prediction: The Decision Value of GUI World ModelsOctober 17, 2026 · arXiv
- MMA-SafetyBench: A Benchmark for Multimodal Agent Safety EvaluationOctober 14, 2026 · arXiv
- GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement LearningOctober 13, 2026 · arXiv