Interaction-as-a-Judge
Wayne Chi , I-Hung Hsu , Li Chen , Wei-Lin Chiang , Ion Stoica , Anastasios Nikolas Angelopoulos , Ameet Talwalkar , Chris Donahue , Cho-Jui Hsieh
- 🏛 Institutions
- CMU , UC Berkeley , UCLA
- 📅 Date
- October 13, 2026
- 📑 Publisher
- arXiv
- 💻 Env
- Desktop Web
- 🔑 Keywords
TLDR
Proposes Interaction-as-a-Judge, an evaluation paradigm where computer-use agents actively interact with web applications, desktop software, and games to evaluate interactive functionality. Releases 6,858 application-trace pairs and 222 human-verified requirement benchmarks, demonstrating improved alignment with human quality assessments over static code-based evaluation.
Related papers (24)
- AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use TasksOctober 8, 2026 · arXiv
- Secure-CUA: Controlling Untrusted Influence in Computer-Use AgentsOctober 7, 2026 · arXiv
- PlaySuite: A Large-Scale Benchmark for Interactive Visual IntelligenceOctober 5, 2026 · arXiv
- What Does "Now" Mean to Your Computer-Use Agent? Stale Screenshots, Page Loads, and Which Check to Run Before a ClickOctober 29, 2026 · arXiv
- CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use AgentsOctober 20, 2026 · arXiv
- PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse PersonasOctober 16, 2026 · arXiv
- A Baseline Assessment for the Agentic Web Technology StackOctober 9, 2026 · arXiv
- SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software EnvironmentsOctober 8, 2026 · arXiv
- Control OSWorld: An AI Control Environment for GUI Computer Use AgentsOctober 2, 2026 · arXiv
- Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and TrajectoriesOctober 1, 2026 · NeurIPS 2026 Workshop on Who Verifies the Agents? Toward Reliable Agent Development
- WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web AgentsSeptember 28, 2026 · arXiv
- Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search SystemsApril 9, 2026 · arXiv
- CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use AgentsMarch 11, 2026 · HEAL @ CHI 2026 Workshop
- WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph RepresentationOctober 22, 2025 · NeurIPS 2025 Workshop on Multi-Turn Interactions in Large Language Models
- An Illusion of Progress? Assessing the Current State of Web AgentsApril 2, 2025 · COLM 2025
- Who Verifies the Agent's Success Claim? Self-Report Leakage in GUI-Agent Completion JudgingOctober 21, 2026 · arXiv
- Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action ReplayOctober 20, 2026 · arXiv
- Beyond Next-Screen Prediction: The Decision Value of GUI World ModelsOctober 17, 2026 · arXiv
- Towards Safe Autonomy for Computer UseOctober 13, 2026 · arXiv
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon AgentsOctober 13, 2026 · arXiv
- Learning How to Remember: Memory Modality Selection for GUI AgentsOctober 13, 2026 · arXiv
- Adaptive-Group Process Rewards for GUI AgentsOctober 13, 2026 · arXiv
- PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI AgentsOctober 12, 2026 · arXiv
- Learn from Move: the Next Step for GUI AgentsOctober 12, 2026 · arXiv