SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environments
Syed Yusuf Ahmed , Shiwei Feng , Chanwoo Bae , Calix Barrus Xiangyu Zhang
- 🏛 Institutions
- Purdue University , University of Texas at San Antonio
- 📅 Date
- March 10, 2026
- 📑 Publisher
- ICSE 2026
- 💻 Env
- Desktop Web
- 🔑 Keywords
TLDR
SpecOps is a fully automated testing framework that uses four specialist agents to generate cases, set up environments, execute tasks, and validate outcomes for real-world software agents. Across five deployed agents spanning CLI tools, web apps, and browser extensions, it finds 164 true bugs with 0.89 F1 while keeping each test under eight minutes and under $0.73.
Related papers (24)
- Towards Safe Autonomy for Computer UseOctober 13, 2026 · arXiv
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon AgentsOctober 13, 2026 · arXiv
- Learning How to Remember: Memory Modality Selection for GUI AgentsOctober 13, 2026 · arXiv
- Interaction-as-a-JudgeOctober 13, 2026 · arXiv
- Adaptive-Group Process Rewards for GUI AgentsOctober 13, 2026 · arXiv
- PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI AgentsOctober 12, 2026 · arXiv
- Learn from Move: the Next Step for GUI AgentsOctober 12, 2026 · arXiv
- Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill InductionOctober 9, 2026 · arXiv
- Latent World Action Model for GUI AgentsOctober 9, 2026 · arXiv
- Right Screen, Wrong Transition: World Models as Verifiers for GUI AgentsOctober 8, 2026 · arXiv
- AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use TasksOctober 8, 2026 · arXiv
- Secure-CUA: Controlling Untrusted Influence in Computer-Use AgentsOctober 7, 2026 · arXiv
- PlaySuite: A Large-Scale Benchmark for Interactive Visual IntelligenceOctober 5, 2026 · arXiv
- Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent TrainingOctober 5, 2026 · arXiv
- TeleTune: Evolving Agent Skills From Offline TelemetryOctober 4, 2026 · arXiv
- CUAWright: A Minimal Unified Interface for Digital AgentsOctober 2, 2026 · arXiv
- CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement LearningSeptember 26, 2026 · arXiv
- RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use AgentsSeptember 18, 2026 · arXiv
- MintAct: A Unified Visual Agent for Digital EnvironmentsSeptember 18, 2026 · arXiv
- LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI AgentsSeptember 9, 2026 · arXiv
- Qwen-CUA: Native Computer Use for (almost) EverythingAugust 3, 2026 · arXiv
- MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action DistillationJuly 31, 2026 · arXiv
- Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI AgentsJuly 30, 2026 · arXiv
- OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsJuly 30, 2026 · arXiv