How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors

Kuai Yu, Naicheng Yu, Han Wang, Rui Yang, Huan Zhang

🏛 Institutions: Columbia, UC San Diego, UIUC
📅 Date: January 29, 2026
📑 Publisher: arXiv
💻 Env: Web
🔑 Keywords: evaluation pipeline visual attribute factors VAF visual grounding robustness

TLDR

This paper introduces VAF, a controlled evaluation pipeline for measuring how benign UI visual attributes affect web-agent choices while preserving page semantics. Across 48 variants, 5 real websites, and 4 agents, it shows that contrast, size, position, and card clarity strongly influence agent behavior, while several text-style factors matter much less.

Open paper arXiv Edit on GitHub Report issue