The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge

Alexander Gill , Md Farhan Ishmam , Xuyen Nguyen , Neha Bhat , Parker Henry DeYoung , Fateme Hashemi Chaleshtori , Nathan Stringham , Kenneth Marino , Ana Marasović

🏛 Institutions: University of Utah
📅 Date: September 24, 2026
📑 Publisher: Findings of EMNLP 2026
💻 Env: Web
🔑 Keywords: KNOWS benchmark long-horizon tasks artifact generation

TLDR

KNOWS is a benchmark of open-ended, long-horizon, browser-based tasks in which an agent retrieves information, synthesizes it, and produces an artifact such as a document, presentation, or spreadsheet, scored by evaluators that combine deterministic checks with LLM judgments. The best frontier computer-use agent or browser harness fully succeeds on fewer than 3% of tasks, and failures on visual steps make the produced artifacts unusable.

Open paper arXiv Code Dataset Report issue