From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs

Yangzhe Peng , Xiaoyang Wang , Yiyang Zhao , Lijun Wu , Kun He

🏛 Institutions: HUST , Shanghai AI Laboratory , ZJU , Shanghai Innovation Institute
📅 Date: September 26, 2026
📑 Publisher: arXiv
💻 Env: Web
🔑 Keywords: SurfPRM dataset process reward model reward model data synthesis WebArena-Lite WebPRMBench

TLDR

SurfPRM synthesizes process preference data for comparative web process reward models by building an Interaction Element Graph from demonstrations and using it to propose grounded contrastive negative actions, raising the share of grounded minimal contrastive pairs from 24.19% to 74.60%. PRMs trained on the resulting data improve on WebPRMBench and guide step-level search on WebArena-Lite.

Open paper Code Report issue