Beyond Clicking:

Graphical user interface (GUI) grounding, the process of mapping human instructions to GUI actions, serves as a fundamental basis to autonomous GUI agents. While existing grounding models achieve promising performance to simulate the mouse click action on various click-based benchmarks, another essential mode of mouse interaction, namely dragging, remains largely underexplored. Yet, dragging the mouse to select and manipulate textual content represents a prevalent and important usage in practical GUI scenarios. To narrow this gap, we first introduce GUI-Drag, a diverse dataset of 161K text dragging examples synthesized through a scalable pipeline. To support systematic and robust evaluation, we further construct Screen-Drag, a benchmark with 5,333 examples spanning three levels of interface context, together with three dedicated metrics designed for assessing text dragging capability. Models trained on GUI-Drag with an efficient continual training strategy achieve substantial improvements on Screen-Drag, while preserving the original click-based performance on ScreenSpot, ScreenSpot-v2, and OSWorld-G. Our work encourages further research on broader GUI grounding beyond just clicking and paves way toward a truly generalist GUI grounding model.

• We design an scalable pipeline to synthesize text dragging examples directly from screenshots. It contains three stages: 1) Instruction Generation, which covers five categories of instructions for referencing text spans in five granularities, 2) Grounding, which leverages the OCR module to ground the instruction into pixel-level coordinates, and 3) Filtering, which filters out the examples that have ambiguous intention or have incorrect grounding.
• We carefully analyze and filter existing GUI grounding datasets, including UGround and Jedi, to retain screenshots that are rich in textual content. In addition, we collect 20K public academic-style documents to better reflect real-world scenarios where text-dragging capabilities are frequently used, particularly in academic and document-editing contexts.
• Overall, we synthesize 161K high-quality text dragging examples, which we denote as GUI-Drag.
• Unlike prior works that train grounding models from a general foundation model with tons of examples, we adopt an efficient continual training strategy to enhance the text dragging capability while preserving the original click-based performance of the base model. Our models, trained based on Jedi-3B/7B, are denoted as GUI-Drag-3B/7B..

• ScreenDrag crafts 5,333 human-annotated examples across three popular productivity applications: Word, PowerPoint, and PDF reader, where the text dragging is a core part of user workflows.
• The screenshots within ScreenDrag span three levels of interface context — document view, application window, and full desktop. This design captures diverse real-world usage scenarios of GUI agents, particularly when employing MCP techniques.
• The instructions within ScreenDrag are also categorized into different granularities and categories, and are formatted in both explicit and implicit forms.
• We further split the benchmark into text-sparse and text-dense subsets to quantify the fine-grained localization difficulty.

ScreenDrag reports complementary metrics that jointly capture action triggering, spatial alignment, and exact selection quality.

• Drag Trigger Rate (DTR): frequency of producing a complete dragging trajectory.
• Boundary Distance (B-Dist): average index differences between predicted and ground-truth boundary boxes.
• Selection Rate (SR): percentage of cases where the selected text span matches the gold annotation exactly.

1. Bias toward clicking. Existing grounding models are heavily biased toward click actions, even when the instructions explicitly require dragging.

   

2. Continual training effectiveness. Continual training effectively enhances text dragging capability while preserving the original click-based performance.

   

3. Practical agent gains. Case studies on OSWorld show GUI-Drag enables agents to follow natural strategies instead of relying on peculiar shortcuts.

   OSWorld case study success rates (SR).

   Model	OpenAI CUA	Claude CUA	o3	o3 + GUI-Drag-7B
   SR	2/3	2/3	0/3	3/3