VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought
Gabriel Herbert Sarch , Lawrence Jang , Michael J. Tarr , William W. Cohen , Kenneth Marino , Katerina Fragkiadaki
- 🏛 Institutions
- CMU , Google DeepMind
- 📅 Date
- June 20, 2024
- 📑 Publisher
- NeurIPS 2024 (Spotlight)
- 💻 Env
- 🔑 Keywords
TLDR
ICAL turns sub-optimal demonstrations and feedback into reusable multimodal memories that improve VLM and LLM agents across TEACh, VisualWebArena, and Ego4D. It is relevant to GUI work because one evaluation domain is web agents, but the method itself is a broader embodied-agent memory approach rather than a direct GUI paper.
Related papers (8)
- From Interaction to Capability: Task Advancement as State and Supervision for GUI AgentsOctober 27, 2026 · arXiv
- ScopeMem: Latent Memory That Learns Where to Revise Frozen GUI AgentsOctober 19, 2026 · arXiv
- Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI AgentsOctober 1, 2026 · arXiv
- Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and TrajectoriesOctober 1, 2026 · NeurIPS 2026 Workshop on Who Verifies the Agents? Toward Reliable Agent Development
- Action Conditioned Bisimulation For GUI Agent MemorySeptember 30, 2026 · arXiv
- PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience RetrievalSeptember 29, 2026 · arXiv
- Relevance Does Not Imply Applicability: Experience Activation for Personal GUI AgentsSeptember 27, 2026 · arXiv
- Probe to Act: Elevating Browser-Use Agent via Active Visual ProbingSeptember 27, 2026 · Findings of EMNLP 2026