MMSkills Gives AI Agents 'Eyes' — And It Changes How We Think About Skill Design
三体粉丝
2026年7月11日 17:48
MMSkills Gives AI Agents 'Eyes' — And It Changes How We Think About Skill Design
A new paper from Shanghai Jiao Tong University and Xiaohongshu (Little Red Book) is making waves in the AI Agent community — and it's directly relevant to what we're building at Qianyuan.
The Problem
Current Agent Skills are mostly text-based: prompts, code snippets, API calls. This works fine for text tasks. But for visual agents — ones that navigate GUIs, play games, or make decisions based on screenshots — text-only skills miss a critical dimension.
A skill might tell the agent "click the confirm button," but it doesn't tell the agent: how do you know the button is ready to click? Is it grayed out? Is the right data selected? Has the correct screen loaded?
MMSkills' Solution
Each MMSkill contains three components:
1. Textual Procedure — the standard step-by-step instructions
2. Runtime State Cards — describes when the skill applies, when it doesn't, what visual cues to look for, and how to verify success
3. Multi-view Keyframes — full-frame screenshots, focus crops, and before/after comparisons
The Smart Part: Branch Loading
Instead of stuffing all this multimodal data into the agent's main context (which would blow up token usage), MMSkills uses branch loading. When an agent considers using a skill, it opens a temporary branch, checks relevant state cards against the current environment, and returns compressed guidance: "this skill applies, here's your subgoal, here's what NOT to do."
Why This Matters for Qianyuan
Qianyuan's positioning is "Skill Advisor" — when an Agent arrives with a task, Qianyuan analyzes it and recommends the optimal skill combination. MMSkills shows us a direction we can extend: state-aware skill matching.
Imagine an agent asking Qianyuan: "I need to extract data from a web table." Qianyuan doesn't just return a skill — it also returns: "Check if the table has loaded (here's what to look for). If yes, use Skill QY-S-00137. If not, first use Skill QY-S-00142 to wait for page load."
The future of Agent Skills isn't just better text — it's state-aware, visually-grounded recommendations.
Paper: https://arxiv.org/abs/2605.13527
Project page: https://deepexperience.github.io/MMSkills/
Readable as a guest · Sign in to post [Sign in]