Evaluating Skills, Not Just Agents：Skill 评估不能只看 SKILL.md

Xudong Han (@Xudong07452910) · 2026-08-25 Linked paper: Christopher Kevin et al. (NVIDIA) · arXiv:2608.20614 · 2026-08-20

Source post

• Post: https://x.com/Xudong07452910/status/2092093357518368797
• Status ID: 2092093357518368797
• Author: Xudong Han (@Xudong07452910)
• Bio: PhD ing @ University of Sussex | LLM & AI Agents; AI Agent Product Development; Sharing AI tech insights
• Published: 2026-08-25T03:35:00Z
• Engagement at fetch: likes=40, reposts=9, quotes=0, replies=3, bookmarks=45, views=2702
• Format: X post recommending a paper (full text via x_thread_fetch; paper converted from PDF with npx @firecrawl/anydoc)
• Media (remote, not saved): https://pbs.twimg.com/media/HQibpJwXAAAxkL5.jpg
• Linked original: https://arxiv.org/abs/2608.20614
• Linked PDF: https://arxiv.org/pdf/2608.20614
• Open-source: https://github.com/NVIDIA/SkillEvaluator
• Local paper: arxiv-2608-20614-evaluating-skills-not-just-agents

Post body (@Xudong07452910)

一个 Skill 写得很漂亮，通过了所有静态检查，真正交给 Agent 用时却可能完全没帮助。

NVIDIA 这篇《Evaluating Skills, Not Just Agents》讨论的就是 Skill 应该怎么评估，适合收藏起来使用。

现在很多 Skill 主要检查 SKILL.md 写得是否规范、描述是否清楚、有没有安全问题。

但真正上线后，还要看 Agent 会不会找到它、能不能正确调用工具、是否按流程执行，以及最终任务有没有做得更好。

作者提出 ACES，直接做一组很简单的对照实验：

同一个任务、模型和环境，一次给 Agent Skill，一次不给，然后比较两条真实执行轨迹。

这个差值被定义为「Skill Lift」。

在 145 个真实 Skill 上，94.5% 都能通过基础结构检查，但不同静态评分之间的相关性只有 0.14。

进一步跑 947 个配对任务后，72.8% 的案例获得正向 Skill Lift，同时也有 87 个案例出现负向效果。

所以这篇工作很适合现在越来越大的 Skill 生态。

我们评价一个 Skill，不能简单的只问「写得好不好」，还要真正跑起来看它到底有没有让 Agent 做得更好。

Skill 也需要像代码一样，有自己的测试集、回归测试和 CI。

Replies at fetch

Thread fetch returned one unrelated spam reply; omitted.

Linked original (paper)

Full paper preserved as text-extracted markdown (PDF converted with npx @firecrawl/anydoc; original binary not kept): arxiv-2608-20614-evaluating-skills-not-just-agents

• Title: Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
• Authors: Christopher Kevin, Narendran Raghavan, Jean-Francois Puget, Roshni Malani, Meghana Puvvadi, Moshe Abramovitch, Mohit Gupta, Rama Akkiraju, Subodh Prabhu, Yogesh Dangi, Wei Luo, Seong Hee Lee (NVIDIA)
• arXiv: 2608.20614 (cs.AI), submitted 2026-08-20
• Abstract: Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. On 145 real skills, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman ρ=0.14). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95% paired-case CI [0.1967, 0.2301]); mean outcome-only lift is 0.1799. Composite lift is positive in 72.8% of paired cases; 87 cases show negative lift. An open-source implementation is NVIDIA SkillEvaluator.