SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction

Overview

Agent skills — reusable packages of instructions, references, and scripts — sit in a privileged position: agents are expected to follow and execute them, which turns third-party skills into a potent attack surface. Existing studies capture only single-session attacks and enumerate harms through ad-hoc lists. SkillHarm closes both gaps with a benchmark of skill-based attacks across the full skill-use lifecycle, paired with a systematic taxonomy of skill-relevant risks.

Two lifecycle scenarios

Fixed-Payload Poisoning FPP — a fixed poisoned package compromises any session that invokes it; Self-Mutating Poisoning SMP — a benign first run silently mutates the skill, deferring harm to a later reuse.

12-risk taxonomy

Risks organized by the workflow component harmed — data pipeline, system environment, and agent autonomy — rather than an unstructured list.

AutoSkillHarm

An automated construction pipeline: coding agents driven by natural-language harnesses generate contextualized payloads with deterministic evaluators, at scale.

Current agents remain highly vulnerable — attack success up to 86.3% (FPP) and 69.3% (SMP). Crucially, much apparent resistance is the agent simply never engaging the poisoned file rather than genuine refusal, and standard defenses fail to reliably help.

Key Results

Six model–harness configurations on SkillHarm, across both attack scenarios.

ASR attack success rate (↑ worse) · cASR conditional on engaging the poisoned file · ARR attack refusal rate (↑ safer)

Key findings

Apparent safety is mostly non-engagement. cASR exceeds ASR by up to 32.1 points — many "failures" are the agent never touching the poisoned file, not resisting it; better skill-use could raise real harm.
Explicit refusal is rare (ARR < 5% for most agents). Only Claude-family agents refuse noticeably, and refusal collapses under SMP, where temporal decoupling hides intent.
Scripts beat documents. Script-mediated payloads reach much higher ASR (88%) than document payloads (47%) — agents execute helper scripts without inspecting them.
Defenses don't hold. Skill scanners and defensive system prompts leave nearly every configuration above 70% ASR.

Threat Model & Attack Scenarios

A poisoned skill is published and installed; an agent later loads its instructions, references, and scripts. Harm can land in the same session, or be deferred to a later reuse.

SkillHarm: two attack scenarios — Fixed-Payload Poisoning and Self-Mutating Poisoning.

Fixed-Payload Poisoning FPP

The attacker publishes a skill whose malicious payload is present at installation and remains fixed thereafter. The payload may live in SKILL.md, reference documents, helper scripts, or attacker-added artifacts. When an agent invokes the poisoned skill for a relevant task, the payload can realize the harmful outcome within that same session.

Captures direct single-session compromise.

Self-Mutating Poisoning SMP

The skill contains a mutation hook rather than an immediately harmful payload. The first session using the skill appears harmless, while silently modifying persistent skill content. The harmful outcome materializes only when a later session reuses the modified skill.

Exposes a failure mode single-session evaluations cannot observe.

See it on real skills

Worked examples drawn from the released benchmark. Injected content is highlighted in place; switch the risk type to explore other attacks.

Risk Taxonomy

12 risk types organized by the agent workflow component through which harm materializes.

Category	Risk	Definition
Data Pipeline Exploitation	Data Exfiltration	Disclose sensitive or private data to an external destination.
	Output Manipulation	Alter a user-facing deliverable to be false, biased, or misleading.
	Poisoning	Contaminate trusted data or configuration.
System Environment Exploitation	Privilege Escalation	Weaken access controls on a protected file or directory.
	Unauthorized File Mod.	Modify, delete, or encrypt out-of-scope user or system files.
	Backdoor Injection	Create persistent unauthorized access or hidden execution.
	Denial of Service	Plant a persistent disruption to degrade system availability.
	Malware Deployment	Download and install malicious payloads.
	System Corruption	Corrupt trusted system, tool, network, or runtime configuration.
Agent Autonomy Exploitation	Goal Hijacking	Divert the agent from the user's task to an unrelated objective.
	Anti-Forensics	Remove or alter evidence of prior malicious activity.
	Proxy Attack	Send attacker-authored content to a third party via agent authority.

AutoSkillHarm: Automated Attack Construction

A coding agent drives three sequential stages, each specified by a natural-language harness that defines stage inputs, objectives, constraints, required outputs, and review criteria.

1

Target Selection

Identify reachable attack targets — skill files agents actually read or execute during normal use — so payloads are not hidden where they are never reached.

2

Attack Design

Instantiate a risk type into a contextualized payload with a deterministic success evaluator, iteratively refined against an LLM-based detector for stealth.

3

Quality Filter

Run each candidate end-to-end and discard construction-level flaws — infeasible goals, weak framing, or evaluators with false positives.

Benchmark Statistics

879

attack samples

71

skills

12

risk types

57

user tasks

FPP 687 Fixed-Payload Poisoning samples · SMP 192 Self-Mutating Poisoning samples

Comparison with Prior Benchmarks

SkillHarm is the first to jointly cover cross-session attacks and a systematic risk taxonomy, with deterministic attack-success evaluation.

Benchmark	# Samples	# Skills	# Risks	Construction
SkillHarm Ours	879	71	12	NL-harness Agent
SKILL-INJECT	202	23	8	Manual design
SkillJect	100	100	4	Victim-in-loop
SkillAttack	171	171	8	Victim-in-loop
PoisonedSkills	1,070	81	15	Fixed LLM workflow
SkillTrojan	3,000+	1,200	1	Template-based
BadSkill	967	13	1	Template-based
SkillSafetyBench	155	47	30	Manual design

Cross-Session: supports attacks that persist across sessions. Det. Eval: attack success judged by a deterministic evaluator, not LLM judges alone.