skill-scanner

Skill Scanner

A best-effort security scanner for AI Agent Skills that detects prompt injection, data exfiltration, and malicious code patterns. It combines pattern-based detection (YAML + YARA-X), AST and dataflow analysis, an optional LLM-as-a-judge, and a bounded CEL decision layer over typed detector facts.

Important: This scanner provides best-effort detection, not comprehensive or complete coverage. A scan that returns no findings does not guarantee that a skill is free of all threats. See Scope and Limitations below.

Supports OpenAI Codex Skills and Cursor Agent Skills formats following the Agent Skills specification. With --lenient, also scans non-standard formats such as Claude Code .claude/commands/*.md and flat markdown skill repos.

Highlights

• Multi-Engine Detection - Static analysis, behavioral dataflow, LLM semantic analysis, and cloud-based scanning for layered, best-effort coverage
• Typed CEL Decisions - The core scanner uses the official cel-go v0.32.0 runtime to correlate bounded facts after deterministic detection and before optional LLM analysis
• Finding Review - The optional Meta-analyzer correlates, prioritizes, and can filter findings; paired accuracy validation remains pending
• CI/CD Ready - SARIF output for GitHub Code Scanning, reusable GitHub Actions workflow, exit codes for build failures
• Pre-commit Hook - Standard pre-commit framework integration to scan skills before every commit
• Extensible - Plugin architecture for custom analyzers

Join the Cisco AI Discord to discuss, share feedback, or connect with the team.

Scope and Limitations

Skill Scanner is a detection tool. It identifies known and probable risk patterns, but it does not certify security.

Key limitations:

• No findings ≠ no risk. A scan that returns "No findings" indicates that no known threat patterns were detected. It does not guarantee that a skill is secure, benign, or free of vulnerabilities.
• Coverage is inherently incomplete. The scanner combines signature-based detection, LLM-based semantic analysis, behavioral dataflow analysis, optional cloud services, and configurable rule packs. While this approach improves coverage, no automated tool can detect every technique, especially novel or zero-day attacks.
• False positives and false negatives can occur. Consensus modes and meta-analysis can help review findings, but no configuration eliminates all incorrect classifications. Tune the scan policy to your risk tolerance.
• Human review remains essential. Automated scanning is one component of a defense-in-depth strategy. High-risk or production deployments should pair scanner results with manual code review and/or threat modeling.

Current modernization evidence

The final core + CEL development benchmark contains 5,256 malicious and 1,338 benign MaliciousSkillBench packages. Compared with origin/main, the current scanner raised F1 from 32.92% to 47.73% and recall from 19.88% to 31.43%, while reducing benign false-positive rate from 3.59% to 1.05%. Precision is 99.16%. Five CEL-shadow runs were exact and deterministic; CEL evaluated 154 candidates without proposing a suppression or falling back.

The locked source-disjoint split is weaker: TP=65, FP=42, TN=503, and FN=774, for 60.75% precision, 7.75% recall, 13.74% F1, and 7.71% FPR. This improves F1 over origin/main (7.40%) but regresses FPR (3.67%), so it does not pass the promotion gate. Every bundled CEL rule therefore remains in shadow; this change does not promote any CEL suppression.

Compatibility and supplemental checks found identical CEL-OFF/CEL-SHADOW findings on 111 official Codex, Claude Code, and Cursor skills (30 MEDIUM+ and 8 HIGH/CRITICAL packages), with five stable runs. The NotInject hard-negative set had 0/339 actionable matches. HarmfulSkillBench had 7/200 actionable and 6/200 HIGH+ packages with one quarantined sample, while OpenSkillRisk had 76/263 actionable packages with two host quarantines. The latter two are positive-only recall diagnostics and cannot measure precision or FPR. Optional ATR results are outside this release scope. See Detection Evaluation and Rollout for methodology, provenance, confidence intervals, and limitations.

Documentation

Installation

Prerequisites for this checkout and a release containing these changes: CPython 3.11–3.14 and uv (recommended) or pip

A release containing this branch's CEL changes will include the required helper in its wheels; there is no separate CEL extra. That release is configured for CPython 3.11–3.14, while source installs additionally require Go 1.27.1+ to build the helper. Until it is published, the package currently served by PyPI may have a different compatibility contract. See Installation and Configuration for details.

# Using uv (recommended)
uv pip install cisco-ai-skill-scanner

# Using pip
pip install cisco-ai-skill-scanner

# AWS Bedrock support
pip install cisco-ai-skill-scanner[bedrock]

# Google AI Studio / Gemini support
pip install cisco-ai-skill-scanner[google]

# Google Vertex AI support
pip install cisco-ai-skill-scanner[vertex]

# Azure OpenAI support
pip install cisco-ai-skill-scanner[azure]

# All cloud providers
pip install cisco-ai-skill-scanner[all]

Quick Start

Environment Setup (Optional)

# For LLM analyzer and Meta-analyzer
export SKILL_SCANNER_LLM_API_KEY="your_api_key"
export SKILL_SCANNER_LLM_MODEL="claude-3-5-sonnet-20241022"
# Optional: disabled, minimal, low, medium, high, xhigh, or max
export SKILL_SCANNER_LLM_REASONING_EFFORT="low"

# For VirusTotal binary scanning
export VIRUSTOTAL_API_KEY="your_virustotal_api_key"

# For Cisco AI Defense
export AI_DEFENSE_API_KEY="your_aidefense_api_key"

Interactive Wizard

Not sure which flags to use? Run skill-scanner with no arguments to launch the interactive wizard:

skill-scanner

The wizard walks you through selecting a scan target, analyzers, policy, and output format, then shows the assembled command before running it. Great for learning the CLI.

CLI Usage

# Scan a single skill (core analyzers: static + bytecode + pipeline + correlation)
skill-scanner scan /path/to/skill

# Scan with behavioral analyzer (dataflow analysis)
skill-scanner scan /path/to/skill --use-behavioral

# Scan with all engines
skill-scanner scan /path/to/skill --use-behavioral --use-llm --use-aidefense

# Scan with meta-analyzer for false positive filtering
skill-scanner scan /path/to/skill --use-llm --enable-meta

# Scan with trigger analyzer for vague description checks
skill-scanner scan /path/to/skill --use-trigger

# Run LLM analyzer multiple times and keep majority-agreed findings
skill-scanner scan /path/to/skill --use-llm --llm-consensus-runs 3

# Scan multiple skills recursively
skill-scanner scan-all /path/to/skills --recursive --use-behavioral

# Scan multiple skills with cross-skill overlap detection
skill-scanner scan-all /path/to/skills --recursive --check-overlap

# Scan a GitHub repository (owner/repo shorthand or full URL)
skill-scanner scan-repo owner/repo
skill-scanner scan-repo https://github.com/owner/repo --use-llm

# Lenient mode: tolerate malformed skills instead of failing
skill-scanner scan /path/to/skill --lenient
skill-scanner scan-all /path/to/skills --recursive --lenient

# Lenient mode with non-standard skill formats (no SKILL.md required)
skill-scanner scan .claude/commands/deploy --lenient
skill-scanner scan-all .claude/commands --recursive --lenient

# Use a custom metadata filename instead of SKILL.md
skill-scanner scan /path/to/skill --skill-file README.md

# CI/CD: Fail build if threats found
skill-scanner scan-all ./skills --fail-on-severity high --format sarif --output results.sarif

# Generate interactive HTML report with attack correlation groups
skill-scanner scan /path/to/skill --use-llm --enable-meta --format html --output report.html

# Use custom YARA rules
skill-scanner scan /path/to/skill --custom-rules /path/to/my-rules/

# Use custom taxonomy + threat mapping profiles (JSON/YAML)
skill-scanner scan /path/to/skill --taxonomy /path/to/taxonomy.json --threat-mapping /path/to/threat_mapping.json

# VirusTotal hash scan with optional unknown-file uploads
skill-scanner scan /path/to/skill --use-virustotal --vt-upload-files

# Use a scan policy preset (strict, balanced, permissive)
skill-scanner scan /path/to/skill --policy strict

# Inspect CEL decisions without suppressing findings
skill-scanner scan /path/to/skill --cel-mode shadow --format json

# Use a custom org policy file
skill-scanner scan /path/to/skill --policy my_org_policy.yaml

# Generate a policy file to customise
skill-scanner generate-policy -o my_org_policy.yaml

# Interactive policy configurator (TUI)
skill-scanner configure-policy

Consensus mode keeps a finding only when it appears in more than half of the