skill-scanner

Skill Scanner

A best-effort security scanner for AI Agent Skills that detects prompt injection, data exfiltration, and malicious code patterns. It combines pattern-based detection (YAML + YARA-X), AST and dataflow analysis, an optional LLM-as-a-judge, and a bounded CEL decision layer over typed detector facts.

Important: This scanner provides best-effort detection, not comprehensive or complete coverage. A scan that returns no findings does not guarantee that a skill is free of all threats. See Scope and Limitations below.

Supports OpenAI Codex Skills and Cursor Agent Skills formats following the Agent Skills specification. With --lenient, also scans non-standard formats such as Claude Code .claude/commands/*.md and flat markdown skill repos.

Highlights

• Multi-Engine Detection - Static analysis, behavioral dataflow, LLM semantic analysis, and cloud-based scanning for layered, best-effort coverage
• Typed CEL Decisions - The core scanner uses the official cel-go v0.32.0 runtime to correlate bounded facts after deterministic detection and before optional LLM analysis
• Measured Presets - low-noise and quiet presets and LLM caps, with recall, false-positive rate and F1 published for each (Recommended Settings)
• CI/CD Ready - SARIF output for GitHub Code Scanning, reusable GitHub Actions workflow, exit codes for build failures
• Pre-commit Hook - Standard pre-commit framework integration to scan skills before every commit
• Extensible - Plugin architecture for custom analyzers

Join the Cisco AI Discord to discuss, share feedback, or connect with the team.

Scope and Limitations

Skill Scanner is a detection tool. It identifies known and probable risk patterns, but it does not certify security.

Key limitations:

• No findings ≠ no risk. A scan that returns "No findings" indicates that no known threat patterns were detected. It does not guarantee that a skill is secure, benign, or free of vulnerabilities.
• Coverage is inherently incomplete. The scanner combines signature-based detection, LLM-based semantic analysis, behavioral dataflow analysis, optional cloud services, and configurable rule packs. While this approach improves coverage, no automated tool can detect every technique, especially novel or zero-day attacks.
• False positives and false negatives can occur. Presets, the LLM judge and scoped suppressions reduce noise, but no configuration eliminates all incorrect classifications. Pick a measured setup from Recommended Settings and tune the scan policy to your risk tolerance.
• Human review remains essential. Automated scanning is one component of a defense-in-depth strategy. High-risk or production deployments should pair scanner results with manual code review and/or threat modeling.

Measured results

On held-out skills (MaliciousSkillBench's frozen test split, 839 malicious and 545 harmless):

• Rules alone catch 7.7% of malicious skills at HIGH, at a 4.0% false-positive rate.
• With the LLM judge (Gemma 4 26B, balanced), 66.7% reach review at MEDIUM+ (15.4% FPR), and 33.7% are blocked at HIGH (6.4% FPR).

Every figure, with its corpus and method, is in Measured Results and on the evaluation Space.

Documentation

The documentation website is cisco-ai-defense.github.io/docs/skill-scanner. Deep-dive pages live in docs/.

Installation

Prerequisites: CPython 3.11–3.14 and uv (recommended) or pip.

Wheels include the CEL helper for glibc Linux x86-64/ARM64, macOS 14+ x86-64/ARM64 and Windows x86-64. Other platforms build from the source distribution, which needs Go 1.27.1+. See Installation and Configuration for details.

# Using uv (recommended)
uv pip install cisco-ai-skill-scanner

# As a standalone tool
uv tool install cisco-ai-skill-scanner   # or: pipx install cisco-ai-skill-scanner

# Using pip
pip install cisco-ai-skill-scanner

# Using Homebrew (macOS 14+)
brew tap cisco-ai-defense/skill-scanner https://github.com/cisco-ai-defense/skill-scanner
brew install cisco-ai-defense/skill-scanner/skill-scanner

The presets and settings in Recommended Settings need 2.2.0 or newer (skill-scanner --version).

Cloud Provider Extras

# AWS Bedrock support (IAM credentials, no API key)
pip install cisco-ai-skill-scanner[bedrock]

# Google AI Studio / Gemini support
pip install cisco-ai-skill-scanner[google]

# Google Vertex AI support
pip install cisco-ai-skill-scanner[vertex]

# Azure OpenAI support
pip install cisco-ai-skill-scanner[azure]

# On-device Apple Foundation Model (experimental: macOS 26+, Apple Intelligence, full Xcode)
pip install "apple-fm-sdk>=0.2.1,<0.3"

# All cloud providers
pip install cisco-ai-skill-scanner[all]

Quick Start

Recommended settings

Every recommended setup runs the LLM judge (--use-llm): rules alone catch only about 8% of held-out malicious skills.

Rates are for the MEDIUM+ review queue on MaliciousSkillBench's held-out split, with Gemma 4 26B as the judge. Measured precision for each, and the same setups for pre-commit, GitHub Actions, Python and the REST API, are in Recommended Settings.

Environment Setup

# The LLM judge, used by every recommended setup (local models: see LLM Providers)
export SKILL_SCANNER_LLM_API_KEY="your_api_key"
export SKILL_SCANNER_LLM_MODEL="claude-sonnet-5-5"   # the default

# On-device Apple Foundation Model (experimental, no API key). Behavioral
# alignment verification is skipped with a warning on this model.
# export SKILL_SCANNER_LLM_MODEL="apple-fm/system"
# Optional: disabled, minimal, low, medium, high, xhigh, or max
export SKILL_SCANNER_LLM_REASONING_EFFORT="low"

# For VirusTotal binary scanning
export VIRUSTOTAL_API_KEY="your_virustotal_api_key"

# For Cisco AI Defense
export AI_DEFENSE_API_KEY="your_aidefense_api_key"

Interactive Wizard

Not sure which flags to use? Run skill-scanner with no arguments to launch the interactive wizard:

skill-scanner

The wizard walks you through selecting a scan target, analyzers, policy, and output format, then shows the assembled command before running it. It recommends the LLM judge and leaves the meta-analyzer off. Great for learning the CLI.

CLI Usage