Metadata-Version: 2.4
Name: skillscan
Version: 0.4.0
Summary: CLI tool that scans SKILL.md agent skill files for security threats
Project-URL: Homepage, https://github.com/NMitchem/SkillScan
Project-URL: Documentation, https://github.com/NMitchem/SkillScan#readme
Project-URL: Issues, https://github.com/NMitchem/SkillScan/issues
Project-URL: Changelog, https://github.com/NMitchem/SkillScan/blob/main/CHANGELOG.md
License: MIT
License-File: LICENSE
Keywords: agent,ai,audit,llm,sandbox,scanner,security,skill
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: yara-python>=4.5
Provides-Extra: dev
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# SkillScan

[![CI](https://github.com/NMitchem/SkillScan/actions/workflows/ci.yml/badge.svg)](https://github.com/NMitchem/SkillScan/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/skillscan)](https://pypi.org/project/skillscan/)
[![Python](https://img.shields.io/pypi/pyversions/skillscan)](https://pypi.org/project/skillscan/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Security scanner for AI agent skills. Audit. Predict. Test.**

SkillScan detects malicious patterns in SKILL.md agent skill files before they
reach your machine. Three-stage pipeline: static analysis catches known patterns,
LLM behavioral prediction catches temporal attacks, and Docker Sandbox execution
catches everything else.

## Try It Now

```bash
pip install skillscan

# Scan an example malicious skill (included in the repo)
git clone https://github.com/NMitchem/SkillScan.git && cd SkillScan
skillscan audit examples/malicious/keychain-stealer.md
```

## Why SkillScan?

The AI agent skills ecosystem is growing fast -- and so are supply-chain attacks
targeting it. Static scanners catch obvious patterns, but sophisticated attacks
use temporal triggers, obfuscation, and delayed activation that only emerge at
runtime.

SkillScan is the only tool that combines all three detection approaches:

| Capability | SkillScan | Static Scanners |
|---|---|---|
| Pattern matching (shell, URLs, env vars) | Yes | Yes |
| YARA rules (59 rules, 7 categories) | Yes | Partial |
| macOS-specific detection (Keychain, AppleScript) | Yes | No |
| LLM behavioral prediction | Yes | No |
| Temporal/delayed-activation detection | Yes | No |
| Docker Sandbox execution with honeypots | Yes | No |
| SARIF output for GitHub Security tab | Yes | Some |

## Three-Stage Pipeline

**Audit** -- Static analysis with 8 built-in analyzers and 59 YAML rules across
7 categories. Catches shell injection, obfuscated payloads, suspicious URLs,
credential theft patterns, and macOS-specific threats in milliseconds.

**Predict** -- LLM behavioral dry-run that simulates what an agent would actually
do with the skill at runtime. Reveals temporal triggers, delayed-activation
payloads, and multi-step attack chains that static analysis cannot see.

**Test** -- Sandbox execution inside a Docker microVM with honeypot canary files
(fake credentials, SSH keys, API tokens). If the skill reads, copies, or
exfiltrates any canary, the test flags it as malicious with critical severity.

## Installation

```bash
pip install skillscan
```

Requires Python 3.10+.

## Usage

### Quick Start

Scan a single skill file:

```bash
skillscan audit path/to/SKILL.md
```

Scan a directory of skills:

```bash
skillscan audit ./skills/ --dir
```

Output as JSON or SARIF:

```bash
skillscan audit SKILL.md --format json
skillscan audit SKILL.md --format sarif
```

### Predict Mode

Predict mode performs an LLM behavioral dry-run that simulates what an agent
would do with a skill at runtime. Instead of only scanning for static patterns,
it asks an LLM to role-play as the agent, revealing risks that emerge from
actual tool use -- including temporal triggers, delayed-activation payloads,
and multi-step attack chains that static analysis misses.

Run a prediction against a skill file:

```bash
skillscan predict path/to/SKILL.md
```

Choose an LLM provider:

```bash
skillscan predict path/to/SKILL.md --provider claude
skillscan predict path/to/SKILL.md --provider openai
skillscan predict path/to/SKILL.md --provider gemini
skillscan predict path/to/SKILL.md --provider ollama
```

Control how many scenarios are simulated:

```bash
skillscan predict path/to/SKILL.md --scenarios 5
```

### Combined Scan

The `scan` meta-command chains `audit`, `predict`, and `test` in a single
pass, giving you static analysis, behavioral prediction, and sandbox
execution together:

```bash
skillscan scan path/to/SKILL.md
```

This is the recommended way to get full coverage on a skill file. If Docker
Desktop is not available, the test stage is skipped automatically.

### Test Mode (Sandbox Execution)

Test mode runs the skill with a real AI agent inside a Docker Sandbox
microVM. Unlike predict mode (which simulates behavior via LLM dry-run),
test mode actually executes the skill in an isolated sandbox environment
and observes what the agent does at runtime -- file writes, network calls,
process spawns, and credential access are all recorded and analyzed.

#### Prerequisites

- **Docker Desktop 4.58+** with the Sandbox plugin enabled.
  Sandbox testing uses Docker's microVM isolation to safely execute
  untrusted skills. Docker Desktop is required (not available in CI).

#### Agent Selection

Choose which agent(s) to test with:

```bash
skillscan test path/to/SKILL.md --agent claude
skillscan test path/to/SKILL.md --agent openclaw
skillscan test path/to/SKILL.md --agent claude,openclaw
```

#### Test Depth

Control how many invocations the agent performs:

```bash
# Quick mode (default): 1 invocation
skillscan test path/to/SKILL.md

# Standard mode: 3 invocations
skillscan test path/to/SKILL.md --standard

# Deep mode: 7 invocations
skillscan test path/to/SKILL.md --deep
```

#### Extended Observation

Keep the sandbox running longer to catch delayed-activation behavior:

```bash
skillscan test path/to/SKILL.md --run-for 10m
```

#### Honeypot Canary Detection

During sandbox execution, SkillScan plants honeypot canary files --
fake credentials, SSH keys, and API tokens -- in well-known locations
inside the microVM. If the skill (or the agent acting on the skill's
instructions) reads, copies, or exfiltrates any of these canary files,
the test immediately flags the skill as malicious with a critical-severity
finding. This catches data-theft behavior that neither static analysis
nor LLM prediction can reliably detect.

> **Note:** Sandbox testing requires Docker Desktop and is not available
> in CI environments. When Docker is not present, `scan` gracefully skips
> the test stage and runs only audit + predict.

## CLI Reference

### `skillscan audit`

Audit one or more SKILL.md files for security threats.

```
skillscan audit <path> [options]
```

| Flag | Description | Default |
|------|-------------|---------|
| `--dir` | Treat path as a directory and scan all skills inside it | `false` |
| `--format {text,json,sarif}` | Output format | `text` |
| `--threshold <float>` | Risk score threshold for pass/fail (0.0-10.0) | `6.0` |
| `--no-color` | Disable ANSI color codes in text output | `false` |
| `--llm` | Enable optional LLM-powered analysis | `false` |
| `--provider {auto,claude,openai,gemini,ollama}` | LLM provider to use | `auto` |

Exit code is `0` if all skills pass, `1` if any skill fails or on error.

### `skillscan predict`

Run an LLM behavioral dry-run against a skill file.

```
skillscan predict <path> [options]
```

| Flag | Description | Default |
|------|-------------|---------|
| `--provider {claude,openai,gemini,ollama}` | LLM provider to use | `claude` |
| `--scenarios <int>` | Number of behavioral scenarios to simulate | `3` |
| `--format {text,json}` | Output format | `text` |
| `--no-color` | Disable ANSI color codes in text output | `false` |

Predict mode detects temporal and delayed-activation risks that static analysis
cannot catch, such as skills that behave differently on the second invocation
or after a time delay.

### `skillscan scan`

Run `audit`, `predict`, and `test` in a single pass.

```
skillscan scan <path> [options]
```

Accepts the same flags as `audit`, `predict`, and `test`. Exit code is `0`
if all stages pass, `1` if any stage fails. The test stage is skipped when
Docker Desktop is not available.

### `skillscan install`

Fetch, scan, and install a skill in one step.

```
skillscan install <source> [options]
```

| Flag | Description | Default |
|------|-------------|---------|
| `--target <dir>` | Override install directory | auto-detect |
| `--threshold <float>` | Risk score threshold for pass/fail | `6.0` |
| `--force` | Install even if the scan fails | `false` |
| `--no-color` | Disable ANSI color codes | `false` |

Sources can be a local path, a GitHub URL, or a direct HTTP URL to a SKILL.md file.

## Detection Coverage

SkillScan combines 8 built-in analyzers with a YAML rule engine containing 59 rules
across 7 categories.

### Built-in Analyzers

| Analyzer | What it detects |
|----------|-----------------|
| `shell` | Dangerous shell commands, sudo usage, privilege escalation |
| `macos` | macOS-specific threats (LaunchAgents, TCC bypass, Gatekeeper) |
| `obfuscation` | Base64-encoded payloads, hex strings, Unicode tricks |
| `url` | Suspicious URLs, IP-based endpoints, known malicious domains |
| `envvar` | Reads of sensitive environment variables and credentials |
| `entitlement` | Overly broad macOS entitlements and permissions |
| `typosquat` | Typosquatted package names in npm, pip, and other registries |
| `pattern` | General suspicious patterns (eval, exec, dynamic code loading) |

### YAML Rule Categories

| Category | Rules | Description |
|----------|------:|-------------|
| Prompt Injection | 12 | Instruction override, role hijacking, delimiter attacks |
| MCP Security | 8 | Tool poisoning, schema manipulation, permission escalation |
| SSRF & Cloud | 6 | Internal network probing, cloud metadata access |
| Exfiltration | 8 | Data theft via DNS, HTTP, webhooks, and side channels |
| Credentials | 10 | Hardcoded keys, token extraction, keychain access |
| Supply Chain | 10 | Dependency confusion, post-install scripts, registry attacks |
| Unicode Attacks | 5 | Bidi overrides, homoglyph substitution, invisible characters |

Rules are defined in `src/skillscan/data/rules/` as YAML files. Each rule includes
a regex pattern, severity level, and descriptive message.

## GitHub Action

Add SkillScan to your CI pipeline to block malicious skills before merge.

```yaml
name: SkillScan
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: NMitchem/SkillScan@v0.4.0
        with:
          path: '.'
          threshold: '6.0'
          format: 'sarif'
          upload-sarif: 'true'
```

When `upload-sarif` is enabled, findings appear directly in the GitHub Security tab.

## Contributing

### Adding Rules

1. Create or edit a YAML file in `src/skillscan/data/rules/`.
2. Follow the existing format:

```yaml
category: your_category
rules:
  - id: YOUR_RULE_ID
    pattern: "regex pattern here"
    severity: high
    message: "Description of what this rule detects"
```

3. Add tests in `tests/` to verify your rules fire on expected input.
4. Run the full test suite: `pytest tests/ -v`

### Running Tests

```bash
pip install -e ".[dev]"
pytest tests/ -v
```

## Roadmap

- ~~**v0.2** -- Static analysis with YAML rules, SARIF output, GitHub Action~~
- ~~**v0.3** -- Predict: LLM behavioral dry-run and risk prediction~~
- ~~**v0.4** -- Test: Sandbox execution and runtime threat detection~~

What's next: registry integration, community rule packs, and IDE plugins.

## License

MIT -- see [LICENSE](LICENSE) for details.
