Metadata-Version: 2.4
Name: nyuwayskillsandbox
Version: 0.1.0
Summary: Dynamic behavioral sandbox for AI agent skill bundles (SKILL.md)
Author-email: Nyuway AI <security@nyuway.ai>
Maintainer-email: Nyuway AI <security@nyuway.ai>
License-Expression: Apache-2.0
Project-URL: Homepage, https://nyuway.ai
Project-URL: Repository, https://github.com/Nyuway-Cybersecurity/nyuwayagentskillsandbox
Project-URL: Bug Tracker, https://github.com/Nyuway-Cybersecurity/nyuwayagentskillsandbox/issues
Keywords: skills,skill-md,agent,security,sandbox,behavioral-analysis,dynamic-analysis,ai-security,llm-security,agent-security
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: nyuwaymcpsandbox>=1.0.0
Requires-Dist: nyuwayskillscanner>=0.1.0
Requires-Dist: click>=8.1
Requires-Dist: rich>=13.0
Requires-Dist: docker>=7.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: litellm>=1.40
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Dynamic: license-file

# nyuwayagentskillsandbox

[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-314%20passing-brightgreen)](#development)
[![Rules](https://img.shields.io/badge/detection%20rules-9-blue)](#detection-rules)
[![Status](https://img.shields.io/badge/status-v0.1.0%20beta-yellow)](#)
[![Security](https://img.shields.io/badge/focus-agent%20skill%20security-red)](#detection-rules)
[![SARIF](https://img.shields.io/badge/output-SARIF%20%7C%20JSON%20%7C%20Timeline-informational)](#output-formats)

**Dynamic behavioral sandbox for AI agent skill bundles (`SKILL.md`).**

Skills ship as a folder: a `SKILL.md` file (YAML frontmatter + prose instructions an agent loads into context) plus bundled scripts the agent may invoke. Static analysis can flag suspicious patterns in that text and code, but it can't tell you what actually happens when an agent *uses* the skill. `nyuwayagentskillsandbox` detonates a skill bundle inside a hardened, network-sinkholed Docker container, records every observable behavior, and evaluates the resulting timeline against a bundled ruleset — producing the same kind of verdict (`PASS` / `LOW` / `MEDIUM` / `HIGH` / `CRITICAL`) that [`nyuwaymcpsandbox`](https://github.com/Nyuway-Cybersecurity/nyuwaymcpsandbox) produces for MCP servers.

```
$ nyuwayskillsandbox detonate ./malicious-notes-skill --mode fast --allow-network

Verdict:  HIGH  (score 60/100)              Duration: 00:16   Findings: 3
+----------------------- nyuwayagentskillsandbox - Behavioral Analysis --------+
|   Target:  ./malicious-notes-skill                                           |
|     Mode:  fast                                                              |
|  Verdict:  HIGH  (score 60/100)                                              |
| Duration:  00:16                                                             |
| Findings:  3                                                                 |
+------------------------------------------------------------------------------+
                              Behavioral Timeline
+------------------------------------------------------------------------------+
|  Time |   | Event                                                            |
|-------+---+------------------------------------------------------------------|
| 00:00 | . | Container started, skill bundle mounted read-only at /src        |
| 00:01 | . | Script 'scripts_harvest_files' invoked (python, deterministic)   |
| 00:01 | X | filesystem.read  /etc/passwd      [DETECTION: sensitive_read_…]  |
| 00:01 | X | filesystem.read  /root/.ssh/id_rsa [DETECTION: sensitive_read_…] |
| 00:01 | . | Script 'scripts_harvest_files' completed (exit 0, 0.34s)         |
| 00:14 | . | Script 'scripts_organize' invoked (python, deterministic)        |
| 00:14 | X | network.dns_lookup  attacker-telemetry.example.tk               |
|       |   |    [DETECTION: outbound_network_from_script]                     |
|       |   |    [DETECTION: suspicious_dns_tld_from_script]                   |
| 00:15 | . | Script 'scripts_organize' completed (exit 0, 1.0s)               |
| 00:16 | . | Container stopped                                                |
+------------------------------------------------------------------------------+
| X | HIGH   | sensitive_read_outside_bundle    | Script read /etc/passwd, … |
| X | HIGH   | outbound_network_from_script     | DNS/network during script  |
| X | MEDIUM | suspicious_dns_tld_from_script   | .tk TLD lookup             |
+------------------------------------------------------------------------------+
```

---

## How it fits in

`nyuwayagentskillsandbox` is the dynamic layer of the Nyuway Agent Skill Security toolkit:

| Phase | Product | Question |
|---|---|---|
| Pre-deployment static | [nyuwayskillscanner](https://github.com/Nyuway-Cybersecurity/nyuwayskillscanner) | What does the code say this skill does? |
| Pre-deployment dynamic | **nyuwayagentskillsandbox** | What does this skill actually do when an agent runs it? |
| MCP server runtime | [nyuwaymcpsandbox](https://github.com/Nyuway-Cybersecurity/nyuwaymcpsandbox) | What does this MCP server actually do at runtime? |

---

## Two detonation modes

| Mode | What runs | When to use |
|---|---|---|
| **`--mode fast`** (default) | Every bundled script is invoked once with a synthetic empty argument — no LLM involved | CI pipelines, pre-deploy gates, no API key required |
| **`--mode full`** | The skill's own `SKILL.md` body is loaded as an LLM's system prompt — exactly as a real agent would — and the LLM is given a test task. The driver observes which scripts the LLM invokes and what they do | Tool-choice and intent-mismatch behaviors the deterministic harness cannot surface |

---

## Install

```bash
git clone https://github.com/Nyuway-Cybersecurity/nyuwayagentskillsandbox
cd nyuwayagentskillsandbox
python -m venv .venv
.venv/Scripts/activate        # Windows
# source .venv/bin/activate   # Linux / macOS
pip install -e .
nyuwayskillsandbox setup      # verifies Docker, pulls base images
```

---

## Quick start

```bash
# Fast mode against a local skill directory
nyuwayskillsandbox detonate ./my-skill

# Full mode with a real LLM and a custom test task
nyuwayskillsandbox detonate ./my-skill --mode full --llm claude-sonnet-4-5 \
    --test-task "Summarise this quarterly report."

# Exercise the CLI without Docker or an LLM key
nyuwayskillsandbox detonate ./my-skill --dry-run

# CI gate — non-zero exit on any HIGH or above finding
nyuwayskillsandbox detonate ./my-skill --output json --fail-on high

# SARIF output for GitHub Advanced Security
nyuwayskillsandbox detonate ./my-skill --output sarif > results.sarif

# From GitHub (same source resolver as nyuwaymcpsandbox)
nyuwayskillsandbox detonate github:someuser/some-skill
```

`TARGET` may be a local path to a skill directory (or a `SKILL.md` file directly), `github:owner/repo[@ref]`, `npm:package[@version]`, or `pypi:package[@version]`.

---

## Detection rules

Nine bundled rules cover the behaviors most relevant to skill detonation:

| Rule | Severity | What it flags |
|---|---|---|
| `shell_exec_in_script` | HIGH | A bundled script spawned a subprocess |
| `outbound_network_from_script` | HIGH | Network activity during script execution |
| `sensitive_read_outside_bundle` | HIGH | A script read SSH keys, cloud credentials, or other sensitive system paths |
| `post_execution_network_activity` | HIGH | Network activity observed *after* a script completed — a stronger exfil signal than in-execution activity |
| `intent_action_mismatch` | HIGH | The LLM requested a script not declared in the bundle (Full mode) |
| `credential_env_access_in_script` | MEDIUM | A script read process environment variables |
| `file_write_outside_bundle` | MEDIUM | A script wrote outside its own bundle directory |
| `suspicious_dns_tld_from_script` | MEDIUM | DNS lookup for a high-risk TLD (`.tk`, `.gq`, `.xyz`, …) |
| `slow_script_execution` | LOW | A script ran longer than the configured threshold |

Rules are pure YAML and follow the same schema as `nyuwaymcpsandbox`'s detection engine — both exact and regex payload matching with dotted key paths.

---

## Verdicts

Score = `min(100, sum_of_finding_weights)`.

| Verdict | Score | Action |
|---|---|---|
| PASS | 0–19 | Deploy. No suspicious behaviour observed. |
| LOW | 20–39 | Deploy with monitoring. |
| MEDIUM | 40–59 | Review before deployment. |
| HIGH | 60–79 | Block deployment. |
| CRITICAL | 80–100 | Do not deploy. |

---

## Key flags

| Flag | Default | Purpose |
|---|---|---|
| `--mode fast\|full` | `fast` | `fast` = deterministic harness; `full` = adds LLM driver |
| `--llm <model>` | — | Any [litellm](https://docs.litellm.ai/) model identifier (required for full mode) |
| `--api-key` | env `NYUWAY_LLM_API_KEY` | LLM API key |
| `--test-task` | derived from skill description | Prompt given to the LLM in full mode |
| `--allow-network` | off | Grant real outbound egress (sinkholed by default) |
| `--script-timeout` | 30s | Hard timeout per script execution |
| `--post-exec-grace` | 2.0s | Seconds to observe the container after each script completes |
| `--fail-on low\|medium\|high\|critical` | off | Non-zero exit when findings meet or exceed the threshold |
| `--output timeline\|json\|sarif` | `timeline` | Output format |
| `--env-file` | — | Load extra container env vars from a `.env`-style file |
| `--dry-run` | off | In-memory fakes only — no Docker, no LLM needed |

---

## Output formats

```bash
--output timeline   # Rich terminal view (default)
--output json       # Stable JSON schema for scripting / dashboards
--output sarif      # SARIF 2.1.0 for GitHub Advanced Security / VS Code Problems
```

---

## What gets captured

| Layer | Mechanism | What lands on the timeline |
|---|---|---|
| Container | `docker run` lifecycle | `container.started`, `container.stopped` |
| Script execution | `docker exec` (one-shot) | `skill.script_invoked`, `skill.script_completed`, `skill.slow_script` |
| Filesystem reads | In-process `builtins.open` patch (Python scripts) | `filesystem.read` |
| Filesystem writes | watchdog on the bind-mounted `/src` | `filesystem.write`, `filesystem.delete` |
| Process | `docker container.top()` polling | `process.spawn`, `process.exit` |
| Network (DNS) | `tcpdump` sidecar | `network.dns_lookup` |
| Environment | `sitecustomize.py` shim (Python scripts) | `environment.read` |
| LLM driver (full mode) | litellm | `skill.llm_decision`, `skill.intent_mismatch` |

---

## Security defaults

Every detonation runs the skill's scripts inside a container with:

- `network_mode='none'` — sinkholed; only `--allow-network` opts in to real egress
- `read_only=True` root filesystem
- `cap_drop=['ALL']` + `security_opt=['no-new-privileges:true']`
- Resource caps: memory (`512m`), CPU, pid count (`100`)
- Skill bundle mounted **read-only** at `/src`
- `/tmp` mounted as tmpfs (`64m`) for script scratch space
- Full container destruction on session exit — nothing persists

---

## Architecture

```
TARGET
  └─ source resolver         (local / github: / npm: / pypi:)
       └─ Docker orchestrator   (hardened container)
            └─ monitor session   (filesystem / process / network / env)
                 ├─ deterministic driver   (invoke every script once)
                 └─ agent-loop driver      (full mode — LLM drives script selection)
                      └─ script executor   (docker exec, hard timeout, read-capture)
            └─ detection engine   (YAML rules over behavioral timeline)
                 └─ verdict + renderer   (timeline / JSON / SARIF)
```

This project reuses `nyuwaymcpsandbox`'s container orchestration, behavioral monitors, detection engine, LLM backend, and source resolver wholesale. The new components are the agent-loop driver, deterministic script prober, Docker script executor, monitor-event correlation (`sandbox/correlate.py`), and the in-process read-capture bootstrap (`sandbox/py_read_capture.py`).

---

## Known limitations (v0.1)

- **Single-turn agent loop.** One LLM response produces zero or more script invocations; there is no multi-turn loop where script output is fed back. Same boundary as `nyuwaymcpsandbox`'s LLM driver today.
- **Filesystem read-capture is Python-only.** `sensitive_read_outside_bundle` has a producer only for Python scripts (in-process `builtins.open` patch). A bash/JS/Ruby script reading `~/.ssh/id_rsa` leaves no `filesystem.read` evidence.
- **Process-monitor polling granularity.** ProcessMonitor polls `docker top` every 0.1s. A subprocess that spawns and exits in milliseconds can fall between polls and never appear on the timeline.
- **`file_write_outside_bundle` only sees writes under the bundle mount.** The filesystem monitor watches `/src` via a host-side watchdog observer. Writes to the container's own writable layer (e.g. `/etc/cron.d/…`) are not seen.
- **NetworkMonitor needs a shared network namespace.** With the default `network_mode=none`, pass `--allow-network` to observe (not just block) network behavior.

---

## Requirements

- Python 3.11+
- Docker (Linux or macOS with Docker Desktop; Windows via Docker Desktop + WSL2)
- An LLM API key — any provider [litellm](https://docs.litellm.ai/) supports, or local Ollama for air-gapped runs (Full mode only)

---

## Development

```bash
pip install -e ".[dev]"
pytest tests/ -q        # 314 unit tests
ruff check .
```

Test fixtures under `tests/fixtures/skill_bundles/` include a benign converter and deliberately malicious skills for end-to-end smoke testing:

```bash
nyuwayskillsandbox detonate tests/fixtures/skill_bundles/benign_converter   # expect PASS
nyuwayskillsandbox detonate tests/fixtures/skill_bundles/malicious_exfil --allow-network  # expect HIGH
nyuwayskillsandbox detonate tests/fixtures/skill_bundles/shell_persistence  # expect LOW
```

---

## Contributing

- **New detection rules** — drop a YAML file into `nyuwayskillsandbox/detection/builtin/`, add a test in `tests/test_detection_rules.py`, open a PR.
- **Bug reports** — include the rendered timeline (`--output timeline`) and raw JSON (`--output json`).
- **Real-world skill findings** — file an issue with the target, the detonation command, and the full timeline output.

---

## License

Apache 2.0. See [LICENSE](LICENSE).

---

## Links

- Website: https://nyuway.ai
- Static skill scanner: https://github.com/Nyuway-Cybersecurity/nyuwayskillscanner
- MCP server sandbox: https://github.com/Nyuway-Cybersecurity/nyuwaymcpsandbox
- Issues: https://github.com/Nyuway-Cybersecurity/nyuwayagentskillsandbox/issues
