Metadata-Version: 2.4
Name: darkarts
Version: 0.1.0
Summary: AI Red-Team Assessment CLI toolkit for evaluating adversarial robustness of locally-hosted language models
Author: hinchk
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/hinchk/darkarts
Project-URL: Changelog, https://github.com/hinchk/darkarts/releases
Project-URL: Issues, https://github.com/hinchk/darkarts/issues
Project-URL: CI, https://github.com/hinchk/darkarts/actions
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: httpx
Requires-Dist: rich
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-httpx; extra == "test"
Dynamic: license-file

# DarkArts

AI Red-Team Assessment CLI toolkit for evaluating the adversarial robustness of locally-hosted language models. Inspired by the [OWASP Top 10 for LLMs](https://owasp.org/www-project-top-10-for-large-language-model-applications/) and adversarial datasets like [OBLITERATUS](https://huggingface.co/datasets/elder-plinius/OBLITERATUS).

DarkArts automates the full red-team lifecycle: **ingest** known jailbreak datasets, **generate** attack variants using a local LLM, **assess** target models with multi-turn adversarial prompts, and **report** findings with CVSS-AI severity scoring and plain-language reproduction guides.

## Prerequisites

- **Python 3.10+**
- **Git** (for cloning jailbreak datasets)
- **Ollama** (for running local LLMs) — [install instructions](https://ollama.com/download)

## Installation

```bash
# Clone the repository
git clone https://github.com/hinchk/darkarts.git
cd darkarts

# Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate

# Install DarkArts and its dependencies
pip install -e '.[test]'

# Verify the installation
darkarts --help
```

After installation, the `darkarts` command is available in your terminal whenever the virtual environment is active.

## Quick Start

This walkthrough takes you from zero to a completed assessment report. You'll need Ollama running with at least one model pulled.

### 1. Pull a target model

```bash
# Pull a model to test against
ollama pull llama3.1:8b-instruct

# Verify it's running
darkarts assess recon --target http://localhost:11434
```

### 2. Ingest a jailbreak dataset

```bash
# Clone a public jailbreak dataset
darkarts ingest clone --url https://github.com/elder-plinius/OBLITERATUS

# Parse it into the prompt database
darkarts ingest parse --repo OBLITERATUS

# Verify prompts were imported
darkarts ingest list
```

### 3. Generate attack variants

```bash
# List available attack templates
darkarts generate templates

# Generate variants using your local LLM
darkarts generate run --model llama3.1:8b-instruct --template rephrase-variants --limit 10
```

### 4. Run an assessment

```bash
# Run all generated variants against the target model
darkarts assess run \
  --target http://localhost:11434 \
  --target-model llama3.1:8b-instruct \
  --goal-type harmful-content \
  --judge
```

The `--judge` flag enables LLM-as-judge scoring, where the same model evaluates whether each response actually complied with the adversarial request.

### 5. View results and export reports

```bash
# View the summary in the terminal
darkarts report summary --session <session-id-prefix>

# Export an HTML report with executive summary and CVSS-AI explainer
darkarts report export --session <session-id> --format html -o report.html

# Generate plain-language reproduction steps for each exploit
darkarts report reproduce --session <session-id> -o findings.md
```

## Commands

DarkArts is organized into five command groups. Run `darkarts <group> --help` for detailed options.

### `darkarts config`

Manage configuration stored at `~/.darkarts/config.json`.

| Command | Description |
|---------|-------------|
| `config show` | Display current configuration |
| `config set` | Set a configuration value (e.g., `config set default_model llama3.1:8b-instruct`) |

### `darkarts ingest`

Ingest jailbreak datasets from Git repositories.

| Command | Description |
|---------|-------------|
| `ingest clone` | Clone a jailbreak dataset repository |
| `ingest parse` | Parse a cloned repo into the prompt database (JSON, CSV, TXT, MD) |
| `ingest list` | List ingested datasets and prompt counts |
| `ingest filter` | Filter prompts by technique, source, or keyword |

### `darkarts generate`

Generate adversarial prompt variants using a local LLM.

| Command | Description |
|---------|-------------|
| `generate templates` | List available attack templates |
| `generate run` | Generate variants from ingested prompts using a template |

**Built-in templates:**

| Template | Technique |
|----------|-----------|
| `rephrase-variants` | Academic framing, fictional narrative, authority impersonation, technical jargon |
| `pliny-liberator-override` | L1B3RT4S structural overload with system prompt injection |
| `encoding-wrapper` | Cyrillic homoglyphs, zero-width token splitting, ROT13 with prefix locking |
| `goal-directed` | Markdown/JSON extraction targeting specific data types |
| `multi-turn-escalation` | Foot-in-the-door escalation across multiple turns |
| `technique-transfer` | Cross-category technique application |

### `darkarts assess`

Run adversarial assessments against target model endpoints.

| Command | Description |
|---------|-------------|
| `assess recon` | Probe a target endpoint for available models and health status |
| `assess run` | Execute a full assessment with generated variants |
| `assess judge` | Re-run LLM-as-judge scoring on an existing session |

**Key options for `assess run`:**

| Option | Description |
|--------|-------------|
| `--target` | Target endpoint URL (e.g., `http://localhost:11434`) |
| `--target-model` | Model name on the target |
| `--goal-type` | Judge rubric: `harmful-content`, `prompt-leak`, or `policy-bypass` |
| `--judge / --no-judge` | Enable LLM-as-judge scoring |
| `--concurrency` | Number of parallel workers |
| `--actual-system-prompt` | For `prompt-leak` assessments: the true system prompt to compare against |
| `--target-policy` | For `policy-bypass` assessments: the constraint being tested |

### `darkarts report`

View metrics and export assessment reports.

| Command | Description |
|---------|-------------|
| `report summary` | Display assessment metrics in the terminal |
| `report export` | Export as HTML or JSON (`--format html` or `--format json`) |
| `report reproduce` | Generate plain-language reproduction steps for successful exploits |

**Report features:**

- **Executive summary** with color-coded risk badge and plain-English findings
- **CVSS-AI score explainer** with visual severity scale and links to CVSS/OWASP documentation
- **Detection breakdown** of sensitive patterns found in model responses (PII, API keys, system prompt leaks)
- **Reproduction guide** (`report reproduce`) — finding cards with exact prompts, observed responses, and step-by-step instructions a human tester can follow

## CVSS-AI Scoring

DarkArts uses a CVSS-AI score (0-10) adapted from the [Common Vulnerability Scoring System](https://www.first.org/cvss/). The score combines three factors:

| Factor | Weight | What it measures |
|--------|--------|-----------------|
| Attack Success Rate | 60% | What fraction of adversarial prompts bypassed guardrails |
| Judge Score | 40% | How fully the model complied with adversarial requests |
| Detection Severity | Multiplier | How sensitive the leaked information was (API keys > emails > generic text) |

| Score Range | Severity | Meaning |
|-------------|----------|---------|
| 0.0 | None | No successful bypasses |
| 0.1 - 3.9 | Low | Minor exposures under aggressive testing |
| 4.0 - 6.9 | Medium | Moderate vulnerabilities; hardening recommended before production |
| 7.0 - 8.9 | High | Significant vulnerabilities; deployment not recommended until remediated |
| 9.0 - 10.0 | Critical | Severe, easily exploitable vulnerabilities |

## Architecture

```
darkarts/
  cli.py                # Root Click group, registers all command subgroups
  config.py             # ~/.darkarts/config.json management
  models.py             # Dataclasses: JailbreakPrompt, GeneratedVariant, AssessmentSession, AssessmentResult
  db.py                 # SQLite CRUD at ~/.darkarts/darkarts.db
  commands/
    config_cmd.py       # darkarts config {show, set}
    ingest.py           # darkarts ingest {clone, parse, list, filter}
    generate.py         # darkarts generate {templates, run}
    assess.py           # darkarts assess {recon, run, judge}
    report.py           # darkarts report {summary, export, reproduce}
  core/
    parser.py           # Git clone + JSON/CSV/TXT/MD parsing with technique classification
    llm_client.py       # Ollama + OpenAI-compatible HTTP client (synchronous, httpx)
    pipeline.py         # ThreadPoolExecutor-based assessment orchestration
    detector.py         # Regex-based leakage detection (PII, system prompts, API keys)
    judge.py            # LLM-as-judge scoring with goal-specific rubrics and meta-analysis detection
    metrics.py          # ASR, evasion rate, CVSS-AI severity scoring
    reporter.py         # JSON and HTML report generation with executive summary
  templates/
    default_prompts.py  # 6 built-in attack generation templates
```

## Development

```bash
# Run the full test suite (65 tests)
python -m pytest tests/ -v

# Run a specific test file
python -m pytest tests/test_assess.py -v

# Run tests matching a keyword
python -m pytest tests/ -k "judge" -v
```

Tests use `pytest` + `click.testing.CliRunner` + `pytest-httpx` for HTTP mocking. No live Ollama instance is required for testing.

## License

Apache 2.0
