Metadata-Version: 2.5
Name: vesk-cli
Version: 0.2.1
Summary: Vesk Verify: test LLM endpoints and produce compliance-mapped reports.
Author: Vesk
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml>=6
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Requires-Dist: respx; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Description-Content-Type: text/markdown

# vesk

**Vesk Verify** tests an OpenAI-compatible LLM endpoint with prompt-injection, jailbreak and
PII-leakage probes, then renders a report mapped to EU AI Act, DPDP and NIST AI RMF.

> Status: alpha (0.1.x). Interfaces may change between minor versions.

## Install

```
pip install vesk-cli
```

Requires Python 3.10 or newer.

## Quick start

```
vesk init                      # write a commented vesk.yaml
export VESK_API_KEY=...        # the env var named by target.api_key_env
vesk scan                      # run probes, write results.json
vesk report results.json -o report.md
```

`api_key_env` holds the *name* of an environment variable (e.g. `VESK_API_KEY`), not a file path or the key. By default vesk reads only the process environment, so export the variable in your shell. To use a `.env` file instead, pass it explicitly: `vesk scan --env-file .env`. Variables already set in the environment take precedence over the file. Omit `api_key_env` if the endpoint needs no key.

## Commands

### `vesk init`

Writes a starter `vesk.yaml`.

- `--path PATH`: where to write it (default `vesk.yaml`).
- `--force`: overwrite an existing file.

### `vesk scan`

Runs the configured probes against the target and writes a results JSON file.

- `-c, --config PATH`: config file (default `vesk.yaml`).
- `--probe NAME`: run only this probe; repeatable.
- `--out PATH`: results file (default `results.json`).
- `--env-file PATH`: optional dotenv-style file (`KEY=VALUE` lines) to read the API key from. Not read unless given; real environment variables win over the file.
- `--trials N`: trials per case (default 5). A fresh canary is generated for every trial.
- `--max-tokens N`, `--reasoning-effort LEVEL`: sent to the target and recorded in the results. Thinking models need a larger `--max-tokens`, or they spend it all on reasoning and return empty content (reported as `error` at preflight, `inconclusive` mid-run; never a pass).
- `--system-prompt-file PATH`, `--system-name NAME`: the system prompt to test under, and the name shown in the report. Without a prompt, a versioned baseline prompt is used and recorded.
- `--include-system-prompt`: store the system prompt text in the results (off by default; only its hash is stored).
- `--redact`: replace prompts, responses and reasoning in the results with length markers.
- `--cases PATH`: your own test cases (YAML or JSON, repeatable). See [docs/custom-cases.md](docs/custom-cases.md).
- `--judge`: also ask the bring-your-own judge from the config. Responses are sent to the judge endpoint.

While running, `vesk scan` prints progress to stderr: target host and model, each probe's start and status, and request counts. It never prints prompts, responses, API keys, URL paths or credentials. The results file contains full prompts and responses and may hold sensitive data; use `--redact` to strip them.

Every case runs several trials, and a trial is `pass`, `fail`, `inconclusive` or `error`. A pass needs a valid, non-empty, non-truncated answer. Before the probes, a benign control prompt must get a usable answer; if not, no probe runs and each is marked `error`.

Exit codes:

| Code | Meaning |
|------|---------|
| 0 | All passed. |
| 1 | At least one failure at or above `scan.fail_on`. |
| 2 | No failures, but at least one inconclusive result (also used for configuration or file errors at startup). |
| 3 | Tool, configuration or target error with no failure proven. |

A proven failure always wins over an error. This makes `vesk scan` usable as a CI gate.

### `vesk report`

Renders a report from a results file.

```
vesk report results.json --format html -o report.html
```

- `--format md|html|sarif|junit`: output format (default `md`).
- `-o, --output PATH`: write to a file. A `<file>.sha256` sidecar is written next to it. The SHA-256 detects file changes only; it is not a signature and does not prove who made the file.

Every report carries the notice: "Point-in-time test of the stated target and configuration. It is not a certification, legal opinion, or guarantee of security or regulatory compliance." Reports warn when a result is inconclusive or when the target is a raw model endpoint rather than an application.

### `vesk compare`

```
vesk compare old.json new.json [--json]
```

Lists regressions, improvements, added and removed cases, and config changes between two local runs. Exit code 1 if there is any regression.

## Configuration

`vesk init` writes a fully commented file. In short:

```yaml
version: 1
target:
  base_url: https://api.example.com/v1   # OpenAI-compatible chat completions API
  model: my-model
  api_key_env: VESK_API_KEY              # name of the env var; never put the key in the file
  # system_prompt: "You are a support bot."
  timeout_s: 30
  max_retries: 2
scan:
  probes: [prompt_injection, jailbreak_marker, pii_leakage]
  samples_per_probe: 20
  max_concurrency: 4
  seed: 0
  fail_on: high                          # none | low | medium | high
report:
  frameworks: [eu_ai_act, dpdp, nist_ai_rmf]
  system_name: "My AI system"
```

## Probes

| Probe | Checks |
|-------|--------|
| `prompt_injection` | Whether text in the user turn or in quoted data overrides the instructions. |
| `jailbreak_marker` | Jailbreak Marker Compliance: whether persona and role-play framings make the model emit a harmless marker. It does **not** measure harmful-content compliance. |
| `pii_leakage` | Whether the model repeats synthetic personal data planted in its own context. Training-data regurgitation is out of scope for 0.1. |

Probes are tagged with OWASP Top 10 for LLM Applications IDs (edition 2025, the latest published; `prompt_injection` and `jailbreak_marker` map to LLM01, `pii_leakage` to LLM02). The edition is recorded in the results.

## Targets, custom cases, judge

- **Targets:** Ollama and any OpenAI-compatible API (`target.adapter: openai`, the default), or any HTTP endpoint via a request template (`target.adapter: http`). See [docs/targets.md](docs/targets.md).
- **Custom cases:** [docs/custom-cases.md](docs/custom-cases.md).
- **Plugins:** extension points for suites, detectors, target adapters and reporters: [docs/plugins.md](docs/plugins.md).
- **Judge:** optional `judge:` config section. Its verdict is stored separately; deterministic detectors stay authoritative unless `judge.authoritative: true`. A judge failure never produces a pass.
- **Results format:** [docs/schema-v2.md](docs/schema-v2.md). Known limits: [docs/limitations.md](docs/limitations.md). Changes: [CHANGELOG.md](CHANGELOG.md).

## Privacy

The API key is read from an environment variable and is not stored in the config. Probe
requests go only to the endpoint you configure. PII probes use synthetic values, masked in
stored results.

## Not legal advice

Reports are generated automatically from test results. They are not legal or compliance advice
and do not certify compliance with any law, regulation or standard.

## Development

```
pip install -e ".[dev]"
pytest
ruff check .
```

## License

Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
