# Sample evaluation report — 2026-04-28

This directory holds the six artifacts produced by `agentsec evaluate` when
run against the in-process OpenClaw mock in
[`tests/integration/test_openclaw_mock.py`](../../../tests/integration/test_openclaw_mock.py).
It is committed so reviewers can see the v0.2.0 output shape without
provisioning a real OpenClaw deployment.

## What was evaluated

- **Remote:** 3-case suite covering data leakage, tool abuse, and prompt
  injection. The mock OpenClaw responds with one leak (fail), one tool call
  to `shell_exec` (fail), and one clean refusal (pass).
- **Server-audit:** stubbed `SSHExecutor` returns 2 unique findings + 1
  duplicate from `openclaw security audit --json`; the duplicate is
  collapsed by fingerprint into `findings.json`.
- **Authorization:** synthetic `AUTHORIZATION.txt` with
  `signature_mode: none`, so the run is `low_assurance`.

## Files

| File | Spec ref | Purpose |
|---|---|---|
| `remote-report.md` | §9.3 | Per-case verdict table for the remote suite. |
| `server-audit-report.md` | §9.3 | Per-finding table for the server-audit half. |
| `combined-report.md` | §9.3 | Overall grade + links to both halves. |
| `findings.json` | §9.3 | Machine-readable, redacted findings (CI consumable). |
| `threat-intel-snapshot.yaml` | §9.3 | TI rows referenced by this run. |
| `audit-log.jsonl` | §9.3 | SSH command metadata (no raw output). |

## How to regenerate

```bash
AGENTSEC_REFRESH_SAMPLE=1 .venv/bin/pytest \
  tests/integration/test_openclaw_mock.py::test_openclaw_mock_end_to_end -v
git diff -- docs/samples/
```

`test_committed_sample_matches_pipeline_output` will fail in CI if the
committed copies drift from the pipeline output, so refresh + commit
whenever the renderer or scoring formulas change.

## Limitations of this sample

This sample was produced against an in-process mock OpenClaw + a stubbed
`SSHExecutor`, so a few artifacts diverge from what a live evaluation
emits. Important caveats for readers:

- **`**Date**: <stripped>` placeholder** in the three Markdown reports is a
  drift-guard normalization, not literal renderer output. The live
  renderer stamps `datetime.now(timezone.utc)` (e.g.
  `**Date**: 2026-04-28 15:42 UTC`).
- **`audit-log.jsonl` is empty** because the test monkeypatches
  `SSHExecutor.run` directly, bypassing `_append_audit`. A live run
  records one JSON line per non-rejected SSH command (sha256 of rendered
  argv only — never the rendered string itself; spec §6.1). Rejected
  commands additionally log `argv` + `reject_reason`.
- **`server-audit-report.md` is intentionally a finding-list overview.**
  Per-finding `description` / `evidence` / `remediation` / `threat_refs`
  live in `findings.json`; the Markdown is a quick scanning surface. A
  richer per-finding Markdown view is tracked for v0.2.1.
- **`threat-intel-snapshot.yaml` projects only TI rows referenced this
  run** — here, only `TI-OPENCLAW-CVE-2026-25253` (the single TI ref the
  mock's two unique findings carry). On a live run with broader
  coverage, this file expands to every TI row any check or test case
  cited.
