GuardMeter Evaluation Report
Run: {{ run_id }} · Samples: {{ total_samples }} · Dataset SHA: {{ dataset_sha[:12] }}
· Git: {{ git_commit[:8] }} · Generated: {{ generated_at }}
Baseline: {{ baseline_name }}{% if guard_info.baseline and guard_info.baseline.model %} ({{ guard_info.baseline.model }}){% endif %}
· Candidate: {{ candidate_name }}{% if guard_info.candidate and guard_info.candidate.model %} ({{ guard_info.candidate.model }}){% endif %}
{% if mcnemar_p is not none %} · McNemar p: {{ mcnemar_p }}{% endif %}
{% if environment %}
GuardMeter {{ environment.guardmeter_version }} · Python {{ environment.python_version }} · policy {{ environment.policy }}{% endif %}
Baseline — Strict
Recall: {{ strict_base.recall }} ({{ strict_base.recall_lo }}–{{ strict_base.recall_hi }})
Precision: {{ strict_base.precision }}
F1: {{ strict_base.f1 }}
FPR: {{ strict_base.fpr }} ({{ strict_base.fpr_lo }}–{{ strict_base.fpr_hi }})
FNR: {{ strict_base.fnr }}
TP:{{ strict_base.tp }} FP:{{ strict_base.fp }} TN:{{ strict_base.tn }} FN:{{ strict_base.fn }}
Latency p50/p90/p99: {{ strict_base.latency_p50 }}/{{ strict_base.latency_p90 }}/{{ strict_base.latency_p99 }} ms
{% if strict_base.hijacked %}
Hijacked: {{ strict_base.hijacked }} ({{ '%.2f%%' | format(strict_base.hijack_rate * 100) }})
{% endif %}
Candidate — Strict
Recall: {{ strict_cand.recall }} ({{ strict_cand.recall_lo }}–{{ strict_cand.recall_hi }})
Precision: {{ strict_cand.precision }}
F1: {{ strict_cand.f1 }}
FPR: {{ strict_cand.fpr }} ({{ strict_cand.fpr_lo }}–{{ strict_cand.fpr_hi }})
FNR: {{ strict_cand.fnr }}
TP:{{ strict_cand.tp }} FP:{{ strict_cand.fp }} TN:{{ strict_cand.tn }} FN:{{ strict_cand.fn }}
Latency p50/p90/p99: {{ strict_cand.latency_p50 }}/{{ strict_cand.latency_p90 }}/{{ strict_cand.latency_p99 }} ms
{% if strict_cand.hijacked %}
Hijacked: {{ strict_cand.hijacked }} ({{ '%.2f%%' | format(strict_cand.hijack_rate * 100) }})
{% endif %}
Slice Breakdown — Strict (Category × Language)
Baseline
| Category | Lang | N | Recall | FPR | FNR |
|---|
{% for s in strict_base_slices %}
| {{ s.category }} | {{ s.language }} | {{ s.n }} |
{{ s.recall }} |
{{ s.fpr }} |
{{ s.fnr }} |
{% endfor %}
Candidate
| Category | Lang | N | Recall | FPR | FNR |
|---|
{% for s in strict_cand_slices %}
| {{ s.category }} | {{ s.language }} | {{ s.n }} |
{{ s.recall }} |
{{ s.fpr }} |
{{ s.fnr }} |
{% endfor %}
{% if strict_cand_attack %}
Attack Type — Strict (Candidate)
| Attack type | N | Recall | FPR |
|---|
{% for s in strict_cand_attack %}
| {{ s.attack_type }} | {{ s.n }} |
{{ s.recall }} | {{ s.fpr }} |
{% endfor %}
{% endif %}
Baseline — Lenient
Recall: {{ lenient_base.recall }}
Precision: {{ lenient_base.precision }}
F1: {{ lenient_base.f1 }}
FPR: {{ lenient_base.fpr }}
FNR: {{ lenient_base.fnr }}
TP:{{ lenient_base.tp }} FP:{{ lenient_base.fp }} TN:{{ lenient_base.tn }} FN:{{ lenient_base.fn }}
Candidate — Lenient
Recall: {{ lenient_cand.recall }}
Precision: {{ lenient_cand.precision }}
F1: {{ lenient_cand.f1 }}
FPR: {{ lenient_cand.fpr }}
FNR: {{ lenient_cand.fnr }}
TP:{{ lenient_cand.tp }} FP:{{ lenient_cand.fp }} TN:{{ lenient_cand.tn }} FN:{{ lenient_cand.fn }}
Per-sample latency (ms) — Baseline
Per-sample latency (ms) — Candidate
{% if sweep_data %}
Candidate threshold sweep (real scores)
{% endif %}
{% if sample_results %}
Sample Results
| Text | Label | Category | Lang | Baseline | Candidate | {% if has_judge %}Judge | {% endif %}
|---|
{% for s in sample_results %}
| {{ s.text }} |
{{ s.label }} |
{{ s.category }} |
{{ s.language }} |
{{ s.baseline_pred }} |
{{ s.candidate_pred }} |
{% if has_judge %}{{ s.judge_verdict or '—' }} | {% endif %}
{% endfor %}
{% endif %}
Regulatory mapping (informational)
Informational aid for your own documentation — not a compliance certification or legal assessment.
| Framework | Requirement | How GuardMeter Addresses It |
|---|
| EU AI Act Art. 9 |
Risk management system for high-risk AI |
Automated evaluation pipeline quantifies guard performance risk per slice, with CI gate enforcement |
| EU AI Act Art. 15 |
Accuracy, robustness and cybersecurity |
Per-slice recall/FPR metrics with Wilson confidence intervals; adversarial augmentation testing |
| NIST AI RMF Measure 2.5 |
AI risk monitoring and measurement |
Run history stored with dataset SHA + git commit for full reproducibility; regression detection via comparison thresholds |
| ISO 42001 §8.4 |
AI system performance assessment |
Structured EvalResults with McNemar significance test; HTML report exportable for audit evidence |