Agent Release Safety Gates Evaluation Report

Executive Summary

This report summarizes a public AI-agent release-readiness evaluation system. The core benchmark uses a synthetic operations domain, with separate public TechQA and WixQA retrieval benchmarks. It does not use real company documents, customer data, employee data, confidential processes, or real operational actions.

• Golden retrieval cases: 358
• Synthetic ticket extraction and agent cases: 180
• Red-team safety cases: 60
• Best current retriever: Local TF-IDF vector
• Current vector experiments: local TF-IDF vector retrieval and local embedding-store retrieval

Evaluation Release Gates

Incident Replay Suite

Incident Response Plan

Dataset Profile

This profile is generated from the same JSONL artifacts as the eval runner. It makes the synthetic benchmark mix visible, including manual-case share, abstention coverage, risk coverage, and known data gaps.

Retrieval Evaluation

The retrieval experiment compares a deliberately weak baseline, a lexical retriever, a local hybrid sparse semantic retriever, a TF-IDF vector retriever, and a local embedding-store retriever. The embedding row uses stable feature-hashed vectors; it is not a paid provider model. A separate provider-backed embedding script is available but not included in deterministic CI.

Retriever Metric Snapshots

External Public RAG Benchmark

WixQA Public Enterprise RAG Benchmark

Public RAG Findings

Public RAG Reranking Opportunity

Public RAG Reranker Evaluation

Hosted Public RAG Reranker Adapter

RAG Grounding Intervention Study

Historical Evaluation Snapshots

Retriever Failure Analysis

Failure Taxonomy

Total taxonomy-labeled cases: 618

Baseline To Improved Delta

Agent Safety Intervention Study

Responsible release boundary: Results are controlled benchmark evidence. They are not production safety claims and should be strengthened with independent human labels.

Memory Context Intervention Study

Goal Conflict Intervention Study

This section turns the lab into a mitigation-aware study: each experiment compares a baseline variant against layered safeguards and reports safety improvement alongside review burden or usefulness cost.

Structured Extraction

Extraction currently uses deterministic synthetic ticket patterns. The value of this stage is the schema, routing contract, and evaluation harness rather than a claim that messy real tickets are solved.

Safety Red-Team

Block rate requires an explicit policy refusal. Safe response rate checks that forbidden behavior is absent from the response. Weighted safe response rate prioritizes higher-severity attack types, and residual risk score is the remaining unsafe severity-weighted case total.

Safety Classifier Workflow

Publication policy: This artifact is derived only from sanitized reviewed summaries and maintainer-labelled calibration metadata. It excludes raw provider response ids and full provider rationales.

Controlled Agent Workflow

The controlled workflow separates read-only tools from side-effecting actions. The mock ticket routing tool is prepared but blocked until approval is granted, and every run returns trace, audit, and monitoring fields.

Agent Trace Examples

Observability Span Export

The combined export includes workflow-level spans, agent tool/audit spans, case-level retriever failure spans, retriever ranking-detail spans, case-level extraction spans, case-level agent approval spans, plus API contract and error-case spans for local inspection. The collector adapter translates this local JSONL into OTLP/HTTP JSON; the Docker Compose observability profile verifies that the same payloads are accepted by an OpenTelemetry Collector using collector self-metrics.

What This Proves

• The project can generate synthetic enterprise operations data safely.
• The retrieval harness can also run against selected public technical-support data.
• Public RAG grounding interventions report unsupported-answer reduction alongside abstention and review cost.
• Memory/context pollution controls test whether stale, cross-user, or injected memory is ignored in favor of current evidence.
• Retrieval quality can be measured across exact, paraphrased, noisy, conflicting, and adversarial cases.
• Structured extraction, routing, refusal behavior, approval gates, and audit traces are evaluated as product behavior, not only as model output.
• The dashboard, API, Docker runtime, and CI workflow make the lab reproducible.

Current Limitations

• The dataset is synthetic and templated.
• Extraction is deterministic rather than LLM-backed.
• The vector retriever is local TF-IDF, not an embedding model or vector database.
• The embedding-store retriever uses local feature-hashed embeddings, not a paid API.
• The TechQA public track is a 160-case compact external sample, not the full dataset.
• The WixQA public track is an 80-case compact expert-written sample, not the full benchmark suite.
• Scores should be read as regression-test results for this lab, not as claims about production accuracy.
• Human-review workflow labels are simulated; the calibration sample is maintainer-labelled and not yet independently reviewed.
• Hosted LLM-as-judge evidence is currently a single reviewed OpenAI calibration run, not a multi-model comparison.
• Independent external human labels and inter-rater agreement are not yet published.

Recommended Next Work

• Formalize the failure taxonomy across safety, retrieval, citation, privacy, tool-use, and usefulness failures.
• Add independent external human review for the calibration sample and compare it with deterministic rules and hosted LLM-as-judge decisions.
• Extend the reviewed hosted judge track beyond the first OpenAI run and publish disagreement slices separately from the local rubric baseline.
• Add optional multi-model evaluation adapters and publish only reproducible result tables.
• Run safety intervention experiments across refusal policy, retrieval grounding, tool approval gates, secondary review, and classifier thresholds.
• Validate the public RAG grounding thresholds with a provider-backed reranker.
• Collect external labels for memory/context pollution cases.
• Expand the TechQA public benchmark beyond 160 cases and compare against provider embeddings.
• Expand the WixQA public track and add provider-backed embedding comparison.