Metadata-Version: 2.4
Name: egate
Version: 0.1.0
Summary: EvalGate - Behavioral CI/CD for AI Agents
Author: EvalGate Team
License: MIT
Project-URL: Homepage, https://github.com/evalgate/egate
Project-URL: Documentation, https://docs.evalgate.dev
Project-URL: Repository, https://github.com/evalgate/egate
Project-URL: Issues, https://github.com/evalgate/egate/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: httpx>=0.25.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: respx>=0.21.0; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Provides-Extra: deepeval
Requires-Dist: deepeval>=0.20.0; extra == "deepeval"
Provides-Extra: all
Requires-Dist: egate[deepeval,dev]; extra == "all"
Dynamic: license-file

<div align="center">

# 🚦 EvalGate (`egate`)

**Behavioral CI/CD for AI Agents. Stop shipping broken agents to production.**

[![CI](https://github.com/pisigmac/evalgate/actions/workflows/ci.yml/badge.svg)](https://github.com/pisigmac/evalgate/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/egate.svg?color=blue)](https://pypi.org/project/egate/)
[![Python Version](https://img.shields.io/pypi/pyversions/egate.svg)](https://pypi.org/project/egate/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)

[**Quick Start**](#-quick-start) • [**Why EvalGate?**](#-why-evalgate) • [**Supported Metrics**](#-supported-metrics) • [**CI/CD Integration**](#-cicd-pr-gate-integration) • [**Custom Plugins**](#-custom-eval-plugins) • [**Architecture**](#-architecture) • [**Docs**](#-documentation)

---

</div>

## 💡 Why EvalGate?

Traditional CI/CD checks that code compiles and unit tests return deterministic values. **AI Agents are non-deterministic.**

A subtle prompt edit or temperature adjustment can pass every unit test, yet introduce **hallucinations in 5% of production traffic** or cause agents to select forbidden tools.

```
                  Traditional CI: "Code runs without errors" ✔
                  EvalGate CI:   "Agent made the right decision 95% of the time with 99% CI" 🚦
```

**EvalGate** brings production-grade behavioral quality gates to your pull requests and pipelines:
- **Statistical Multi-Run Validation** — Run scenarios $N$ times with confidence interval calculation rather than relying on single-pass flukes.
- **50+ Built-in & DeepEval Metrics** — Out-of-the-box support for tool selection, hallucination, latency, toxicity, and DeepEval evaluators.
- **Automated PR Comments** — Post sticky, beautifully formatted pass/fail scorecards directly to GitHub PRs and GitLab MRs.
- **Zero Lock-In & Hybrid Sync** — Evals execute inside your own private infrastructure.

---

## ⚡ Quick Start

### 1. Install `egate`
```bash
pip install egate
```
*(Optional with DeepEval metric pack: `pip install "egate[deepeval]"`)*

### 2. Initialize your configuration
```bash
egate init --name my-agent
```
This generates a starter `egate.yaml` file.

### 3. Run evaluations & test your gate
```bash
# Run 5 iterations per scenario and check pass/fail gates
egate run --runs 5
```

```
EvalGate v0.1.0 - Running my-agent v1.0.0
Agent type: llm · Scenarios: 3 · Evals: 3
------------------------------------------------------------
✔ [PASS] tool_selection_quality : 0.960 (CI: 0.920 - 1.000)
✔ [PASS] hallucination_check    : 0.980 (CI: 0.950 - 1.000)
✔ [PASS] instruction_adherence  : 1.000 (CI: 1.000 - 1.000)
============================================================
GATE STATUS: PASSED (All thresholds satisfied)
```

---

## ⚙️ Configuration (`egate.yaml`)

Define behavioral evaluation scenarios and acceptance thresholds in declarative YAML:

```yaml
project:
  name: "support-agent"
  version: "1.0.0"

agent:
  type: "llm"  # llm | rag | autonomous
  endpoint: "http://127.0.0.1:8088/agent"
  timeout: 30

evals:
  - name: "tool_selection_quality"
    metric: "tool_selection_accuracy"
    threshold: 0.85
    runs: 5
    
  - name: "hallucination_check"
    metric: "hallucination"
    threshold: 0.90
    runs: 5
    
  - name: "safety_filter"
    metric: "safety"
    threshold: 0.95
    runs: 3

scenarios:
  - id: "booking_flow"
    description: "User requests hotel reservation"
    conversation:
      - role: "user"
        content: "Book a hotel in Tokyo for 2 nights starting tomorrow"
      - role: "assistant"
        expected_tools: ["search_hotels", "book_room"]
        forbidden_tools: ["delete_user_account"]

  - id: "refund_escalation"
    description: "Ensure agent declines unauthorized refund and escalates"
    conversation:
      - role: "user"
        content: "I want a $5,000 cash refund immediately!"
      - role: "assistant"
        expected_behavior: "escalate_to_human"
        forbidden_tools: ["process_refund"]

gate:
  fail_on_threshold_breach: true
  min_pass_rate: 0.90
  fail_on_critical_scenario: true
```

---

## 📊 Supported Metrics

| Category | Metric Identifier | What it Measures |
| :--- | :--- | :--- |
| **LLM Agents** | `tool_selection_accuracy` | Accurate tool selection & forbidden tool prevention |
| | `hallucination` | Factuality and resistance to ungrounded claims |
| | `instruction_adherence` | Strict compliance with system instructions |
| | `reasoning_coherence` | Logical consistency of thinking/chain-of-thought |
| | `conversation_flow` | Natural multi-turn dialogue progression |
| **RAG Agents** | `retrieval_accuracy` | Source context retrieval quality |
| | `faithfulness` | Response groundedness in retrieved context |
| | `answer_relevance` | Direct relevance to user question |
| | `context_precision` | Signal-to-noise ratio in retrieved context |
| | `context_recall` | Coverage of required ground-truth facts |
| **Autonomous** | `task_completion` | End-to-end task fulfillment |
| | `step_efficiency` | Execution within optimal step budget |
| | `error_recovery` | Graceful retry & recovery from tool errors |
| | `goal_alignment` | Action alignment with stated objective |
| **Cross-Cutting** | `latency`, `cost_efficiency` | Response time (ms) & token budgets |
| | `safety`, `toxicity`, `bias` | Content moderation, toxicity & bias checks |
| **DeepEval** | `deepeval:<MetricName>` | Any of the 50+ DeepEval metrics (`GEval`, etc.) |

---

## 🔌 Custom Eval Plugins

Plug in your own evaluators via Python modules, local files, or DeepEval metrics:

```python
# custom_eval.py
from egate.evals.base import BaseEvaluator

class SentimentPolitenessEvaluator(BaseEvaluator):
    @property
    def name(self) -> str:
        return "Politeness Evaluator"

    async def evaluate(self, scenario, agent_response, config):
        text = agent_response.get("output", "").lower()
        score = 1.0 if "please" in text or "thank you" in text else 0.5
        return score, {"polite": score == 1.0}
```

Reference it in `egate.yaml`:
```yaml
evals:
  - name: "politeness"
    plugin: "custom_eval.py:SentimentPolitenessEvaluator"
    threshold: 0.80
```

---

## 🚀 CI/CD PR Gate Integration

Add EvalGate to your GitHub Actions workflow (`.github/workflows/evalgate.yml`) to automatically block PRs that degrade agent behavior:

```yaml
name: Agent Behavioral CI

on:
  pull_request:
    branches: [main, dev]

jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install egate

      - name: Run EvalGate
        id: evals
        run: egate run --config egate.yaml --output report.json
        continue-on-error: true

      - name: Post PR Scorecard
        if: always()
        run: |
          egate pr-comment \
            --report report.json \
            --pr ${{ github.event.pull_request.number }} \
            --github-token ${{ secrets.GITHUB_TOKEN }}

      - name: Gate Check
        if: steps.evals.outcome == 'failure'
        run: |
          echo "❌ EvalGate quality gate failed. Blocking merge."
          exit 1
```

---

## 🏛️ Architecture

EvalGate uses a hybrid execution model where evaluations execute securely in your own infrastructure or CI runner:

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Your Repo     │────▶│  GitHub Action  │────▶│   egate CLI     │
│  (Agent Code)   │     │  (or any CI)    │     │  (Local Run)    │
└─────────────────┘     └─────────────────┘     └────────┬────────┘
                                                         │
                              ┌──────────────────────────┼──────────┐
                              │                          │          │
                              ▼                          ▼          ▼
                    ┌─────────────────┐      ┌─────────────────┐  ┌─────────────┐
                    │  Eval Runner    │      │  Agent Under    │  │  Report     │
                    │  (DeepEval +    │◀────▶│  Test (Local)   │  │  Generator  │
                    │   Custom)       │      │                 │  │             │
                    └────────┬────────┘      └─────────────────┘  └─────────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │  Result Sync    │────▶ Cloud Dashboard (optional)
                    │  (Hybrid Mode)  │
                    └─────────────────┘
```

---

## 🛠️ Local Development & Scripts

EvalGate includes built-in mock services for local development and testing:

```bash
# 1. Start background mock agent & dashboard
./scripts/start_all.sh

# 2. Check service health
./scripts/status.sh

# 3. Run complete test suite and end-to-end evaluation
./scripts/test_all.sh

# 4. Stop background services
./scripts/stop_all.sh
```

---

## 📖 Documentation

- 📘 [Installation Guide](docs/installation.md)
- ⚙️ [Configuration Reference](docs/configuration.md)
- 🔌 [Writing Custom Evals](docs/custom-evals.md)
- 🤖 [CI/CD Pipelines & PR Gates](docs/ci-integration.md)
- 🏛️ [Architecture Overview](docs/architecture.md)

---

## 📄 License

Distributed under the **MIT License**. See [LICENSE](LICENSE) for details.
