Metadata-Version: 2.4
Name: ignis-evaluation-test
Version: 1.0.3
Summary: Config-driven LLM evaluation framework with decorator-based tracing, FastAPI service, and PostgreSQL persistence
Author: Infogain-GenAI
License: MIT
Project-URL: Homepage, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval
Project-URL: Repository, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval.git
Project-URL: Documentation, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval#readme
Project-URL: Bug Tracker, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval/issues
Keywords: deepeval,llm,evaluation,metrics,faithfulness,hallucination,rag,openai,fastapi,config-driven
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: deepeval==4.0.5
Requires-Dist: openai>=1.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: psycopg2-binary>=2.9.0
Provides-Extra: api
Requires-Dist: fastapi>=0.111.0; extra == "api"
Requires-Dist: uvicorn[standard]>=0.29.0; extra == "api"
Requires-Dist: httpx>=0.27.0; extra == "api"
Requires-Dist: pydantic>=2.0.0; extra == "api"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: license-file

# ignis_evaluation

A **config-driven LLM evaluation framework** built on top of [DeepEval](https://github.com/confident-ai/deepeval).  
Define metrics in a YAML bundle config, run evaluations via REST API or decorator, persist scores to PostgreSQL, and view them on a hierarchical evaluation dashboard.

---

## Table of Contents

- [Features](#features)
- [Architecture](#architecture)
- [Complete User Guide (MVP2)](#complete-user-guide-mvp2)

---

## Features

- **YAML bundle configs** — group metrics into named bundles; switch bundles per API call with no code change
- **Hierarchical dashboard** — domain → project → bundles → metrics with KPIs, 6 chart types, and status badges
- **Observability integration** — eval worker reads production traces from an Observability DB and evaluates them automatically
- **Parallel metric execution** — all metrics run concurrently via ThreadPoolExecutor
- **PostgreSQL persistence** — every run, test case, and metric score stored automatically
- **@ignis_eval decorator** — wrap any LLM function in your app to evaluate output inline
- **REST API** — trigger evaluations and retrieve results over HTTP
- **Confident AI upload** — optional push to Confident AI cloud dashboard
- **Offline mode** — run without an OpenAI key using use_deepeval: false

---

## Architecture

### Evaluation Flow Pattern

```
                          ignis_evaluation — Evaluation Flow

 ┌──────────────────────────────┐
 │         Your App             │
 │                              │
 │  Option A: REST API call     │
 │  POST /evaluate              │
 │                              │
 │  Option B: @ignis_eval       │
 │  decorator on any function   │
 └──────────────┬───────────────┘
                │
                ▼
 ┌──────────────────────────────┐
 │       EvaluationRunner       │
 │                              │
 │  1. Loads YAML bundle config │
 │  2. Resolves metric classes  │
 │  3. Builds DeepEval objects  │
 └──────────────┬───────────────┘
                │
                ▼
 ┌──────────────────────────────┐       ┌─────────────────────────────┐
 │   ThreadPoolExecutor         │──────▶│   OpenAI / LLM Judge        │
 │                              │       │   (gpt-4o-mini or gpt-4o)   │
 │  All metrics run in parallel │◀──────│   Scores each metric        │
 │  AnswerRelevancy · GEval     │ scores│   Returns score + reason    │
 │  Faithfulness · Hallucination│       └─────────────────────────────┘
 └──────────────┬───────────────┘
                │  weighted score + per-metric results
                ▼
 ┌──────────────────────────────┐
 │    EvaluationRunner          │
 │    (collects & aggregates)   │
 └──────┬───────────────┬───────┘
        │               │
        ▼               ▼
 ┌─────────────┐  ┌─────────────────┐
 │  PostgreSQL │  │  JSON + CSV     │
 │  eval runs  │  │  reports/       │
 │  test cases │  │  (local files)  │
 │  metrics    │  └─────────────────┘
 └─────────────┘
```

### Observability Integration Pattern

```
                          Observability → Evaluation Pipeline

 ┌─────────────────────┐     writes     ┌──────────────────────┐
 │   Production App    │───────────────▶│   Observability DB   │
 │   (Your LLM)        │    traces      │   traces table       │
 └─────────────────────┘                └──────────┬───────────┘
                                                    │
                                          polls for new traces
                                          (ingested_at > last checkpoint)
                                                    │
                                                    ▼
                                         ┌──────────────────────┐
                                         │     Eval Worker      │
                                         │   eval_worker.py     │
                                         │                      │
                                         │  Reads new traces    │
                                         │  Saves checkpoint    │
                                         │  Batches to API      │
                                         └──────────┬───────────┘
                                                    │
                                          POST /evaluate/batch
                                                    │
                                                    ▼
                                         ┌──────────────────────┐
                                         │   ignis_evaluation   │
                                         │   REST API           │
                                         │   (EvaluationRunner) │
                                         └──────────┬───────────┘
                                                    │
                                  ┌─────────────────┴──────────────────┐
                                  ▼                                     ▼
                       ┌──────────────────┐               ┌────────────────────────┐
                       │   PostgreSQL     │               │   Confident AI         │
                       │   Evaluation DB  │               │   (Cloud Dashboard)    │
                       │   metric_results │               │   optional upload      │
                       │   trace_id link  │               └────────────────────────┘
                       └──────────────────┘
```

---

## Complete User Guide (MVP2)

A **config-driven LLM evaluation framework** built on [DeepEval](https://deepeval.com).  
Evaluate LLM outputs via REST API, Python decorator, or automated observability worker.  
Persist results to PostgreSQL and visualise them on a hierarchical dashboard.

---

## Table of Contents

- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Configuration](#configuration)
- [Database Setup](#database-setup)
- [Starting the API Server](#starting-the-api-server)
- [Project Structure](#project-structure)
- [Built-in Bundles](#built-in-bundles)
- [Core Usage](#core-usage)
  - [Single Evaluation](#1-single-evaluation)
  - [Batch Evaluation](#2-batch-evaluation)
  - [Conversational Evaluation](#3-conversational-evaluation)
  - [Python Decorator](#4-python-decorator)
  - [Custom Bundles](#5-custom-bundles)
- [Dashboard API](#dashboard-api)
- [Observability Worker](#observability-worker)
- [Custom Metrics](#custom-metrics)
- [Troubleshooting](#troubleshooting)

---

## Prerequisites

| Requirement | Version | Notes |
|---|---|---|
| Python | 3.10 or higher | Required for match statements and modern typing |
| PostgreSQL | 14+ | For result persistence. Optional — runs without it |
| OpenAI API Key | — | Required for real LLM-judged evaluations |
| pip | latest | `python -m pip install --upgrade pip` |

---

## Installation

### Option A — Install from PyPI (recommended for integration)

```bash
pip install ignis_evaluation
```

### Option B — Install from source (for development or contribution)

```bash
git clone https://github.com/Infogain-GenAI/ignis_evaluation.git
cd ignis_evaluation

python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate

pip install -e .          # core install
pip install -e ".[api]"   # adds FastAPI + Uvicorn for the REST server
pip install -e ".[dev]"   # adds pytest and test tools
```

---

## Configuration

Copy the example environment file and fill in your values:

```bash
# Windows
copy .env.example .env

# macOS / Linux
cp .env.example .env
```

### Minimum required

```bash
OPENAI_API_KEY=sk-proj-your-key-here
# EVAL_DB_URL=postgresql://postgres:password@localhost:5432/deepeval_results
```

### Full reference

| Variable | Default | Purpose |
|---|---|---|
| `OPENAI_API_KEY` | — | **Required** for live LLM-judged evaluations |
| `IGNIS_MODEL` | `gpt-4o-mini` | LLM judge model for all evaluations. **Always set this explicitly** — if unset, the base config file's `model:` field takes effect, which may be a slower/more expensive model |
| `EVAL_DB_URL` | — | PostgreSQL connection string for result storage |
| `EVAL_REPORTS_DIR` | `./reports` | Where JSON/CSV report files are saved |
| `PROJECT_DOMAIN` | — | Domain tag applied when a new project is created (e.g. Finance) |
| `IGNIS_DEFAULT_PROJECT` | — | Default project name for single-app deployments |
| `IGNIS_DEFAULT_APPLICATION` | — | Default application label (sprint, version, campaign) |
| `IGNIS_BUNDLE_DIR` | — | Path to your custom bundle YAML folder |
| `IGNIS_BUNDLE_STRICT` | `false` | `true` = refuse startup if any custom bundle YAML is invalid |
| `CORS_ALLOWED_ORIGINS` | — | Comma-separated origins allowed for browser clients |
| `OBS_DB_URL` | — | Observability DB (read-only) — required only for the eval worker |
| `EVAL_API_URL` | `http://localhost:8001` | API URL used by the eval worker |
| `EVAL_WORKER_BATCH_SIZE` | `100` | Max traces per worker run |
| `EVAL_WORKER_LOOKBACK_DAYS` | `7` | How far back to look when no checkpoint exists |
| `CONFIDENT_API_KEY` | — | Optional — upload results to Confident AI cloud |
| `DEEPEVAL_TELEMETRY_OPT_OUT` | — | Set `YES` to disable DeepEval usage telemetry |

---

## Database Setup

Skip this section if you want to run without persistence — evaluations still work and results are saved to JSON/CSV reports.

**Step 1 — Create the database**

```sql
-- Run in pgAdmin or psql
CREATE DATABASE deepeval_results;
```

**Step 2 — Apply the schema**

```bash
psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql
```

This creates:
- `projects` — project registry (UUID primary key)
- `evaluation_runs` — one row per evaluation call
- `test_results` — one row per test case
- `metric_results` — one row per metric per test case
- `bundle_registry` — auto-populated with built-in bundles on first startup
- `eval_worker_checkpoints` — observability worker audit trail
- `v_run_metrics` — flat view joining all tables for dashboards

---

## Starting the API Server

```bash
uvicorn src.ignis_evaluation.api.main:app --reload --port 8001
```

Interactive docs (Swagger UI): [http://localhost:8001/docs](http://localhost:8001/docs)

The service auto-registers all built-in bundles into the database on startup.

> **Important — `.env` changes require a full server restart.**  
> `--reload` only watches Python source files. If you change any value in `.env` (model, API key, DB URL, etc.), you must stop the server (`Ctrl+C`) and start it again. The new env values will not be picked up by a hot reload.

---

## Expected Evaluation Timing

Evaluation time depends on the bundle and model used. All metrics in a bundle run concurrently.

| Bundle | Metrics | Expected time (`gpt-4o-mini`) |
|---|---|---|
| `production_monitor` | 1 metric | ~5–8s |
| `core_quality` | 3 metrics | ~20–35s |
| `customer_support` | 3 metrics | ~20–35s |
| `safety_guard` | 2 metrics | ~10–20s |
| `rag_standard` | 6 metrics | ~30–50s |
| `full_audit` | 8+ metrics | ~40–70s |

Times vary with OpenAI network latency. Each metric makes 2–3 internal LLM calls sequentially; the bundle total is bounded by the slowest metric, not their sum.

Using `gpt-4o` instead of `gpt-4o-mini` increases times by **3–5×**. Always set `IGNIS_MODEL=gpt-4o-mini` in `.env` unless you have a specific reason to use a larger model.

---

## Project Structure

```
ignis_evaluation/
├── api/
│   ├── main.py                     # FastAPI entry point
│   ├── models.py                   # Pydantic request/response models
│   └── routers/
│       ├── evaluate.py             # POST /evaluate, /evaluate/batch, /evaluate/conversational
│       ├── bundles.py              # GET /bundles, GET /bundles/{id}
│       ├── analytics.py            # POST /eval/dashboard + KPI/chart endpoints
│       ├── projects.py             # GET /projects, GET /projects/{name}/applications
│       ├── results.py              # GET /results, GET /results/{run_id}
│       ├── health.py               # GET /health
│       └── metrics.py              # GET /metrics
├── bundle_registry/                # Built-in bundle YAML files + loader
│   ├── core_quality.yaml
│   ├── rag_standard.yaml           # All 6 RAG metrics
│   ├── safety_guard.yaml
│   ├── summarization.yaml
│   ├── production_monitor.yaml
│   ├── customer_support.yaml
│   ├── compliance_finance.yaml
│   ├── chatbot_quality.yaml        # Conversational metrics
│   └── validator.py                # Bundle YAML validation
├── db/
│   ├── connection.py
│   ├── writer.py                   # INSERT evaluation runs
│   ├── reader.py                   # SELECT runs for Results API
│   └── analytics.py                # Dashboard queries
├── eval_library/
│   ├── runner.py                   # Core evaluation engine
│   ├── decorator.py                # @ignis_eval decorator
│   ├── conversational_runner.py    # Multi-turn conversation evaluation
│   ├── metric_registry.py          # Maps metric names → DeepEval classes
│   └── scorer.py                   # Weighted composite scoring
├── services/
│   └── dashboard_service.py        # build_enriched_dashboard()
├── sql/
│   └── schema.sql                  # Database schema (run once)
└── workers/
    └── eval_worker.py              # Observability → Eval pipeline
```

---

## Built-in Bundles

Eight bundles ship with the package. Pass the bundle name in any API call.

| Bundle ID | Metrics | Use Case | Context Required? |
|---|---|---|---|
| `core_quality` | AnswerRelevancy, Clarity, Conciseness | Default for any chat app | No |
| `rag_standard` | All 6 RAG metrics (Faithfulness, AnswerRelevancy, Hallucination, ContextualRelevancy, ContextualPrecision, ContextualRecall) | RAG pipelines | Yes |
| `safety_guard` | Toxicity, Bias (GEval) | Public-facing deployments | No |
| `summarization` | Summarization, Faithfulness | Document summarization | Yes |
| `production_monitor` | AnswerRelevancy only | Lightweight live traffic monitoring | No |
| `customer_support` | AnswerRelevancy, Helpfulness, Empathy | Support chatbots | No |
| `compliance_finance` | FinanceCompliance, AnswerRelevancy | Finance domain | No |
| `chatbot_quality` | ConversationCompleteness, RoleAdherence, KnowledgeRetention | Multi-turn chatbot sessions | Use with `/evaluate/conversational` |

List all available bundles at any time:

```bash
GET http://localhost:8001/bundles
```

---

## Core Usage

> **Prerequisite:** Start the API server before running any `curl` example below:
> ```bash
> uvicorn ignis_evaluation.api.main:app --reload --port 8001
> ```
> (See [Starting the API Server](#starting-the-api-server) for the full command when running from source.)

### 1. Single Evaluation

Evaluate one input/output pair.

> **Reports:** Add `"save_report": true` to the request body to also write a JSON+CSV file to `EVAL_REPORTS_DIR` (prefixed `api_...`). Omit it and no report file is written — only the JSON response is returned.

```bash
curl -X POST http://localhost:8001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "project":     "CustomerSupportBot",
    "application": "sprint-23",
    "bundle":      "core_quality",
    "input":       "How do I reset my password?",
    "output":      "Click Forgot Password on the login screen and follow the link sent to your email."
  }'
```

**With RAG context:**

```bash
curl -X POST http://localhost:8001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "project":     "MedicalRAG",
    "application": "v3",
    "bundle":      "rag_standard",
    "input":       "What is the dosage for ibuprofen?",
    "output":      "Standard adult dosage is 200-400mg every 4-6 hours.",
    "context":     ["Ibuprofen is an NSAID. Adult dose 200-400mg every 4-6h."],
    "expected_output": "200-400mg every 4-6 hours for adults."
  }'
```

**Available fields:**

| Field | Required | Description |
|---|---|---|
| `project` | Recommended | Project name — stored in DB and dashboard |
| `application` | Recommended | Version/sprint/campaign label |
| `bundle` | Yes (or `config`) | Bundle ID from the registry |
| `input` | Yes | User query |
| `output` | Yes | LLM answer to evaluate |
| `context` | For RAG bundles | Retrieved chunks |
| `expected_output` | Optional | Ground truth — improves ContextualPrecision/Recall |
| `model` | Optional | Override LLM judge for this call only |

---

### 2. Batch Evaluation

Evaluate multiple cases in one request. Useful for observability trace batches.

```bash
curl -X POST http://localhost:8001/evaluate/batch \
  -H "Content-Type: application/json" \
  -d '{
    "project":     "CustomerSupportBot",
    "application": "sprint-23",
    "bundle":      "core_quality",
    "cases": [
      {"input": "What is ML?",  "output": "ML learns patterns from data.", "test_name": "trace-001"},
      {"input": "What is RAG?", "output": "RAG retrieves documents before generating.", "test_name": "trace-002"}
    ]
  }'
```

Each case can independently have `context`, `expected_output`, and `test_name`.  
Pass the observability `trace_id` as `test_name` to link eval scores back to production traces.

---

### 3. Conversational Evaluation

Evaluate a full multi-turn conversation session.

```bash
curl -X POST http://localhost:8001/evaluate/conversational \
  -H "Content-Type: application/json" \
  -d '{
    "project":     "CustomerSupportBot",
    "application": "sprint-23",
    "bundle":      "chatbot_quality",
    "turns": [
      {"user": "Hi, I cannot log in.", "assistant": "Can you tell me the error message?"},
      {"user": "It says password incorrect.", "assistant": "I will send a reset link to your email."},
      {"user": "Got it, reset done.", "assistant": "Great! You should be able to log in now."}
    ],
    "test_name": "session-abc123"
  }'
```

Conversational metrics evaluate the **full conversation as a unit**, not individual turns:

- **ConversationCompletenessMetric** — did the bot fully resolve the user's need?
- **RoleAdherenceMetric** — did the bot stay in its assigned persona/role?
- **KnowledgeRetentionMetric** — did the bot remember what was said in earlier turns?

---

### 4. Python Decorator

Wrap any LLM function to evaluate its output inline.

**Installation (from your app):**

```bash
pip install ignis_evaluation
```

**Bundle-only (recommended — no config file needed):**

```python
from dotenv import load_dotenv
load_dotenv(".env")

import openai, openai.resources.chat  # pre-import to avoid thread deadlock
from ignis_evaluation.eval_library.decorator import ignis_eval

@ignis_eval(
    bundle="core_quality",
    project="CustomerSupportBot",
    application="sprint-23",
    include_results=True,
)
def handle_query(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

result = handle_query(
    input_text="How do I reset my password?",
    output_text="Click Forgot Password on the login screen.",
)

print("Score:", result["evaluation"]["weighted_score"])
print("Bundle:", result["evaluation"]["bundle"])
for metric, data in result["evaluation"]["metrics"].items():
    status = "PASS" if data.get("pass") else "FAIL"
    print(f"  {metric}: {data.get('score')} — {status}")
```

**With full report in result:**

```python
@ignis_eval(
    bundle="rag_standard",
    project="MedicalRAG",
    application="v3",
    include_results=True,
    include_reports=True,   # adds "report" key with serializable report dict
    save_report=True,       # also saves JSON+CSV to EVAL_REPORTS_DIR
)
def rag_answer(*, input_text, output_text, context):
    return {"input_text": input_text, "output_text": output_text, "context": context}
```

**Decorator parameters:**

| Parameter | Default | Description |
|---|---|---|
| `bundle` | — | Bundle ID from registry (recommended) |
| `config` | — | YAML config name (V1 compat) |
| `project` | env default | Project name |
| `application` | env default | Application label |
| `include_results` | `False` | Return `{"result": ..., "evaluation": {...}}` |
| `include_reports` | `False` | Also include `"report"` key in return dict |
| `save_report` | `False` | Save JSON+CSV files to `EVAL_REPORTS_DIR` |
| `eval_model` | `IGNIS_MODEL` | Override LLM judge for this decorator |

> **Note — reports are opt-in here too:** Just like the API's `save_report` field, the decorator's `save_report` defaults to `False`. Set `save_report=True` explicitly to get JSON+CSV files (prefixed `decorator_...`) written to `EVAL_REPORTS_DIR` — otherwise only DB persistence (if configured) happens, no report files.

---

### 5. Custom Bundles

Create your own bundle YAML outside the package.

**Step 1 — Create your bundles folder and YAML file:**

```
your_project/
├── my_bundles/
│   └── support_quality.yaml
└── .env
```

```yaml
# ./my_bundles/support_quality.yaml
bundle_id:   support_quality
description: "Quality check for customer support responses."

metrics:
  - name:      AnswerRelevancyMetric
    threshold: 0.70
    weight:    40

  - name:      GEvalMetric
    geval_name: "Clarity"
    criteria:   "The response must be clear and easy to understand for a non-expert."
    evaluation_params: [input, actual_output]
    threshold:  0.65
    weight:     35

  - name:      GEvalMetric
    geval_name: "Politeness"
    criteria:   "The response must be polite and professional in tone."
    evaluation_params: [input, actual_output]
    threshold:  0.60
    weight:     25
```

> Weights across all metrics in a bundle must sum to exactly 100. Add as many YAML files as you want — one bundle per file.

**Step 2 — Point the service to your folder in `.env`:**

```bash
IGNIS_BUNDLE_DIR=./my_bundles
IGNIS_BUNDLE_STRICT=false
```

**Step 3 — Restart the server, then use by name exactly like built-in bundles:**

```bash
uvicorn ignis_evaluation.api.main:app --reload --port 8001
```

```json
POST /evaluate
{
  "project": "CustomerSupportBot",
  "bundle":  "support_quality",
  "input":   "My order hasn't arrived, what do I do?",
  "output":  "I'm sorry for the delay! I've checked your order and it's on the way — you should receive it within 2 days."
}
```

**Validation rules:** The service validates every YAML at startup.

| Field | Rule |
|---|---|
| `bundle_id` | Required, non-empty string |
| `metrics` | Required, non-empty list |
| `name` (per metric) | Must be a known built-in metric name or include `module:` |
| `threshold` | Float 0.0–1.0 |
| `weight` | Positive number |
| `criteria` | Required when name is `GEvalMetric` |

Set `IGNIS_BUNDLE_STRICT=true` to refuse service startup on invalid bundles (recommended for production).

---

## Dashboard API

Single endpoint for KPIs, charts, and the full project hierarchy.

```bash
POST http://localhost:8001/eval/dashboard
```

**Filter by project:**

```json
{
  "project":     "CustomerSupportBot",
  "window_hours": 168
}
```

**Filter by domain:**

```json
{
  "domain":      "Finance",
  "window_hours": 168
}
```

**Include recent evaluations:**

```json
{
  "project":             "CustomerSupportBot",
  "window_hours":         168,
  "include_evaluations":  true,
  "eval_limit":           20
}
```

**Response structure:**

```
{
  "kpis":     { evaluation_count, avg_weighted_score, pass_rate_pct, total_cost_usd, ... },
  "charts":   { score_over_time, pass_fail, cost_by_metric, score_by_metric, ... },
  "projects": [
    {
      "project_name":  "CustomerSupportBot",
      "domain":        "Finance",              ← metadata on project
      "applications": [
        {
          "application_name": "sprint-23",
          "bundles": [
            {
              "bundle":           "core_quality",
              "avg_weighted_score": 0.87,
              "metrics": [ ... ]
            }
          ]
        }
      ]
    }
  ]
}
```

**Other useful endpoints:**

| Endpoint | Description |
|---|---|
| `GET /bundles` | List all available bundle IDs |
| `GET /bundles/{bundle_id}` | Full bundle detail — metrics, thresholds, weights |
| `GET /projects` | List all projects |
| `GET /projects/{name}/applications` | List applications under a project with stats |
| `GET /results` | Paginated list of evaluation runs |
| `GET /results/{run_id}` | Full detail for one run with all metric scores |
| `GET /eval/metrics/leaderboard` | Per-metric pass rate ranking |
| `GET /eval/projects/{name}/trend` | Score trend over time for a project |
| `GET /eval/domains` | List all domains with project and run counts |
| `GET /health` | Service health check |

---

## Observability Worker

Automatically evaluates production LLM traces from an observability database.

**How it works:**

1. Reads new traces from your Observability DB (`traces` table with `input`, `output`, `trace_id`)
2. Sends them as a batch to `POST /evaluate/batch`
3. Links each eval score back to its trace via `test_name = trace_id`
4. Saves a checkpoint so next run only processes new traces

**Configure in `.env`:**

```bash
OBS_DB_URL=postgresql://user:pass@host:5432/your_obs_db
EVAL_API_URL=http://localhost:8001
EVAL_WORKER_BATCH_SIZE=100
EVAL_WORKER_LOOKBACK_DAYS=7

# Map obs project names → bundle (JSON string)
EVAL_PROJECT_CONFIG_MAP={"my_chatbot": {"config": "chatapp", "bundle": "core_quality"}}
```

**Run the worker:**

```bash
# All projects
python -m ignis_evaluation.workers.eval_worker

# Single project
python -m ignis_evaluation.workers.eval_worker --project my_chatbot

# With bundle override
python -m ignis_evaluation.workers.eval_worker --project my_chatbot --bundle rag_standard

# Dry run (logs without calling the API)
python -m ignis_evaluation.workers.eval_worker --dry-run
```

Worker runs appear in the dashboard under `application: "observability"` separate from direct API calls.

---

## Custom Metrics

Three ways to add a custom metric — start with the simplest that fits your need.

| Tier | Who | What | Python file? |
|---|---|---|---|
| **1 — `KeywordCheckMetric`** | Anyone | Phrase / keyword checks via YAML lists | No |
| **2 — `GEvalMetric` + criteria** | Anyone | Plain-English LLM-judged rule | No |
| **3 — Custom Python class** | Developers | Any logic — regex, ML model, API call | Yes |

---

### Tier 1 — `KeywordCheckMetric` (no Python, YAML only)

The simplest option. Add it to any bundle YAML with two lists — no Python file, no class, no scoring formula.

```yaml
# my_bundles/brand_quality.yaml
bundle_id: brand_quality

metrics:
  - name:      AnswerRelevancyMetric
    threshold: 0.70
    weight:    50

  - name:             KeywordCheckMetric
    must_contain:     ["happy to help", "let me", "feel free"]
    must_not_contain: ["obviously", "you must", "that's wrong"]
    scoring:          fraction       # fraction = partial credit | all_required = all or nothing
    threshold:        0.50
    weight:           50
```

**Fields:**

> **Restart required after adding a new bundle YAML.**  
> Bundle YAMLs are loaded once at server startup. Add your YAML to `IGNIS_BUNDLE_DIR` and restart uvicorn before using the new bundle.

| Field | Required | Description |
|---|---|---|
| `must_contain` | One of the two is required | Phrases that should appear in the output |
| `must_not_contain` | One of the two is required | Phrases that must NOT appear |
| `scoring` | No (default: `fraction`) | `fraction` = partial credit · `all_required` = all or nothing |
| `threshold` | Yes | Pass/fail threshold (0.0–1.0) |
| `weight` | Yes | Bundle weight (all weights must sum to 100) |

**Scoring rules:**

1. If **any** `must_not_contain` phrase is found → score = **0.0** (hard fail, overrides everything)
2. `fraction` (default): score = phrases found ÷ total `must_contain` phrases
3. `all_required`: score = 1.0 only if **all** phrases found, otherwise 0.0

**Examples:**

| `must_contain` | Output | Scoring | Score |
|---|---|---|---|
| `["ticket #", "reference"]` | "Created ticket #1234 as your reference." | fraction | 1.0 |
| `["ticket #", "reference"]` | "Created ticket #1234." | fraction | 0.5 |
| `["ticket #", "reference"]` | "I've noted your issue." | fraction | 0.0 |
| `["a", "b", "c"]` | "a and b present" | all_required | 0.0 |
| — | "Sorry we cannot help." (banned phrase) | any | **0.0** |

---

### Tier 2 — `GEvalMetric` with plain English criteria (already built-in)

When you need an LLM to judge something that can't be captured with keyword lists:

```yaml
- name:       GEvalMetric
  geval_name: "BrandTone"
  criteria:   "The response must use a warm, professional tone. It should not sound dismissive, rushed, or robotic."
  evaluation_params: [input, actual_output]
  threshold:  0.70
  weight:     30
```

Write what you want in plain English — the LLM evaluates it. No Python file needed.

---

### Tier 3 — Custom Python class (developers only)

For checks that need real code — regex, ML models, database lookups, complex scoring:

**Step 1 — Create a Python file:**

```python
# myapp/metrics/ticket_check.py
from ignis_evaluation.eval_library.metric_registry import BaseMetric, MetricResult
import re

class TicketNumberMetric(BaseMetric):
    """Checks that a support ticket reference (#12345) is included."""

    def __init__(self, threshold: float = 0.7, weight: float = 100, **kwargs):
        self.threshold = float(threshold)
        self.weight    = float(weight)

    def evaluate(self, test_case: dict) -> MetricResult:
        output          = test_case.get("output_text") or ""
        has_ticket_ref  = bool(re.search(r"#\d{4,8}", output))
        has_ticket_word = "ticket" in output.lower()
        score           = 1.0 if (has_ticket_ref and has_ticket_word) else 0.5 if has_ticket_word else 0.0

        return MetricResult(
            score=round(score, 4),
            details={
                "reason":   "Ticket reference found." if score == 1.0 else "Missing ticket number.",
                "has_ref":  has_ticket_ref,
                "has_word": has_ticket_word,
            },
        )
```

**Step 2 — Reference it in the bundle YAML with `module:`:**

```yaml
- name:      TicketNumberMetric
  module:    myapp.metrics.ticket_check.TicketNumberMetric
  threshold: 0.80
  weight:    25
```

**Step 3 — Set `IGNIS_BUNDLE_DIR` in `.env` and use the bundle name as usual. Done.**

The `module:` value is the same dotted path you'd use in `from myapp.metrics.ticket_check import TicketNumberMetric`.

> **Restart required after adding a new bundle YAML.**  
> Bundle files are loaded once at server startup. If you add a new YAML to `IGNIS_BUNDLE_DIR` while the server is running, restart uvicorn before calling the API — otherwise the server returns `"bundle not found"`.

**`evaluate()` contract:**

| Requirement | Detail |
|---|---|
| Inherit from | `ignis_evaluation.eval_library.metric_registry.BaseMetric` |
| Implement | `evaluate(self, test_case: dict) -> MetricResult` |
| `test_case` keys available | `input_text`, `output_text`, `context`, `expected_output`, `retrieval_context` |
| Return | `MetricResult(score=float, details=dict)` — score must be `0.0–1.0` |
| Constructor | Must accept `threshold` and `weight` kwargs from YAML |

---

### Built-in rule-based metrics (no setup needed)

| Metric | Checks | Python file? |
|---|---|---|
| `KeywordCheckMetric` | Any must_contain / must_not_contain lists | No |
| `FinanceComplianceMetric` | Banned finance phrases (guaranteed returns, risk-free) | No |
| `MedicalAccuracyMetric` | Dangerous medical claims | No |
| `CompositeQualityMetric` | Faithfulness + AnswerRelevancy blended | No (needs LLM) |
| `SafetyComplianceMetric` | Toxicity + Hallucination blended | No (needs LLM) |

---

### Project structure (Tier 3 only)

```
your_project/
├── myapp/
│   ├── __init__.py
│   └── metrics/
│       ├── __init__.py
│       └── ticket_check.py     ← custom metric class
├── my_bundles/
│   └── support_quality.yaml    ← references module: myapp.metrics.ticket_check...
├── .env                        ← IGNIS_BUNDLE_DIR=./my_bundles
└── main.py
```

---

## Troubleshooting

### `DB write failed: name '_infer_provider' is not defined`

The writer module has a function ordering issue. Run `pip install --upgrade ignis_evaluation` or restart the server after a fresh install.

### Metrics return placeholder scores (0.75) instead of real scores

The config or bundle is running in offline mode. Check that `OPENAI_API_KEY` is set in `.env` and the config file has `use_deepeval: true` (for V1 configs).

### `bundle 'xyz' not found`

- Verify the bundle name spelling: `GET /bundles` returns the full list.
- For custom bundles, confirm `IGNIS_BUNDLE_DIR` is set and the YAML file exists in that folder.
- Restart the server — bundles load at startup, not per request.

### `ConnectionError: Cannot connect to Eval API`

The eval worker cannot reach the server. Confirm the server is running and `EVAL_API_URL` is correct in `.env`.

### `DB not configured` in health check

`EVAL_DB_URL` is not set or is invalid. Evaluations still work — results go to JSON/CSV files in `EVAL_REPORTS_DIR`. Set `EVAL_DB_URL` to enable persistence.

### Timeout errors during evaluation

DeepEval metrics can be slow for large batches. Increase timeouts in `.env`:

```bash
DEEPEVAL_PER_TASK_TIMEOUT=600
DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS_OVERRIDE=300
```

### Swagger UI "Try it out" calls fail (CORS error)

Add your Swagger URL to `CORS_ALLOWED_ORIGINS`:

```bash
CORS_ALLOWED_ORIGINS=http://localhost:8001
```

### `IGNIS_BUNDLE_STRICT=true` prevents server from starting

A custom bundle YAML has validation errors. Check the startup log for the exact error and field. Fix the YAML or set `IGNIS_BUNDLE_STRICT=false` while developing.

---

## Quick Reference — Common Workflows

### Evaluate a chatbot response (minimal)

```bash
POST /evaluate
{ "bundle": "core_quality", "input": "...", "output": "..." }
```

### Evaluate a RAG response with context

```bash
POST /evaluate
{ "bundle": "rag_standard", "input": "...", "output": "...", "context": ["..."] }
```

### Evaluate a full chatbot session

```bash
POST /evaluate/conversational
{ "bundle": "chatbot_quality", "turns": [{"user":"...", "assistant":"..."}, ...] }
```

### View dashboard for a project (last 7 days)

```bash
POST /eval/dashboard
{ "project": "MyProject", "window_hours": 168 }
```

### Add a custom bundle and use it

```bash
# 1. Create ./my_bundles/my_bundle.yaml
# 2. Set IGNIS_BUNDLE_DIR=./my_bundles in .env
# 3. Restart server
# 4. POST /evaluate  { "bundle": "my_bundle", ... }
```

### Run production trace evaluation

```bash
python -m ignis_evaluation.workers.eval_worker --project my_project
```

