Metadata-Version: 2.4
Name: ignis_evaluation
Version: 1.0.2
Summary: Config-driven LLM evaluation framework with decorator-based tracing, FastAPI service, and PostgreSQL persistence
Author: Infogain-GenAI
License: MIT
Project-URL: Homepage, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval
Project-URL: Repository, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval.git
Project-URL: Documentation, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval#readme
Project-URL: Bug Tracker, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval/issues
Keywords: deepeval,llm,evaluation,metrics,faithfulness,hallucination,rag,openai,fastapi,config-driven
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: deepeval==4.0.5
Requires-Dist: openai>=1.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: psycopg2-binary>=2.9.0
Provides-Extra: api
Requires-Dist: fastapi>=0.111.0; extra == "api"
Requires-Dist: uvicorn[standard]>=0.29.0; extra == "api"
Requires-Dist: httpx>=0.27.0; extra == "api"
Requires-Dist: pydantic>=2.0.0; extra == "api"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: license-file

# ignis_evaluation

A **config-driven LLM evaluation framework** built on top of [DeepEval](https://github.com/confident-ai/deepeval).  
Define metrics in a YAML bundle config, run evaluations via REST API or decorator, persist scores to PostgreSQL, and view them on a hierarchical evaluation dashboard.

---

## Table of Contents

- [Features](#features)
- [Architecture](#architecture)
- [Project Structure](#project-structure)
- [Quick Start](#quick-start)
- [Running Tests & Examples](#running-tests--examples)
- [Configuration Reference](#configuration-reference)
- [Metric Bundles](#metric-bundles)
- [REST API](#rest-api)
- [Evaluation Dashboard](#evaluation-dashboard)
- [Decorator-Driven Evaluation](#decorator-driven-evaluation-ignis_eval)
- [Observability Integration](#observability-integration)
- [Database Setup](#database-setup)
- [Environment Variables Reference](#environment-variables-reference)
- [Troubleshooting](#troubleshooting)
- [References](#references)

---

## Features

- **YAML bundle configs** — group metrics into named bundles; switch bundles per API call with no code change
- **Hierarchical dashboard** — domain → project → bundles → metrics with KPIs, 6 chart types, and status badges
- **Observability integration** — eval worker reads production traces from an Observability DB and evaluates them automatically
- **Parallel metric execution** — all metrics run concurrently via `ThreadPoolExecutor`
- **PostgreSQL persistence** — every run, test case, and metric score stored automatically
- **`@ignis_eval` decorator** — wrap any LLM function in your app to evaluate output inline
- **REST API** — trigger evaluations and retrieve results over HTTP
- **Confident AI upload** — optional push to Confident AI cloud dashboard
- **Offline mode** — run without an OpenAI key using `use_deepeval: false`

---

## Architecture

### Evaluation Flow Pattern

```
                          ignis_evaluation — Evaluation Flow

 ┌──────────────────────────────┐
 │         Your App             │
 │                              │
 │  Option A: REST API call     │
 │  POST /evaluate              │
 │                              │
 │  Option B: @ignis_eval       │
 │  decorator on any function   │
 └──────────────┬───────────────┘
                │
                ▼
 ┌──────────────────────────────┐
 │       EvaluationRunner       │
 │                              │
 │  1. Loads YAML bundle config │
 │  2. Resolves metric classes  │
 │  3. Builds DeepEval objects  │
 └──────────────┬───────────────┘
                │
                ▼
 ┌──────────────────────────────┐       ┌─────────────────────────────┐
 │   ThreadPoolExecutor         │──────▶│   OpenAI / LLM Judge        │
 │                              │       │   (gpt-4o-mini or gpt-4o)   │
 │  All metrics run in parallel │◀──────│   Scores each metric        │
 │  AnswerRelevancy · GEval     │ scores│   Returns score + reason    │
 │  Faithfulness · Hallucination│       └─────────────────────────────┘
 └──────────────┬───────────────┘
                │  weighted score + per-metric results
                ▼
 ┌──────────────────────────────┐
 │    EvaluationRunner          │
 │    (collects & aggregates)   │
 └──────┬───────────────┬───────┘
        │               │
        ▼               ▼
 ┌─────────────┐  ┌─────────────────┐
 │  PostgreSQL │  │  JSON + CSV     │
 │  eval runs  │  │  reports/       │
 │  test cases │  │  (local files)  │
 │  metrics    │  └─────────────────┘
 └─────────────┘
```

### Observability Integration Pattern

```
                          Observability → Evaluation Pipeline

 ┌─────────────────────┐     writes     ┌──────────────────────┐
 │   Production App    │───────────────▶│   Observability DB   │
 │   (Your LLM)        │    traces      │   traces table       │
 └─────────────────────┘                └──────────┬───────────┘
                                                    │
                                          polls for new traces
                                          (ingested_at > last checkpoint)
                                                    │
                                                    ▼
                                         ┌──────────────────────┐
                                         │     Eval Worker      │
                                         │   eval_worker.py     │
                                         │                      │
                                         │  Reads new traces    │
                                         │  Saves checkpoint    │
                                         │  Batches to API      │
                                         └──────────┬───────────┘
                                                    │
                                          POST /evaluate/batch
                                                    │
                                                    ▼
                                         ┌──────────────────────┐
                                         │   ignis_evaluation   │
                                         │   REST API           │
                                         │   (EvaluationRunner) │
                                         └──────────┬───────────┘
                                                    │
                                  ┌─────────────────┴──────────────────┐
                                  ▼                                     ▼
                       ┌──────────────────┐               ┌────────────────────────┐
                       │   PostgreSQL     │               │   Confident AI         │
                       │   Evaluation DB  │               │   (Cloud Dashboard)    │
                       │   metric_results │               │   optional upload      │
                       │   trace_id link  │               └────────────────────────┘
                       └──────────────────┘
```

### Tech Stack

| Layer | Technology | Purpose |
|---|---|---|
| **Language** | Python 3.10+ | Modern typing, `match` statements, `asyncio` support |
| **API Framework** | FastAPI + Uvicorn | High-performance async REST API with auto Swagger docs |
| **Evaluation Engine** | DeepEval | Built-in RAG, GEval, hallucination, and toxicity metrics |
| **LLM Judge** | OpenAI (`gpt-4o-mini`) | Scores responses via structured LLM calls |
| **Concurrency** | `ThreadPoolExecutor` | Runs all metrics in parallel per evaluation request |
| **Database** | PostgreSQL | Persists runs, test cases, and per-metric scores |
| **ORM / Driver** | `psycopg2` | Direct PostgreSQL connection and query execution |
| **Config** | YAML (`PyYAML`) | Declarative metric bundles, thresholds, and weights |
| **Validation** | Pydantic v2 | Type-safe request/response models for the API |
| **Observability** | Confident AI (optional) | Cloud dashboard upload via `CONFIDENT_API_KEY` |
| **Reporting** | JSON + CSV | Local report files written to `reports/` per run |
| **Testing** | pytest + `httpx` | Unit and integration tests via FastAPI `TestClient` |

---

## Project Structure

```
deepeval_demo/
├── src/
│   └── ignis_evaluation/
│       ├── api/
│       │   ├── main.py             # FastAPI entry point, router registration
│       │   ├── models.py           # Pydantic request/response models
│       │   └── routers/
│       │       ├── evaluate.py     # POST /evaluate, /evaluate/batch, /evaluate/dataset
│       │       ├── analytics.py    # POST /eval/dashboard + other analytics endpoints
│       │       ├── projects.py     # GET /projects, GET /projects/{name}/bundles
│       │       ├── results.py      # GET /results, GET /results/{run_id}
│       │       ├── metrics.py      # GET /metrics
│       │       ├── health.py       # GET /health
│       │       └── chat.py         # POST /chat (decorator-driven)
│       ├── configs/                # Built-in YAML configs shipped with the package
│       │   ├── chatapp_config.yaml         # Chat app: 9 bundles, input+output only
│       │   ├── geval_bundle_config.yaml    # GEval custom criteria bundles
│       │   ├── bundle_config.yaml          # General purpose quality+safety bundles
│       │   ├── compliance_bundle_config.yaml  # Finance compliance bundles
│       │   ├── rag_config.yaml             # RAG pipeline metrics
│       │   ├── healthcare_config.yaml      # Healthcare domain metrics
│       │   └── ...
│       ├── db/
│       │   ├── connection.py       # PostgreSQL connection
│       │   ├── writer.py           # INSERT evaluation runs, test & metric results
│       │   ├── reader.py           # SELECT runs for Results API
│       │   └── analytics.py        # Dashboard queries (KPIs, charts, bundles)
│       ├── eval_library/
│       │   ├── runner.py           # Core evaluation engine (EvaluationRunner)
│       │   ├── decorator.py        # @ignis_eval decorator
│       │   ├── scorer.py           # Weighted composite scoring
│       │   ├── metric_registry.py  # Maps metric names → DeepEval classes
│       │   └── ...
│       ├── services/
│       │   └── dashboard_service.py  # build_enriched_dashboard() — usable as a library
│       ├── workers/
│       │   └── eval_worker.py      # Observability → Eval pipeline worker
│       └── sql/
│           └── schema.sql          # Database schema (run once)
├── examples_test/
│   ├── configs/                    # Example YAML configs
│   ├── datasets/                   # Versioned evaluation datasets
│   └── ...
├── docs/
│   ├── EVALUATION_ON_OBSERVABILITY.md
│   ├── OBSERVABILITY_EVAL_APPROACH.md
│   ├── DASHBOARD_UI_WIREFRAME.md
│   └── OBSERVABILITY_EVAL_INTEGRATION.md
├── .env.example
├── pyproject.toml
└── README.md
```

---

## Quick Start

### 1. Clone & create virtual environment

```bash
git clone https://github.com/Infogain-GenAI/ignis_evaluation_deepeval.git
cd deepeval_demo

python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate
```

### 2. Install dependencies

```bash
pip install -e .          # base install
pip install -e ".[api]"   # adds FastAPI server
pip install -e ".[dev]"   # adds test/dev tools
```

### 3. Configure environment

```bash
copy .env.example .env    # Windows
cp .env.example .env      # macOS/Linux
```

Minimum required variables in `.env`:

```
OPENAI_API_KEY=sk-...

```

### 4. Set up the database

```bash
# Create the database first:
# CREATE DATABASE deepeval_results;

# Apply schema:
psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql
```

### 5. Start the API server

```bash
uvicorn src.ignis_evaluation.api.main:app --reload --port 8001
```

Interactive docs: [http://localhost:8001/docs](http://localhost:8001/docs)

### 6. Run your first evaluation

```bash
curl -X POST http://localhost:8001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "config": "chatapp",
    "bundle": "core_quality",
    "input": "What is diabetes?",
    "output": "Diabetes is a chronic condition that affects how the body regulates blood sugar."
  }'
```

You should see a JSON response with `weighted_score`, `passed`, and per-metric results.

---

## Running Tests & Examples

Three copy-paste examples to verify the library is working. Start with Example 1 (no API key needed), then progress as needed.

### Prerequisites

```bash
pip install -e ".[dev]"
```

---

### Example 1 — Offline evaluation (no API key needed)

Uses `TestClient` to run the FastAPI app **in-process** — no server to start, no network calls, instant. Setting `use_deepeval: false` skips all LLM calls so no OpenAI key or database is required.

**Step 1 — Create the config file** (`my_offline_config.yaml`):

```yaml
# my_offline_config.yaml
project:             my_project
model:               gpt-4o-mini    # LLM judge (not called when use_deepeval: false)
use_deepeval:        false          # offline mode — returns placeholder scores instantly

composite_threshold: 0.65

metrics:
  - name:      FaithfulnessMetric   # checks answer is grounded in context (no hallucination)
    threshold: 0.80
    weight:    40

  - name:      AnswerRelevancyMetric  # checks answer directly addresses the question
    threshold: 0.75
    weight:    35

  - name:      HallucinationMetric   # penalises claims not in the retrieved context
    threshold: 0.80
    weight:    25

reporting:
  format:     json
  output_dir: reports
```

> The built-in equivalent is `src/ignis_evaluation/configs/rag_config.yaml`. Change `use_deepeval: true` there to go live with real LLM scoring.

**Step 2 — Run the evaluation**:

```bash
python examples_test/test_api.py
```

Or paste this into a Python file and run it:

```python
# If you installed via `pip install ignis_evaluation`, remove the two lines below.
# They are only needed when running from the cloned repo with `pip install -e .`
import sys
from pathlib import Path
sys.path.insert(0, str(Path(".") / "src"))

from fastapi.testclient import TestClient
from ignis_evaluation.api.main import app

client = TestClient(app)

# Health check
print(client.get("/health").json())

# Offline evaluation — use_deepeval: false means no OpenAI call, runs instantly
result = client.post("/evaluate", json={
    "config": "rag",                 # loads src/ignis_evaluation/configs/rag_config.yaml
                                     # or pass config_path to use your own file
    "input": "What is Python?",
    "output": "Python is a high-level programming language.",
    "context": ["Python is a high-level, general-purpose programming language."],
}).json()
print("Score :", result.get("weighted_score"))
print("Passed:", result.get("passed"))
for name, m in result.get("metrics", {}).items():
    status = "PASS" if m.get("pass") else "FAIL"
    print(f"  {name}: {m.get('score')} — {status}")
```

**Expected output (runs in seconds):**
```
{'status': 'ok', 'db': 'not_configured', ...}
Score : 0.75
Passed: True
  FaithfulnessMetric: 0.75 — PASS
  AnswerRelevancyMetric: 0.75 — PASS
  HallucinationMetric: 0.75 — PASS
```

---

### Example 2 — GEval custom criteria (requires `OPENAI_API_KEY`)

GEval lets you define your own rubric criteria instead of using fixed metrics. Each criterion is a plain-English sentence describing what "good" means for your use case.

**Step 1 — Create the config file** (`my_geval_config.yaml`):

```yaml
# my_geval_config.yaml
project:             my_project
model:               gpt-4o-mini   # LLM used as the judge
use_deepeval:        true          # false = offline/mock mode (no API key needed)
composite_threshold: 0.65          # weighted score must exceed this to pass

metrics:
  - name:       GEvalMetric
    geval_name: "Correctness"      # label shown in results table
    criteria: >
      The actual output must be factually correct and consistent with
      the provided context. Any claim not supported by the context
      should be penalised.
    evaluation_params: [input, actual_output, context]
    evaluation_steps:
      - "Check if every factual claim is supported by the context."
      - "Penalise any claim that contradicts or is absent from the context."
      - "Score 0–1: 1 = fully correct, 0 = entirely wrong."
    threshold: 0.70
    weight:    40

  - name:       GEvalMetric
    geval_name: "Clarity"
    criteria: >
      The actual output must be written in clear, plain language that
      a non-expert can understand. Avoid unnecessary jargon.
    evaluation_params: [input, actual_output]
    evaluation_steps:
      - "Is the language simple and accessible to a general audience?"
      - "Is the response free from unnecessary technical jargon?"
      - "Is it logically structured with a clear point?"
    threshold: 0.65
    weight:    35

  - name:       GEvalMetric
    geval_name: "Conciseness"
    criteria: >
      The actual output must answer the question without unnecessary
      repetition, padding, or off-topic content.
    evaluation_params: [input, actual_output]
    evaluation_steps:
      - "Does the response stay on topic?"
      - "Is there repetition of the same point?"
      - "Is the length appropriate for the question asked?"
    threshold: 0.60
    weight:    25
```

> The full working example is at [examples_test/configs/geval_config.yaml](examples_test/configs/geval_config.yaml).

**Step 2 — Run the evaluation**:

```bash
python examples_test/test_geval.py
```

Or paste this into a Python file:

```python
# If you installed via `pip install ignis_evaluation`, the sys.path lines are not needed.
# Only uncomment them if running from the cloned repo without pip install -e .
# import sys; from pathlib import Path
# sys.path.insert(0, str(Path(".") / "src"))

from pathlib import Path
from dotenv import load_dotenv
load_dotenv(".env")  # loads OPENAI_API_KEY

# Pre-import openai before ThreadPoolExecutor starts — prevents Python 3.12
# import-lock deadlock when parallel metric threads first-import openai.
import openai  # noqa: F401
import openai.resources.chat  # noqa: F401

from ignis_evaluation.eval_library.runner import EvaluationRunner

# Point to your config file (or use the provided example)
config_path = Path("my_geval_config.yaml")  # or: Path("examples_test/configs/geval_config.yaml")
runner = EvaluationRunner(config_path)

result = runner.run(
    input_text="What is photosynthesis?",
    actual_output=(
        "Photosynthesis is the process by which plants convert sunlight, "
        "water, and carbon dioxide into glucose and oxygen."
    ),
    context=[
        "Photosynthesis converts light energy into chemical energy stored as glucose.",
        "It requires sunlight, water (H2O), and carbon dioxide (CO2).",
        "Oxygen (O2) is released as a byproduct.",
    ],
)
print("Score:", result.get("weighted_score"))
for name, m in result.get("metrics", {}).items():
    status = "PASS" if m.get("pass") else "FAIL"
    print(f"  {name}: {m.get('score')} — {status}")
```

**Expected output:**
```
Score: 0.91
  Correctness: 0.95 — PASS
  Clarity: 0.88 — PASS
  Conciseness: 0.89 — PASS
```

---

### Example 3 — Decorator (`@ignis_eval`)

The `@ignis_eval` decorator wraps any LLM function to evaluate its output inline using a **bundle** — a named group of metrics defined in a config file. No separate API call needed.

**Step 1 — Understand the config** (`src/ignis_evaluation/configs/chatapp_config.yaml`):

```yaml
# chatapp_config.yaml  (excerpt — the built-in config used by config="chatapp")
project:             chatapp
model:               gpt-4o
use_deepeval:        true
composite_threshold: 0.65
default_bundle:      core_quality

metric_bundles:

  core_quality:           # <-- bundle name passed to @ignis_eval
    - name:       AnswerRelevancyMetric
      threshold:  0.70
      weight:     40

    - name:       GEvalMetric
      geval_name: "Clarity"
      criteria: >
        The response must be written in clear, plain language that a
        non-expert can understand. Avoid jargon and be logically structured.
      evaluation_params: [input, actual_output]
      threshold:  0.65
      weight:     35

    - name:       GEvalMetric
      geval_name: "Conciseness"
      criteria: >
        The response must answer the question without unnecessary
        repetition, padding, or off-topic content.
      evaluation_params: [input, actual_output]
      threshold:  0.60
      weight:     25
```

> To use your own config, pass `config_path=Path("my_config.yaml")` instead of `config="chatapp"`.  
> To add more bundles, add new keys under `metric_bundles:` in your YAML.

**Step 2 — Run the decorator**:

```bash
python examples_test/test_decorator_sample.py
```

Or paste this into a Python file:

```python
# If you installed via `pip install ignis_evaluation`, the sys.path lines are not needed.
# Only uncomment them if running from the cloned repo without pip install -e .
# import sys; from pathlib import Path
# sys.path.insert(0, str(Path(".") / "src"))

from dotenv import load_dotenv
load_dotenv(".env")  # loads OPENAI_API_KEY

# Pre-import openai before ThreadPoolExecutor starts — prevents Python 3.12
# import-lock deadlock when parallel metric threads first-import openai.
import openai  # noqa: F401
import openai.resources.chat  # noqa: F401

from ignis_evaluation.eval_library.decorator import ignis_eval

# config="chatapp"  → loads src/ignis_evaluation/configs/chatapp_config.yaml
# bundle="core_quality" → runs the 3 metrics defined in that bundle
@ignis_eval(config="chatapp", bundle="core_quality", include_results=True)
def evaluate_response(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

result = evaluate_response(
    input_text="How do I reset my password?",
    output_text="Click 'Forgot Password' on the login screen and follow the reset link sent to your email.",
)

eval_data = result["evaluation"]
all_passed = all(m.get("pass", False) for m in eval_data.get("metrics", {}).values() if isinstance(m, dict))
print("Score :", eval_data.get("weighted_score"))
print("Passed:", all_passed)
for name, m in eval_data.get("metrics", {}).items():
    status = "PASS" if m.get("pass") else "FAIL"
    print(f"  {name}: {m.get('score')} — {status}")
```

**Expected output:**
```
Score : 0.97
Passed: True
  Clarity: 0.96 — PASS
  Conciseness: 0.93 — PASS
  AnswerRelevancyMetric: 1.0 — PASS
```

---

### Run all tests

```bash
python -m pytest examples_test/ -v
```

---

## Configuration Reference

All evaluation behaviour is controlled by a YAML config file.

```yaml
project:        chatapp           # Groups runs in DB and dashboard
model:          gpt-4o-mini       # LLM for scoring
use_deepeval:   true              # false = offline/mock mode (no API key needed)

hyperparameters:
  model:        gpt-4o-mini
  temperature:  0.3
  max_tokens:   1024

composite_threshold: 0.65         # Weighted score must exceed this to pass

default_bundle: core_quality      # Used when no bundle is specified in API call

metric_bundles:
  core_quality:
    - name: AnswerRelevancyMetric
      threshold: 0.70
      weight:    40
    - name: GEvalMetric
      geval_name: "Clarity"
      criteria: "The response must be written in clear, plain language."
      evaluation_params: [input, actual_output]
      threshold: 0.65
      weight:    35
    - name: GEvalMetric
      geval_name: "Conciseness"
      criteria: "The response must answer the question without unnecessary content."
      evaluation_params: [input, actual_output]
      threshold: 0.60
      weight:    25

reporting:
  format:     json
  output_dir: reports
```

---

## Metric Bundles

Bundles group metrics under a named key. Pass `bundle` in any API call to select which group runs.

### chatapp_config — Chat application (input + output only, no context needed)

| Bundle | Metrics | Use Case |
|---|---|---|
| `core_quality` ★ | AnswerRelevancy + Clarity + Conciseness | Default — every message |
| `communication` | Tone + Clarity + Conciseness | UX reviews, persona validation |
| `safety_guard` | Toxicity + Bias + ProfessionalBoundaries | Public-facing deployments |
| `helpfulness` | AnswerRelevancy + Completeness + Actionability | Support bots |
| `customer_support` | Helpfulness + Empathy + Specificity | Customer service chat |
| `production_monitor` | AnswerRelevancy + Toxicity | Lightest — 100% live traffic |
| `content_policy` | Toxicity + Bias + Boundaries + FactualHumility | Pre-release compliance |
| `full_audit` | All 8 metrics | Weekly audit, model upgrades |
| `response_quality` | AnswerRelevancy + ReasoningClarity + Completeness + FactualHumility | A/B testing |

### bundle_config — General purpose (requires context for some metrics)

| Bundle | Metrics |
|---|---|
| `quality` | FaithfulnessMetric + AnswerRelevancyMetric |
| `safety` | ToxicityMetric + HallucinationMetric |

### compliance_bundle_config — Finance domain

| Bundle | Metrics |
|---|---|
| `compliance` | FinanceComplianceMetric + AnswerRelevancyMetric |
| `compliance_strict` | FinanceComplianceMetric + AnswerRelevancyMetric + HallucinationMetric |
| `compliance_balanced` | FinanceComplianceMetric + AnswerRelevancyMetric + FaithfulnessMetric |

### Which metrics need `context`?

| Metric | Needs `context`? |
|---|---|
| `FaithfulnessMetric` | ✅ Yes |
| `HallucinationMetric` | ✅ Yes |
| `ContextualPrecision/Recall/Relevancy` | ✅ Yes |
| `AnswerRelevancyMetric` | ❌ No |
| `ToxicityMetric` / `BiasMetric` | ❌ No |
| `GEvalMetric` (custom criteria) | Depends on `evaluation_params` in config |

---

## REST API

Start the server: `uvicorn src.ignis_evaluation.api.main:app --reload --port 8001`

### Evaluation endpoints

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/evaluate` | Single test case |
| `POST` | `/evaluate/batch` | Multiple test cases |
| `POST` | `/evaluate/dataset` | Entire dataset JSON file |

**POST /evaluate — with bundle and test_name**

```json
{
  "config": "chatapp",
  "bundle": "core_quality",
  "input": "How do I reset my password?",
  "output": "Click Forgot Password on the login screen and follow the reset link.",
  "test_name": "obs_trace_abc123",
  "upload_to_confident": false
}
```

**POST /evaluate/batch — multiple cases**

```json
{
  "config": "chatapp",
  "bundle": "customer_support",
  "upload_to_confident": false,
  "cases": [
    {
      "input": "I'm frustrated, my account is locked.",
      "output": "I understand that's frustrating. Let me help — go to Settings > Security > Unlock Account.",
      "test_name": "trace-001"
    },
    {
      "input": "What are your business hours?",
      "output": "We're open Monday to Friday, 9 AM to 6 PM IST.",
      "test_name": "trace-002"
    }
  ]
}
```

> **`test_name`** — pass your Observability `trace_id` here to link eval scores back to production traces.  
> **`upload_to_confident`** — set `true` to push results to Confident AI (requires `CONFIDENT_API_KEY`). Default: `false`.

### Dashboard & analytics endpoints

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/eval/dashboard` | Full dashboard — KPIs, charts, domain → project → bundles |
| `GET` | `/eval/projects/{name}/bundles` | Bundle definitions from YAML config for a project |
| `GET` | `/eval/projects/{project}/trend` | Score trend over time |
| `GET` | `/eval/metrics/leaderboard` | Per-metric rankings |
| `GET` | `/results` | List past runs (paginated) |
| `GET` | `/results/{run_id}` | Full detail for one run |
| `GET` | `/health` | Liveness + readiness |

---

## Evaluation Dashboard

The dashboard provides a single endpoint that returns everything needed to build a UI: KPIs, 6 chart datasets, and a hierarchical breakdown by domain → project → bundles → metrics.

### POST /eval/dashboard

```json
POST /eval/dashboard
{
  "project":              "chatapp",
  "window_hours":         168,
  "include_evaluations":  true,
  "eval_limit":           20
}
```

All fields are optional. Default: last 2 hours, all projects, no evaluation list.

**Response structure:**

```json
{
  "generated_at": "2026-07-30T10:00:00Z",
  "window_hours": 168,
  "kpis": {
    "evaluation_count":     87,
    "avg_weighted_score":   0.89,
    "pass_rate_pct":        85.0,
    "total_cost_usd":       0.021,
    "error_count":          4,
    "avg_eval_duration_ms": 12500,
    "models_used":          ["gpt-4o-mini"]
  },
  "charts": {
    "score_over_time":  [{ "date": "2026-07-29", "avg_score": 0.88, "pass_rate_pct": 84 }],
    "pass_fail":        { "passed": 74, "failed": 13 },
    "cost_by_metric":   [{ "metric_name": "AnswerRelevancyMetric", "cost_usd": 0.012 }],
    "score_by_metric":  [{ "metric_name": "Clarity", "avg_score": 0.79 }],
    "score_by_model":   [{ "model": "gpt-4o-mini", "avg_score": 0.89, "run_count": 87 }],
    "cost_trend":       [{ "date": "2026-07-29", "total_cost_usd": 0.021 }]
  },
  "domains": [
    {
      "domain": "Finance",
      "projects": [
        {
          "project_name": "chatapp",
          "avg_weighted_score": 0.89,
          "bundles": [
            {
              "bundle_name": "core_quality",
              "is_default":  true,
              "metrics": [
                { "metric_name": "AnswerRelevancyMetric", "avg_score": 1.0,  "pass_rate_pct": 100, "status": "healthy" },
                { "metric_name": "Clarity",               "avg_score": 0.79, "pass_rate_pct": 60,  "status": "warning" },
                { "metric_name": "Conciseness",           "avg_score": null, "pass_rate_pct": null,"status": "not_evaluated" }
              ]
            }
          ],
          "recent_evaluations": [
            {
              "run_id": 100, "test_name": "obs_trace_abc123",
              "metric_name": "Clarity", "score": 0.49, "passed": false,
              "reason": "Response uses jargon that reduces clarity."
            }
          ]
        }
      ]
    }
  ]
}
```

**Metric status values:**

| Status | Meaning |
|---|---|
| `healthy` | pass_rate_pct ≥ 80% |
| `warning` | 50% ≤ pass_rate_pct < 80% |
| `needs_attention` | pass_rate_pct < 50% |
| `not_evaluated` | No runs yet for this metric |

### GET /eval/projects/{project_name}/bundles

Returns all bundles defined in the YAML config for a project. Used by the frontend to render bundle tabs before overlaying scores from the dashboard.

```
GET /eval/projects/chatapp/bundles
```

```json
{
  "project_name": "chatapp",
  "default_bundle": "core_quality",
  "bundles": [
    {
      "bundle_name": "core_quality",
      "is_default": true,
      "metrics": [
        { "metric_name": "AnswerRelevancyMetric", "threshold": 0.70, "weight": 40 },
        { "metric_name": "Clarity",               "threshold": 0.65, "weight": 35 }
      ]
    }
  ]
}
```

### Using dashboard in your own app

```python
from ignis_evaluation.services.dashboard_service import build_enriched_dashboard

result = build_enriched_dashboard(
    project="chatapp",
    window_hours=168,
    include_evaluations=True,
    eval_limit=20,
)
# Returns full dict with kpis, charts, domains — same as POST /eval/dashboard
```

---

## Decorator-Driven Evaluation (`@ignis_eval`)

Wrap any LLM function in your application to evaluate its output inline — no separate API call needed.

```python
from ignis_evaluation.eval_library.decorator import ignis_eval

@ignis_eval(config="chatapp", bundle="core_quality", include_results=True)
def evaluate_chat(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

# In your endpoint:
answer = your_llm_call(user_query)
result = evaluate_chat(input_text=user_query, output_text=answer)
# result["evaluation"]["weighted_score"] → 0.91
# result["evaluation"]["metrics"]["Clarity"]["score"] → 0.84
```

**Save a report file per call** — add `save_report=True` to write a `decorator_{project}_{timestamp}.json` + `.csv` to `EVAL_REPORTS_DIR` after every evaluation:

```python
@ignis_eval(config="chatapp", bundle="core_quality", include_results=True, save_report=True)
def evaluate_chat(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}
```

**What happens automatically:**
1. Loads the bundle config from YAML
2. Runs all metrics in parallel
3. Computes weighted composite score
4. Persists result to PostgreSQL (if `EVAL_DB_URL` is set)
5. Saves JSON + CSV report file (if `save_report=True`)
6. Returns answer + evaluation payload

---

## Observability Integration

Automatically evaluate every production LLM response stored in an Observability DB without writing a single line of pipeline code.

### How it works

```
Observability DB (read-only)            Evaluation DB
  traces.input + output + trace_id  →   eval_worker.py   →   metric_results
  traces.ingested_at (checkpoint)   ←   eval_worker_checkpoints
```

The `eval_worker` reads new traces from the Observability DB since the last checkpoint, evaluates them via `POST /evaluate/batch`, and stores scores in the Evaluation DB. The `trace_id` is stored as `test_case_name` — the only link between both databases.

### Setup

Add to `.env`:

```
# DeepEval DB — stores eval scores and checkpoints
EVAL_DB_URL=postgresql://postgres:pass@localhost:5432/deepeval_results

# Observability DB — read-only, traces source
OBS_DB_URL=postgresql://postgres:pass@localhost:5432/obs_db

# Eval worker settings
EVAL_API_URL=http://localhost:8001
EVAL_WORKER_BATCH_SIZE=100
EVAL_WORKER_LOOKBACK_DAYS=7
EVAL_WORKER_APP_NAME=ignis
```

Apply the checkpoint table to the Evaluation DB:

```bash
psql -U postgres -d deepeval_results -c "
CREATE TABLE IF NOT EXISTS public.eval_worker_checkpoints (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    worker_source    VARCHAR(50) NOT NULL DEFAULT 'eval_worker',
    project_name     VARCHAR(255) NOT NULL,
    start_time       TIMESTAMPTZ NOT NULL,
    end_time         TIMESTAMPTZ,
    status           VARCHAR(20) NOT NULL DEFAULT 'running',
    traces_pulled    INTEGER NOT NULL DEFAULT 0,
    last_eval_at     TIMESTAMPTZ,
    config_used      VARCHAR(255),
    bundle_used      VARCHAR(255),
    error            TEXT,
    application_name VARCHAR(255) NOT NULL DEFAULT 'ignis',
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);"
```

### Running the worker

```bash
# All projects — uses last checkpoint per project
python -m ignis_evaluation.workers.eval_worker

# Specific obs project
python -m ignis_evaluation.workers.eval_worker --project langwatch_results

# Override bundle for all projects this run
python -m ignis_evaluation.workers.eval_worker --bundle safety_guard

# Re-process from a specific date (ignores checkpoint)
python -m ignis_evaluation.workers.eval_worker --since 2026-07-01T00:00:00

# Dry run — logs what would happen without calling the API
python -m ignis_evaluation.workers.eval_worker --dry-run
```

### Project → config mapping

The worker maps obs `project_name` → eval config + bundle. Configure via `EVAL_PROJECT_CONFIG_MAP` env var:

```
EVAL_PROJECT_CONFIG_MAP={"langwatch_results": {"config": "chatapp", "bundle": "core_quality"}, "testing_traces": {"config": "chatapp", "bundle": "production_monitor"}}
```

Unknown projects fall back to `chatapp` config + `core_quality` bundle.

### Checkpoint behaviour

- Each run saves `last_eval_at = traces[-1].ingested_at` in `eval_worker_checkpoints`
- Next run fetches only traces with `ingested_at > last_eval_at`
- Failed runs do NOT update the checkpoint — they retry from the same point next run
- `status = 'success'` rows only are used as checkpoints — stale `running` rows are safely ignored

### Linking scores back to traces

```sql
-- In Evaluation DB: find scores for a specific production trace
SELECT mr.metric_name, mr.score, mr.passed, mr.reason
FROM metric_results mr
JOIN test_results tr ON tr.test_result_id = mr.test_result_id
WHERE tr.test_case_name = 'your-obs-trace-uuid';
```

---

## Database Setup

### Schema (run once)

```bash
psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql
```

### Tables

| Table | Description |
|---|---|
| `evaluation_runs` | One row per eval run — hyperparameters, config, bundle, weighted_score, cost |
| `test_results` | One row per test case — input, output, test_case_name (= obs trace_id) |
| `metric_results` | One row per metric per test case — score, threshold, passed, reason |
| `projects` | One row per project — name, domain, created_at |
| `eval_worker_checkpoints` | One row per worker run per obs project — checkpoint, status, traces_pulled |

### Without a database

The framework works fully without PostgreSQL — results are written to JSON/CSV in `examples_test/reports/` instead.

---

## Environment Variables Reference

| Variable | Required | Default | Description |
|---|---|---|---|
| `OPENAI_API_KEY` | Yes (real mode) | — | OpenAI API key for DeepEval scoring |
| `CONFIDENT_API_KEY` | No | — | Confident AI key for cloud dashboard upload |
| `DEEPEVAL_TELEMETRY_OPT_OUT` | No | — | Set `YES` to disable anonymous telemetry |
| `EVAL_DB_URL` | No | — | DeepEval DB connection string (e.g. `postgresql://user:pass@host:5432/deepeval_results`) |
| `OBS_DB_URL` | No | — | Observability DB DSN (eval worker only) |
| `EVAL_API_URL` | No | `http://localhost:8001` | Evaluation API base URL (eval worker) |
| `EVAL_WORKER_BATCH_SIZE` | No | `100` | Max traces per worker batch call |
| `EVAL_WORKER_LOOKBACK_DAYS` | No | `7` | Default lookback when no checkpoint exists |
| `EVAL_PROJECT_CONFIG_MAP` | No | — | JSON: obs project → {config, bundle} |
| `EVAL_WORKER_APP_NAME` | No | `ignis` | application_name in checkpoint table |
| `PROJECT_DOMAIN` | No | — | Default domain for new projects |

---

## Troubleshooting

**`ModuleNotFoundError: ignis_evaluation`**  
Run `pip install -e .` from the repo root.

**`OSError` or path errors on Windows**  
Add `DEEPEVAL_TELEMETRY_OPT_OUT=YES` to `.env`.

**Evaluation takes 5+ minutes**  
Remove `ContextualRelevancyMetric` (makes multiple LLM calls per chunk). Use `use_deepeval: false` for offline runs.

**Eval worker shows "0 traces" on every run**  
Check that `--project` uses the obs DB `project_name`, not the YAML config `project:` field. Run without `--project` to process all discovered projects automatically.

**Dashboard shows `not_evaluated` for all bundle metrics**  
The bundle config was loaded from YAML but no evaluations have run for those specific metrics yet. Run `/evaluate/batch` with the bundle name to populate scores.

**`CONFIDENT_API_KEY` returns "Invalid API key" on upload**  
Run `deepeval login` in the terminal to authenticate the session. The upload flag `upload_to_confident` defaults to `false` — the framework works fully without it.

**Database not connecting**  
1. Confirm the DB exists: `psql -U postgres -l`
2. Confirm schema was applied: `psql -U postgres -d deepeval_results -c "\dt"`
3. Check `EVAL_DB_URL` is set correctly in `.env`.

---


## References

Built with:

- [DeepEval](https://github.com/confident-ai/deepeval) — LLM evaluation framework powering all metrics
- [Confident AI](https://confident-ai.com/) — Cloud dashboard for evaluation results and dataset management
- [OpenAI](https://openai.com/) — LLM judge provider (`gpt-4o-mini`, `gpt-4o`)
- [FastAPI](https://fastapi.tiangolo.com/) — REST API framework with auto Swagger docs
- [Uvicorn](https://www.uvicorn.org/) — ASGI server for the evaluation API
- [PostgreSQL](https://www.postgresql.org/) — Evaluation results persistence
- [psycopg2](https://www.psycopg.org/) — PostgreSQL driver
- [Pydantic](https://docs.pydantic.dev/) — Request/response validation and settings
- [PyYAML](https://pyyaml.org/) — YAML config file parsing
- [python-dotenv](https://github.com/theskumar/python-dotenv) — `.env` file loading
- [Rich](https://rich.readthedocs.io/) — Terminal metric result tables

---

**Questions?** Open an issue or check the [examples_test/](examples_test/) directory for copy-paste examples.

**Ready to evaluate your LLM?** Start with `examples_test/test_api.py` — no API key needed.

