Metadata-Version: 2.4
Name: ignis_evaluation
Version: 1.0.1
Summary: Config-driven LLM evaluation framework with decorator-based tracing, FastAPI service, and PostgreSQL persistence
Author: Infogain-GenAI
License: MIT
Project-URL: Homepage, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval
Project-URL: Repository, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval.git
Project-URL: Documentation, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval#readme
Project-URL: Bug Tracker, https://github.com/Infogain-GenAI/ignis_evaluation_deepeval/issues
Keywords: deepeval,llm,evaluation,metrics,faithfulness,hallucination,rag,openai,fastapi,config-driven
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: deepeval==4.0.5
Requires-Dist: openai>=1.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: psycopg2-binary>=2.9.0
Provides-Extra: api
Requires-Dist: fastapi>=0.111.0; extra == "api"
Requires-Dist: uvicorn[standard]>=0.29.0; extra == "api"
Requires-Dist: httpx>=0.27.0; extra == "api"
Requires-Dist: pydantic>=2.0.0; extra == "api"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"

# ignis_evaluation

A **config-driven LLM evaluation framework** built on top of [DeepEval](https://github.com/confident-ai/deepeval).  
Define metrics in a YAML bundle config, run evaluations via REST API or decorator, persist scores to PostgreSQL, and view them on a hierarchical evaluation dashboard.

---

## Table of Contents

- [Features](#features)
- [Project Structure](#project-structure)
- [Quick Start](#quick-start)
- [Configuration Reference](#configuration-reference)
- [Metric Bundles](#metric-bundles)
- [REST API](#rest-api)
- [Evaluation Dashboard](#evaluation-dashboard)
- [Decorator-Driven Evaluation](#decorator-driven-evaluation-ignis_eval)
- [Observability Integration](#observability-integration)
- [Database Setup](#database-setup)
- [Running Tests & Examples](#running-tests--examples)
- [Environment Variables Reference](#environment-variables-reference)
- [Troubleshooting](#troubleshooting)

---

## Features

- **YAML bundle configs** — group metrics into named bundles; switch bundles per API call with no code change
- **Hierarchical dashboard** — domain → project → bundles → metrics with KPIs, 6 chart types, and status badges
- **Observability integration** — eval worker reads production traces from an Observability DB and evaluates them automatically
- **Parallel metric execution** — all metrics run concurrently via `ThreadPoolExecutor`
- **PostgreSQL persistence** — every run, test case, and metric score stored automatically
- **`@ignis_eval` decorator** — wrap any LLM function in your app to evaluate output inline
- **REST API** — trigger evaluations and retrieve results over HTTP
- **Confident AI upload** — optional push to Confident AI cloud dashboard
- **Offline mode** — run without an OpenAI key using `use_deepeval: false`

---

## Project Structure

```
deepeval_demo/
├── src/
│   └── ignis_evaluation/
│       ├── api/
│       │   ├── main.py             # FastAPI entry point, router registration
│       │   ├── models.py           # Pydantic request/response models
│       │   └── routers/
│       │       ├── evaluate.py     # POST /evaluate, /evaluate/batch, /evaluate/dataset
│       │       ├── analytics.py    # POST /eval/dashboard + other analytics endpoints
│       │       ├── projects.py     # GET /projects, GET /projects/{name}/bundles
│       │       ├── results.py      # GET /results, GET /results/{run_id}
│       │       ├── metrics.py      # GET /metrics
│       │       ├── health.py       # GET /health
│       │       └── chat.py         # POST /chat (decorator-driven)
│       ├── configs/                # Built-in YAML configs shipped with the package
│       │   ├── chatapp_config.yaml         # Chat app: 9 bundles, input+output only
│       │   ├── geval_bundle_config.yaml    # GEval custom criteria bundles
│       │   ├── bundle_config.yaml          # General purpose quality+safety bundles
│       │   ├── compliance_bundle_config.yaml  # Finance compliance bundles
│       │   ├── rag_config.yaml             # RAG pipeline metrics
│       │   ├── healthcare_config.yaml      # Healthcare domain metrics
│       │   └── ...
│       ├── db/
│       │   ├── connection.py       # PostgreSQL connection
│       │   ├── writer.py           # INSERT evaluation runs, test & metric results
│       │   ├── reader.py           # SELECT runs for Results API
│       │   └── analytics.py        # Dashboard queries (KPIs, charts, bundles)
│       ├── eval_library/
│       │   ├── runner.py           # Core evaluation engine (EvaluationRunner)
│       │   ├── decorator.py        # @ignis_eval decorator
│       │   ├── scorer.py           # Weighted composite scoring
│       │   ├── metric_registry.py  # Maps metric names → DeepEval classes
│       │   └── ...
│       ├── services/
│       │   └── dashboard_service.py  # build_enriched_dashboard() — usable as a library
│       ├── workers/
│       │   └── eval_worker.py      # Observability → Eval pipeline worker
│       └── sql/
│           └── schema.sql          # Database schema (run once)
├── examples_test/
│   ├── configs/                    # Example YAML configs
│   ├── datasets/                   # Versioned evaluation datasets
│   └── ...
├── docs/
│   ├── EVALUATION_ON_OBSERVABILITY.md
│   ├── OBSERVABILITY_EVAL_APPROACH.md
│   ├── DASHBOARD_UI_WIREFRAME.md
│   └── OBSERVABILITY_EVAL_INTEGRATION.md
├── .env.example
├── pyproject.toml
└── README.md
```

---

## Quick Start

### 1. Clone & create virtual environment

```bash
git clone https://github.com/Infogain-GenAI/ignis_evaluation_deepeval.git
cd deepeval_demo

python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate
```

### 2. Install dependencies

```bash
pip install -e .          # base install
pip install -e ".[api]"   # adds FastAPI server
pip install -e ".[dev]"   # adds test/dev tools
```

### 3. Configure environment

```bash
copy .env.example .env    # Windows
cp .env.example .env      # macOS/Linux
```

Minimum required variables in `.env`:

```
OPENAI_API_KEY=sk-...

```

### 4. Set up the database

```bash
# Create the database first:
# CREATE DATABASE deepeval_results;

# Apply schema:
psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql
```

### 5. Start the API server

```bash
uvicorn src.ignis_evaluation.api.main:app --reload --port 8001
```

Interactive docs: [http://localhost:8001/docs](http://localhost:8001/docs)

### 6. Run your first evaluation

```bash
# Via API (Swagger or curl)
curl -X POST http://localhost:8001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "config": "chatapp",
    "bundle": "core_quality",
    "input": "What is diabetes?",
    "output": "Diabetes is a chronic condition that affects how the body regulates blood sugar."
  }'
```

---

## Configuration Reference

All evaluation behaviour is controlled by a YAML config file.

```yaml
project:        chatapp           # Groups runs in DB and dashboard
model:          gpt-4o-mini       # LLM for scoring
use_deepeval:   true              # false = offline/mock mode (no API key needed)

hyperparameters:
  model:        gpt-4o-mini
  temperature:  0.3
  max_tokens:   1024

composite_threshold: 0.65         # Weighted score must exceed this to pass

default_bundle: core_quality      # Used when no bundle is specified in API call

metric_bundles:
  core_quality:
    - name: AnswerRelevancyMetric
      threshold: 0.70
      weight:    40
    - name: GEvalMetric
      geval_name: "Clarity"
      criteria: "The response must be written in clear, plain language."
      evaluation_params: [input, actual_output]
      threshold: 0.65
      weight:    35
    - name: GEvalMetric
      geval_name: "Conciseness"
      criteria: "The response must answer the question without unnecessary content."
      evaluation_params: [input, actual_output]
      threshold: 0.60
      weight:    25

reporting:
  format:     json
  output_dir: reports
```

---

## Metric Bundles

Bundles group metrics under a named key. Pass `bundle` in any API call to select which group runs.

### chatapp_config — Chat application (input + output only, no context needed)

| Bundle | Metrics | Use Case |
|---|---|---|
| `core_quality` ★ | AnswerRelevancy + Clarity + Conciseness | Default — every message |
| `communication` | Tone + Clarity + Conciseness | UX reviews, persona validation |
| `safety_guard` | Toxicity + Bias + ProfessionalBoundaries | Public-facing deployments |
| `helpfulness` | AnswerRelevancy + Completeness + Actionability | Support bots |
| `customer_support` | Helpfulness + Empathy + Specificity | Customer service chat |
| `production_monitor` | AnswerRelevancy + Toxicity | Lightest — 100% live traffic |
| `content_policy` | Toxicity + Bias + Boundaries + FactualHumility | Pre-release compliance |
| `full_audit` | All 8 metrics | Weekly audit, model upgrades |
| `response_quality` | AnswerRelevancy + ReasoningClarity + Completeness + FactualHumility | A/B testing |

### bundle_config — General purpose (requires context for some metrics)

| Bundle | Metrics |
|---|---|
| `quality` | FaithfulnessMetric + AnswerRelevancyMetric |
| `safety` | ToxicityMetric + HallucinationMetric |

### compliance_bundle_config — Finance domain

| Bundle | Metrics |
|---|---|
| `compliance` | FinanceComplianceMetric + AnswerRelevancyMetric |
| `compliance_strict` | FinanceComplianceMetric + AnswerRelevancyMetric + HallucinationMetric |
| `compliance_balanced` | FinanceComplianceMetric + AnswerRelevancyMetric + FaithfulnessMetric |

### Which metrics need `context`?

| Metric | Needs `context`? |
|---|---|
| `FaithfulnessMetric` | ✅ Yes |
| `HallucinationMetric` | ✅ Yes |
| `ContextualPrecision/Recall/Relevancy` | ✅ Yes |
| `AnswerRelevancyMetric` | ❌ No |
| `ToxicityMetric` / `BiasMetric` | ❌ No |
| `GEvalMetric` (custom criteria) | Depends on `evaluation_params` in config |

---

## REST API

Start the server: `uvicorn src.ignis_evaluation.api.main:app --reload --port 8001`

### Evaluation endpoints

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/evaluate` | Single test case |
| `POST` | `/evaluate/batch` | Multiple test cases |
| `POST` | `/evaluate/dataset` | Entire dataset JSON file |

**POST /evaluate — with bundle and test_name**

```json
{
  "config": "chatapp",
  "bundle": "core_quality",
  "input": "How do I reset my password?",
  "output": "Click Forgot Password on the login screen and follow the reset link.",
  "test_name": "obs_trace_abc123",
  "upload_to_confident": false
}
```

**POST /evaluate/batch — multiple cases**

```json
{
  "config": "chatapp",
  "bundle": "customer_support",
  "upload_to_confident": false,
  "cases": [
    {
      "input": "I'm frustrated, my account is locked.",
      "output": "I understand that's frustrating. Let me help — go to Settings > Security > Unlock Account.",
      "test_name": "trace-001"
    },
    {
      "input": "What are your business hours?",
      "output": "We're open Monday to Friday, 9 AM to 6 PM IST.",
      "test_name": "trace-002"
    }
  ]
}
```

> **`test_name`** — pass your Observability `trace_id` here to link eval scores back to production traces.  
> **`upload_to_confident`** — set `true` to push results to Confident AI (requires `CONFIDENT_API_KEY`). Default: `false`.

### Dashboard & analytics endpoints

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/eval/dashboard` | Full dashboard — KPIs, charts, domain → project → bundles |
| `GET` | `/eval/projects/{name}/bundles` | Bundle definitions from YAML config for a project |
| `GET` | `/eval/projects/{project}/trend` | Score trend over time |
| `GET` | `/eval/metrics/leaderboard` | Per-metric rankings |
| `GET` | `/results` | List past runs (paginated) |
| `GET` | `/results/{run_id}` | Full detail for one run |
| `GET` | `/health` | Liveness + readiness |

---

## Evaluation Dashboard

The dashboard provides a single endpoint that returns everything needed to build a UI: KPIs, 6 chart datasets, and a hierarchical breakdown by domain → project → bundles → metrics.

### POST /eval/dashboard

```json
POST /eval/dashboard
{
  "project":              "chatapp",
  "window_hours":         168,
  "include_evaluations":  true,
  "eval_limit":           20
}
```

All fields are optional. Default: last 2 hours, all projects, no evaluation list.

**Response structure:**

```json
{
  "generated_at": "2026-07-30T10:00:00Z",
  "window_hours": 168,
  "kpis": {
    "evaluation_count":     87,
    "avg_weighted_score":   0.89,
    "pass_rate_pct":        85.0,
    "total_cost_usd":       0.021,
    "error_count":          4,
    "avg_eval_duration_ms": 12500,
    "models_used":          ["gpt-4o-mini"]
  },
  "charts": {
    "score_over_time":  [{ "date": "2026-07-29", "avg_score": 0.88, "pass_rate_pct": 84 }],
    "pass_fail":        { "passed": 74, "failed": 13 },
    "cost_by_metric":   [{ "metric_name": "AnswerRelevancyMetric", "cost_usd": 0.012 }],
    "score_by_metric":  [{ "metric_name": "Clarity", "avg_score": 0.79 }],
    "score_by_model":   [{ "model": "gpt-4o-mini", "avg_score": 0.89, "run_count": 87 }],
    "cost_trend":       [{ "date": "2026-07-29", "total_cost_usd": 0.021 }]
  },
  "domains": [
    {
      "domain": "Finance",
      "projects": [
        {
          "project_name": "chatapp",
          "avg_weighted_score": 0.89,
          "bundles": [
            {
              "bundle_name": "core_quality",
              "is_default":  true,
              "metrics": [
                { "metric_name": "AnswerRelevancyMetric", "avg_score": 1.0,  "pass_rate_pct": 100, "status": "healthy" },
                { "metric_name": "Clarity",               "avg_score": 0.79, "pass_rate_pct": 60,  "status": "warning" },
                { "metric_name": "Conciseness",           "avg_score": null, "pass_rate_pct": null,"status": "not_evaluated" }
              ]
            }
          ],
          "recent_evaluations": [
            {
              "run_id": 100, "test_name": "obs_trace_abc123",
              "metric_name": "Clarity", "score": 0.49, "passed": false,
              "reason": "Response uses jargon that reduces clarity."
            }
          ]
        }
      ]
    }
  ]
}
```

**Metric status values:**

| Status | Meaning |
|---|---|
| `healthy` | pass_rate_pct ≥ 80% |
| `warning` | 50% ≤ pass_rate_pct < 80% |
| `needs_attention` | pass_rate_pct < 50% |
| `not_evaluated` | No runs yet for this metric |

### GET /eval/projects/{project_name}/bundles

Returns all bundles defined in the YAML config for a project. Used by the frontend to render bundle tabs before overlaying scores from the dashboard.

```
GET /eval/projects/chatapp/bundles
```

```json
{
  "project_name": "chatapp",
  "default_bundle": "core_quality",
  "bundles": [
    {
      "bundle_name": "core_quality",
      "is_default": true,
      "metrics": [
        { "metric_name": "AnswerRelevancyMetric", "threshold": 0.70, "weight": 40 },
        { "metric_name": "Clarity",               "threshold": 0.65, "weight": 35 }
      ]
    }
  ]
}
```

### Using dashboard in your own app

```python
from ignis_evaluation.services.dashboard_service import build_enriched_dashboard

result = build_enriched_dashboard(
    project="chatapp",
    window_hours=168,
    include_evaluations=True,
    eval_limit=20,
)
# Returns full dict with kpis, charts, domains — same as POST /eval/dashboard
```

---

## Decorator-Driven Evaluation (`@ignis_eval`)

Wrap any LLM function in your application to evaluate its output inline — no separate API call needed.

```python
from ignis_evaluation.eval_library.decorator import ignis_eval

@ignis_eval(config="chatapp", bundle="core_quality", include_results=True)
def evaluate_chat(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

# In your endpoint:
answer = your_llm_call(user_query)
result = evaluate_chat(input_text=user_query, output_text=answer)
# result["evaluation"]["weighted_score"] → 0.91
# result["evaluation"]["metrics"]["Clarity"]["score"] → 0.84
```

**Save a report file per call** — add `save_report=True` to write a `decorator_{project}_{timestamp}.json` + `.csv` to `EVAL_REPORTS_DIR` after every evaluation:

```python
@ignis_eval(config="chatapp", bundle="core_quality", include_results=True, save_report=True)
def evaluate_chat(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}
```

**What happens automatically:**
1. Loads the bundle config from YAML
2. Runs all metrics in parallel
3. Computes weighted composite score
4. Persists result to PostgreSQL (if `EVAL_DB_URL` is set)
5. Saves JSON + CSV report file (if `save_report=True`)
6. Returns answer + evaluation payload

---

## Observability Integration

Automatically evaluate every production LLM response stored in an Observability DB without writing a single line of pipeline code.

### How it works

```
Observability DB (read-only)            Evaluation DB
  traces.input + output + trace_id  →   eval_worker.py   →   metric_results
  traces.ingested_at (checkpoint)   ←   eval_worker_checkpoints
```

The `eval_worker` reads new traces from the Observability DB since the last checkpoint, evaluates them via `POST /evaluate/batch`, and stores scores in the Evaluation DB. The `trace_id` is stored as `test_case_name` — the only link between both databases.

### Setup

Add to `.env`:

```
# DeepEval DB — stores eval scores and checkpoints
EVAL_DB_URL=postgresql://postgres:pass@localhost:5432/deepeval_results

# Observability DB — read-only, traces source
OBS_DB_URL=postgresql://postgres:pass@localhost:5432/obs_db

# Eval worker settings
EVAL_API_URL=http://localhost:8001
EVAL_WORKER_BATCH_SIZE=100
EVAL_WORKER_LOOKBACK_DAYS=7
EVAL_WORKER_APP_NAME=ignis
```

Apply the checkpoint table to the Evaluation DB:

```bash
psql -U postgres -d deepeval_results -c "
CREATE TABLE IF NOT EXISTS public.eval_worker_checkpoints (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    worker_source    VARCHAR(50) NOT NULL DEFAULT 'eval_worker',
    project_name     VARCHAR(255) NOT NULL,
    start_time       TIMESTAMPTZ NOT NULL,
    end_time         TIMESTAMPTZ,
    status           VARCHAR(20) NOT NULL DEFAULT 'running',
    traces_pulled    INTEGER NOT NULL DEFAULT 0,
    last_eval_at     TIMESTAMPTZ,
    config_used      VARCHAR(255),
    bundle_used      VARCHAR(255),
    error            TEXT,
    application_name VARCHAR(255) NOT NULL DEFAULT 'ignis',
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);"
```

### Running the worker

```bash
# All projects — uses last checkpoint per project
python -m ignis_evaluation.workers.eval_worker

# Specific obs project
python -m ignis_evaluation.workers.eval_worker --project langwatch_results

# Override bundle for all projects this run
python -m ignis_evaluation.workers.eval_worker --bundle safety_guard

# Re-process from a specific date (ignores checkpoint)
python -m ignis_evaluation.workers.eval_worker --since 2026-07-01T00:00:00

# Dry run — logs what would happen without calling the API
python -m ignis_evaluation.workers.eval_worker --dry-run
```

### Project → config mapping

The worker maps obs `project_name` → eval config + bundle. Configure via `EVAL_PROJECT_CONFIG_MAP` env var:

```
EVAL_PROJECT_CONFIG_MAP={"langwatch_results": {"config": "chatapp", "bundle": "core_quality"}, "testing_traces": {"config": "chatapp", "bundle": "production_monitor"}}
```

Unknown projects fall back to `chatapp` config + `core_quality` bundle.

### Checkpoint behaviour

- Each run saves `last_eval_at = traces[-1].ingested_at` in `eval_worker_checkpoints`
- Next run fetches only traces with `ingested_at > last_eval_at`
- Failed runs do NOT update the checkpoint — they retry from the same point next run
- `status = 'success'` rows only are used as checkpoints — stale `running` rows are safely ignored

### Linking scores back to traces

```sql
-- In Evaluation DB: find scores for a specific production trace
SELECT mr.metric_name, mr.score, mr.passed, mr.reason
FROM metric_results mr
JOIN test_results tr ON tr.test_result_id = mr.test_result_id
WHERE tr.test_case_name = 'your-obs-trace-uuid';
```

---

## Database Setup

### Schema (run once)

```bash
psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql
```

### Tables

| Table | Description |
|---|---|
| `evaluation_runs` | One row per eval run — hyperparameters, config, bundle, weighted_score, cost |
| `test_results` | One row per test case — input, output, test_case_name (= obs trace_id) |
| `metric_results` | One row per metric per test case — score, threshold, passed, reason |
| `projects` | One row per project — name, domain, created_at |
| `eval_worker_checkpoints` | One row per worker run per obs project — checkpoint, status, traces_pulled |

### Without a database

The framework works fully without PostgreSQL — results are written to JSON/CSV in `examples_test/reports/` instead.

---

## Running Tests & Examples

```bash
# Full end-to-end demo (DB write + read)
python examples_test/test_db_results_demo.py

# GEval bundle evaluation
python examples_test/test_geval_bundle.py

# RAG evaluation
python examples_test/test_rag_deepeval.py

# Decorator usage example
python examples_test/test_decorator_sample.py

# Run all tests
python -m pytest examples_test/ -v
```

---

## Environment Variables Reference

| Variable | Required | Default | Description |
|---|---|---|---|
| `OPENAI_API_KEY` | Yes (real mode) | — | OpenAI API key for DeepEval scoring |
| `CONFIDENT_API_KEY` | No | — | Confident AI key for cloud dashboard upload |
| `DEEPEVAL_TELEMETRY_OPT_OUT` | No | — | Set `YES` to disable anonymous telemetry |
| `EVAL_DB_URL` | No | — | DeepEval DB connection string (e.g. `postgresql://user:pass@host:5432/deepeval_results`) |
| `OBS_DB_URL` | No | — | Observability DB DSN (eval worker only) |
| `EVAL_DB_URL` | No | — | DeepEval DB connection string (e.g. `postgresql://user:pass@host:5432/deepeval_results`) |
| `OBS_DB_URL` | No | — | Observability DB DSN (eval worker only) |
| `EVAL_API_URL` | No | `http://localhost:8001` | Evaluation API base URL (eval worker) |
| `EVAL_WORKER_BATCH_SIZE` | No | `100` | Max traces per worker batch call |
| `EVAL_WORKER_LOOKBACK_DAYS` | No | `7` | Default lookback when no checkpoint exists |
| `EVAL_PROJECT_CONFIG_MAP` | No | — | JSON: obs project → {config, bundle} |
| `EVAL_WORKER_APP_NAME` | No | `ignis` | application_name in checkpoint table |
| `PROJECT_DOMAIN` | No | — | Default domain for new projects |

---

## Troubleshooting

**`ModuleNotFoundError: ignis_evaluation`**  
Run `pip install -e .` from the repo root.

**`OSError` or path errors on Windows**  
Add `DEEPEVAL_TELEMETRY_OPT_OUT=YES` to `.env`.

**Evaluation takes 5+ minutes**  
Remove `ContextualRelevancyMetric` (makes multiple LLM calls per chunk). Use `use_deepeval: false` for offline runs.

**Eval worker shows "0 traces" on every run**  
Check that `--project` uses the obs DB `project_name`, not the YAML config `project:` field. Run without `--project` to process all discovered projects automatically.

**Dashboard shows `not_evaluated` for all bundle metrics**  
The bundle config was loaded from YAML but no evaluations have run for those specific metrics yet. Run `/evaluate/batch` with the bundle name to populate scores.

**`CONFIDENT_API_KEY` returns "Invalid API key" on upload**  
Run `deepeval login` in the terminal to authenticate the session. The upload flag `upload_to_confident` defaults to `false` — the framework works fully without it.

**Database not connecting**  
1. Confirm the DB exists: `psql -U postgres -l`
2. Confirm schema was applied: `psql -U postgres -d deepeval_results -c "\dt"`
3. Check `EVAL_DB_URL` is set correctly in `.env`.

---

## License

MIT — see [LICENSE](LICENSE) for details.

