Metadata-Version: 2.5
Name: llama-index-evaluation-nace
Version: 0.1.0
Summary: LlamaIndex evaluators backed by Drex decision models from Nace
Project-URL: Homepage, https://console.nace.ai/docs
Project-URL: Repository, https://github.com/nace-ai/llama-index-nace
Project-URL: Issues, https://github.com/nace-ai/llama-index-nace/issues
Author-email: Nace AI <hello@nace.ai>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <4.0,>=3.11
Requires-Dist: httpx<1,>=0.27
Requires-Dist: llama-index-core<0.15,>=0.14.0
Requires-Dist: nace-sdk<1.0.0,>=0.3.0
Requires-Dist: pydantic<3,>=2.11
Description-Content-Type: text/markdown

# llama-index-evaluation-nace

Replace your LLM judge. LlamaIndex evaluators backed by Drex decision models from [Nace](https://nace.ai): Drex returns probabilities, not text that has to be parsed.

## Install

Coming soon to PyPI (`pip install llama-index-evaluation-nace`). Install from source meanwhile:

```
pip install "git+https://github.com/nace-ai/llama-index-nace#subdirectory=packages/llama-index-evaluation-nace"
```

Set `NACE_API_KEY`. Optional: `NACE_BASE_URL`, `NACE_DEFAULT_DECISION_MODEL`. For OpenRouter pass `base_url="https://openrouter.ai/api/v1"`.

The default model is `drex-max-v1.1`. An explicit `model=` is sent exactly as given; without one, `NACE_DEFAULT_DECISION_MODEL` is used when set, else `drex-max-v1.1`. If that default is a text-only model (`drex-v1.5`, `drex-v2.0`) and the request has images, `drex-max-v1.1` is used for that request. OpenRouter doesn't list v1.1 yet, so there the default is the original Drex Max, `nace-ai/drex-max`.

## Usage

```python
from llama_index.evaluation.nace import NaceFaithfulnessEvaluator

result = NaceFaithfulnessEvaluator().evaluate(
    query="When was the Eiffel Tower completed?",
    response="It was completed in 1889.",
    contexts=["The Eiffel Tower was completed in March 1889."],
)
print(result.passing, result.score, result.feedback)  # True 0.95 P(faithful)=0.95
```

`evaluate()` is sync and works in Jupyter; `aevaluate()` is async. `evaluate_response()` reads source nodes, so the evaluators work with `BatchEvalRunner`.

| Evaluator | Question | score / passing |
|---|---|---|
| `NaceFaithfulnessEvaluator` | Is every claim supported by the context? (strict) | P(yes); `passing = P >= threshold` (0.5) |
| `NaceRelevancyEvaluator` | Does the response answer the query, in line with the context? | P(yes); same threshold |
| `NaceCorrectnessEvaluator` | 5-level score, against `reference=` when given | 1 to 5; `passing = score >= score_threshold` (4.0) |

`raise_error=True` (faithfulness, relevancy) raises `ValueError` on a failing result, as LlamaIndex's evaluators do. The question text is editable through `update_prompts({"instructions": ...})`.

Results are `NaceEvaluationResult`: `probability`, `probabilities` (score levels), `model`, `request_id`, `usage`.

## Images

Evaluators read `ImageNode` sources (`image_path`, base64 `image`, or a `data:` URL) and send them to Drex. The default model, `drex-max-v1.1`, reads images; when `NACE_DEFAULT_DECISION_MODEL` names a text-only model, requests with images still go to `drex-max-v1.1`, and an explicit text-only `model=` with images raises `NaceImageModelError`. `http(s)` image URLs are refused, because Drex never fetches URLs. `max_images=N` keeps the N highest-scored image nodes, whatever their count; without it, more than 8 images (Drex's per-request limit) raise `NaceImageLimitError` rather than being trimmed silently. Strict faithfulness on image-only sources tends to score lower than on text; compare probabilities rather than trusting the default cut-off.

See [`examples/`](../../examples) for runnable scripts and notebooks.
