Metadata-Version: 2.4
Name: gllm-evals-binary
Version: 0.1.36.post1
Summary: A library for evaluating LLM-based applications.
Author: Evals Team
Requires-Python: <3.14,>=3.11
Description-Content-Type: text/markdown
Requires-Dist: aioboto3<16.0.0,>=15.2.0
Requires-Dist: aiohttp<4.0.0,>=3.14.3
Requires-Dist: datasets<6.0.0,>=5.0.1
Requires-Dist: deepmerge<3.0,>=2.0
Requires-Dist: filelock<4.0.0,>=3.20.3
Requires-Dist: filetype<2.0.0,>=1.2.0
Requires-Dist: gllm-core-binary<0.5.0,>=0.4.54
Requires-Dist: gllm-inference-binary[anthropic,google,openai,typesafe]<0.7.0,>=0.6.181
Requires-Dist: google-api-python-client<3.0.0,>=2.0.0
Requires-Dist: google-auth<3.0.0,>=2.0.0
Requires-Dist: gspread<7.0.0,>=6.0.0
Requires-Dist: json-repair<0.61.0,>=0.60.1
Requires-Dist: jsonschema<5.0.0,>=4.0.0
Requires-Dist: orjson<4.0.0,>=3.11.6
Requires-Dist: pyasn1<0.7.0,>=0.6.4
Requires-Dist: pydantic<3.0.0,>=2.12.0
Requires-Dist: python-box[all]<8.0.0,>=7.3.2; sys_platform != "win32"
Requires-Dist: python-box<8.0.0,>=7.3.2; sys_platform == "win32"
Requires-Dist: python-magic<0.5.0,>=0.4.27; sys_platform != "win32"
Requires-Dist: python-magic-bin<0.5.0,>=0.4.14; sys_platform == "win32"
Requires-Dist: pytrec-eval-terrier<0.6.0,>=0.5.7
Requires-Dist: pyyaml<7.0.0,>=6.0
Requires-Dist: sutoppu<2.0.0,>=1.2.0
Requires-Dist: urllib3<3.0.0,>=2.7.0
Requires-Dist: cryptography<51.0.0,>=50.0.0
Provides-Extra: dev
Requires-Dist: coverage==7.4.4; extra == "dev"
Requires-Dist: mypy==1.15.0; extra == "dev"
Requires-Dist: pre-commit==3.7.0; extra == "dev"
Requires-Dist: pytest<9.1.0,>=9.0.3; extra == "dev"
Requires-Dist: pytest-asyncio<2.0.0,>=1.0.0; extra == "dev"
Requires-Dist: pytest-cov<6.0.0,>=5.0.0; extra == "dev"
Requires-Dist: pytest-mock<3.15.0,>=3.14.1; extra == "dev"
Requires-Dist: ruff<0.7.0,>=0.6.7; extra == "dev"
Provides-Extra: deepeval
Requires-Dist: deepeval<4.0.0,>=3.9.7; extra == "deepeval"
Provides-Extra: ragas
Requires-Dist: marshmallow<3.27.0,>=3.26.2; extra == "ragas"
Requires-Dist: pillow<13.0.0,>=12.1.1; extra == "ragas"
Requires-Dist: ragas<0.5.0,>=0.4.0; extra == "ragas"
Provides-Extra: langchain
Requires-Dist: agentevals<0.0.9,>=0.0.8; extra == "langchain"
Requires-Dist: openevals<0.2.0,>=0.1.0; extra == "langchain"
Requires-Dist: langchain-community<0.4.0,>=0.3.0; extra == "langchain"
Requires-Dist: langchain-core>=1.3.3; extra == "langchain"
Requires-Dist: langsmith<0.9.0,>=0.8.18; extra == "langchain"

# GLLM Evaluator SDK

A comprehensive evaluation framework for Generative AI applications including LLM outputs, AI Agent responses, and RAG (Retrieval-Augmented Generation) systems.

## Overview

The GLLM Evaluator SDK provides a robust, extensible framework designed to make AI evaluation as simple and seamless as possible across the GDP Labs ecosystem. Built with integration-first philosophy, it enables teams to easily assess the quality of generated content from any AI system while seamlessly connecting with experiment tracking and observability platforms.

### Philosophy

**Easy Evaluation Everywhere**: Standardize evaluation practices across all GDP Labs AI applications with minimal setup and maximum flexibility.

**Integration-First Design**: Built to work seamlessly with your existing experiment tracking, observability, and MLOps infrastructure.

**Extensible by Design**: Add new evaluators, metrics, and integrations without breaking existing workflows.

### Key Features

- 🌐 **GDP Labs Ecosystem Ready**: Standardized evaluation framework across all internal AI applications
- 🔌 **Seamless Integration**: Easy integration with experiment tracking and observability platforms
- 🚀 **Async-First Design**: High-performance async evaluation with parallel processing
- 🔧 **Extensible Architecture**: Easy to add new evaluators and metrics for any use case
- 🤖 **LLM as a Judge**: Advanced language models for nuanced, contextual evaluation
- 📐 **Traditional Metrics**: Support for classical evaluation metrics and custom scoring functions
- 🔗 **Popular Evaluator Integration**: Integration with popular evaluators such as RAGAS, DeepEval, and LangChain
- ⚡ **Zero-Config Start**: Get started with sensible defaults, customize as needed

## Installation

### Prerequisites

Mandatory:
1. Python 3.11+ — [Install here](https://www.python.org/downloads/)
2. pip — [Install here](https://pip.pypa.io/en/stable/installation/)
3. uv — [Install here](https://docs.astral.sh/uv/getting-started/installation/)
4. gcloud CLI (for authentication) — [Install here](https://cloud.google.com/sdk/docs/install), then log in using:
   ```bash
   gcloud auth login
   ```

---


### Install from Artifact

Because `gllm-evals` is a private library hosted in a secure Google Cloud repository, you must provide an access token to install it. The command below handles this authorization inline by using an access token from the `gcloud` CLI.

```bash
uv pip install --extra-index-url https://oauth2accesstoken:$(gcloud auth print-access-token)@glsdk.gdplabs.id/gen-ai-internal/simple/ gllm-evals
```

---

## Local Development Setup

### Prerequisites

1. Python 3.11+ — [Install here](https://www.python.org/downloads/)
2. pip — [Install here](https://pip.pypa.io/en/stable/installation/)
3. uv — [Install here](https://docs.astral.sh/uv/getting-started/installation/)
4. gcloud CLI — [Install here](https://cloud.google.com/sdk/docs/install), then log in using:

   ```bash
   gcloud auth login
   ```
5. Git — [Install here](https://git-scm.com/downloads)
6. Trivy — [Install here](https://trivy.dev/latest/getting-started/installation/) (required by the `pre-push` git hook, which runs a vulnerability scan on changed libraries before every push)
7. Access to the [GDP Labs SDK GitHub repository](https://github.com/GDP-ADMIN/gl-sdk)

---

### 1. Clone Repository

```bash
git clone git@github.com:GDP-ADMIN/gl-sdk.git
cd gl-sdk/libs/gllm-evals
```

---

### 2. Setup Authentication

Because `gllm-evals` is a private library, you first need to configure `uv` to authenticate with our secure Google Cloud repositories.
Set the following environment variables to authenticate with internal package indexes:

```bash
export UV_INDEX_GEN_AI_INTERNAL_USERNAME=oauth2accesstoken
export UV_INDEX_GEN_AI_INTERNAL_PASSWORD="$(gcloud auth print-access-token)"
```

---

### 3. Quick Setup

Run:

```bash
make setup
```

---

### 4. Activate Virtual Environment

```bash
source .venv/bin/activate
```

---

## Local Development Utilities

The following Makefile commands are available for quick operations:

### Install uv

```bash
make install-uv
```

### Install Pre-Commit

```bash
make install-pre-commit
```

This installs both the `pre-commit` and `pre-push` git hooks from the repository-root `.pre-commit-config.yaml`. The `pre-push` hook runs unit tests, a changed-line coverage check, and a [Trivy](https://trivy.dev/latest/getting-started/installation/) vulnerability scan for every changed library before a push completes, so make sure Trivy is installed and `gcloud auth login` is active.

### Install Dependencies

```bash
make install
```

### Update Dependencies

```bash
make update
```

### Run Tests

```bash
make test
```

---

## Adding the Package

Once authorization is configured, you can add `gllm-evals` to your project:

```bash
uv add gllm-evals
```

### Dependencies

The SDK requires:
- `gllm-core` and `gllm-inference` for LLM interactions
- `pydantic` for data validation

## Quick Start

### Basic Usage

```python
import asyncio
import os
from gllm_evals.evaluator.geval_generation_evaluator import GEvalGenerationEvaluator

async def main():
    # Initialize the evaluator
    evaluator = GEvalGenerationEvaluator(
        model_credentials=os.getenv("GOOGLE_API_KEY")
    )

    # Prepare evaluation data
    data = {
        "query": "What is the capital of France?",
        "expected_response": "Paris is the capital of France.",
        "generated_response": "The capital of France is Paris.",
        "retrieved_context": "Paris is the capital and largest city of France."
    }

    # Evaluate
    result = await evaluator.evaluate(data)
    print(result)

if __name__ == "__main__":
    asyncio.run(main())
```

### Multimodal Evaluation

Use `[ATTACHMENT:<id-or-uri>]` placeholders in evaluation fields when the judge needs image context. Attachments may be declared by ID through `LLMTestCase.attachments`, or referenced inline with `https://`, `http://`, `file://`, or `data:` URIs.

```python
import asyncio

from gllm_evals import AttachmentRef, LLMTestCase, evaluate
from gllm_evals.evaluator.geval_generation_evaluator import GEvalGenerationEvaluator


async def main():
    data = [
        LLMTestCase(
            input="Describe this image: [ATTACHMENT:sample_image]",
            actual_output="The image shows a mountain landscape.",
            expected_output="A mountainous outdoor landscape is visible.",
            retrieved_context=["Reference image: [ATTACHMENT:sample_image]"],
            attachments={
                "sample_image": AttachmentRef(
                    uri="https://picsum.photos/id/29/640/480.jpg",
                    mime_type="image/jpeg",
                )
            },
        )
    ]

    results = await evaluate(data=data, evaluators=[GEvalGenerationEvaluator()])
    print(results)


if __name__ == "__main__":
    asyncio.run(main())
```

You can place image placeholders in `input`, `actual_output`, `expected_output`, `retrieved_context`, and `expected_context`. The evaluator resolves placeholders before invoking the judge; unresolved placeholders fail validation instead of being sent as raw text.

For CSV and spreadsheet datasets, put row-level attachments in an `attachments` JSON cell:

```csv
input,actual_output,expected_output,attachments
"Describe [ATTACHMENT:image_1]","A mountain landscape.","A mountain landscape.","{""image_1"":{""uri"":""https://picsum.photos/id/29/640/480.jpg"",""mime_type"":""image/jpeg""}}"
```

See `examples/evaluate/example_multimodal_evaluate_from_csv.py` for a runnable CSV example that references bundled local image files, including a chart image used for a chart-comprehension question.

Inline URI and data URL placeholders are also supported without a named attachment entry:

```python
LLMTestCase(
    input="Describe [ATTACHMENT:https://example.com/image.jpg]",
    actual_output="A product photo.",
)
```

Pass `multimodal=False` to suppress multimodal judging rules on the metrics that expose the option — `LMBasedMetric` and the GEval-family metrics (`DeepEvalGEvalMetric` and its subclasses, e.g. groundedness, completeness, and redundancy). Placeholder resolution still occurs, so raw `[ATTACHMENT:...]` text is not sent to the judge.

### Batch Evaluation

```python
import asyncio
import os
from gllm_evals.dataset.dict_dataset import DictDataset
from gllm_evals.evaluator.geval_generation_evaluator import GEvalGenerationEvaluator
from gllm_evals.runner import Runner
from gllm_evals.experiment_tracker.csv_experiment_tracker import CSVExperimentTracker

async def batch_evaluation():
    # Initialize evaluator
    evaluator = GEvalGenerationEvaluator(
        model_credentials=os.getenv("GOOGLE_API_KEY"),
        run_parallel=True  # Enable parallel processing
    )

    # Create dataset
    dataset = DictDataset([
        {
            "query": "What is the capital of France?",
            "expected_response": "Paris",
            "generated_response": "Paris is the capital of France.",
            "retrieved_context": "Paris is the capital of France."
        },
        {
            "query": "What is 1 + 1?",
            "expected_response": "2",
            "generated_response": "The answer is 2.",
            "retrieved_context": "1 + 1 equals 2."
        }
    ])

    # Run evaluation
    runner = Runner(evaluator, batch_size=10)
    results = await runner.evaluate(dataset)

    # Track results
    tracker = CSVExperimentTracker(score_key="generation/score")
    tracker.log_batch(results)

    print(f"Evaluation Results: {tracker.get_results()}")

if __name__ == "__main__":
    asyncio.run(batch_evaluation())
```

### Using Jev with GEval

Every GEval metric (built-in and custom `DeepEvalGEvalMetric`) accepts a decisions-model (DM) invoker such as TypeSafe
Jev in `models` and `fallback_models`, next to regular LM invokers. Set `OPENROUTER_API_KEY` first.

```python
from gllm_inference.dm_invoker import build_dm_invoker
from gllm_inference.lm_invoker import build_lm_invoker
from gllm_evals.metrics.generation.geval_completeness import GEvalCompletenessMetric

jev = build_dm_invoker("openrouter/typesafe/jev-1.13")

metric = GEvalCompletenessMetric(models=jev)                                         # Jev only
metric = GEvalCompletenessMetric(models=jev, fallback_models=[build_lm_invoker("openai/gpt-5-mini")])  # Jev, then an LM
metric = GEvalCompletenessMetric(models=[jev, jev, jev])                             # three judges, majority vote

result = await metric.evaluate(row)
result.score                    # normalized score in [0, 1]
result.decision.choice          # e.g. "3", the rubric score Jev chose
result.decision.confidence      # e.g. 0.99
result.decision.probabilities   # e.g. {"1": 0.0, "2": 0.01, "3": 0.99}
```

- **Fallback is per complete attempt, in the configured order.** A judge that raises an invoker error, times out, or
  gives an unusable answer (a choice that was not offered) hands the row to the next judge. If exactly one judge was
  tried and failed, its own error is raised; if two or more failed, `EvalJudgesExhaustedError` is raised with every
  attempt in `children`. A configuration, input or unsupported-operation failure stops the chain at once.
- **`result.decision`** is set only when a single Jev judge produced the score. With several judges it is `None` on the
  aggregate; each judge's own entry keeps its decision.
- **Fewshot, `evaluation_steps`, custom GEval and thresholds** behave as with an LM judge. Jev receives only the
  *configured* steps, never steps another LM generated. For a criteria-only custom metric, Jev judges from the criteria
  and the rubric.
- **YAML suites:** give a model entry `type: dm` (`type` defaults to `lm`):

  ```yaml
  evaluators:
    - class_name: GEvalGenerationEvaluator
      models:
        - type: dm
          model_id: openrouter/typesafe/jev-1.13
  ```

- **GEval only.** Every other metric (LM-based, RAGAS, LangChain, the other DeepEval metrics) raises
  `NotImplementedError` naming the metric when handed a DM invoker, and so do the evaluators built on them
  (for example `LMBasedRetrievalEvaluator`).
- **Fallback scope.** The chain moves on when a judge's invoker fails or times out, when Jev's answer is unusable, and when
  an LM judge returns invalid JSON. Other errors, such as a row missing a required field, are raised.
- **Rows with images.** Jev cannot take images, so it is skipped for such a row and an image-capable LM in
  `fallback_models` judges it. If no configured judge can, the row fails with `EvalUnsupportedOperationError`.
  A capability probe that fails for an operational reason (authentication, rate limit,
  connection, timeout) raises that failure instead and is never read as "this judge cannot take images".
- **Batch API.** A chain that contains Jev never uses the provider Batch API; list input is evaluated row by row.
- **Token usage** is reported per model; no cost is calculated for Jev.
- **`get_full_prompt`** returns the Jev request as JSON when Jev is the primary judge.

### Custom Metrics

Create domain-specific metrics easily:

```python
from gllm_evals.metrics.metric import BaseMetric
from gllm_evals.types import LLMTestCase, MetricOutput

class DomainSpecificMetric(BaseMetric):
    """Custom metric for domain-specific evaluation."""

    name = "domain_accuracy"

    async def _evaluate(self, data: LLMTestCase) -> MetricOutput:
        # Your domain-specific evaluation logic
        score = self.calculate_domain_score(data)
        return {"score": score, "explanation": "Domain-specific reasoning"}
```

A list-capable `_evaluate` that has no batch mode for its configuration signals it by raising
`BatchNotSupportedError` (from `gllm_evals.exceptions`); the rows are then evaluated one by one. Any other
exception raised for a list, including a bare `TypeError` or `NotImplementedError`, is treated as a failure.

## Error Handling

Every error the SDK raises on purpose is a `gllm_core.exceptions.BaseError` with a stable `code`. gllm-evals'
own errors are `BaseEvalError` subclasses with codes `GLLM-EVAL-*`, and each also inherits the closest built-in
exception (for example `EvalConfigurationError` is a `ValueError`), so existing `except ValueError` blocks keep
working. Errors from gllm-inference (`GLLM-INF-*`) pass through unchanged.

| Class | Code | Typical cause |
|---|---|---|
| `EvalConfigurationError` | `GLLM-EVAL-CONFIGURATION` | Invalid option, metric setup, suite document or credentials setup |
| `EvalInputError` | `GLLM-EVAL-INPUT` | Invalid row, dataset content or attachment |
| `EvalDependencyUnavailableError` | `GLLM-EVAL-DEPENDENCY-UNAVAILABLE` | Missing or incompatible optional integration |
| `EvalUnsupportedOperationError` | `GLLM-EVAL-UNSUPPORTED-OPERATION` | Row type, media, judge type or feature the component or judge does not support |
| `EvalUnsupportedSyncOperationError` | `GLLM-EVAL-UNSUPPORTED-SYNC-OPERATION` | A synchronous path was called on an async-only component. A subclass of `EvalUnsupportedOperationError` |
| `EvalResourceNotFoundError` | `GLLM-EVAL-RESOURCE-NOT-FOUND` | Missing file, worksheet or remote object |
| `EvalAccessError` | `GLLM-EVAL-ACCESS` | Authentication failed or permission denied |
| `EvalConnectionError` | `GLLM-EVAL-CONNECTION` | Connection could not be made or was lost; retryable |
| `EvalTimeoutError` | `GLLM-EVAL-TIMEOUT` | Request, batch polling or retry deadline expired |
| `EvalRateLimitError` | `GLLM-EVAL-RATE-LIMIT` | A non-inference service throttled the request |
| `EvalServiceError` | `GLLM-EVAL-SERVICE` | A non-inference service failed or rejected the request |
| `EvalServiceUnavailableError` | `GLLM-EVAL-SERVICE-UNAVAILABLE` | A non-inference service failed temporarily (HTTP 500, 502, 503 and 504); retryable. A subclass of `EvalServiceError` |
| `EvalIOError` | `GLLM-EVAL-IO` | Local read, write or cleanup failed |
| `EvalSerializationError` | `GLLM-EVAL-SERIALIZATION` | SDK-owned artifact could not be encoded or decoded |
| `EvalJudgeResponseError` | `GLLM-EVAL-JUDGE-RESPONSE` | A judge answered, but the answer is unusable |
| `EvalResultContractError` | `GLLM-EVAL-RESULT-CONTRACT` | A metric or evaluator returned a result of the wrong shape |
| `EvalAggregationError` | `GLLM-EVAL-AGGREGATION` | Results could not be combined, e.g. every metric or judge failed |
| `EvalJudgesExhaustedError` | `GLLM-EVAL-JUDGES-EXHAUSTED` | Two or more judges in a fallback chain failed |
| `EvalExecutionError` | `GLLM-EVAL-EXECUTION` | Unexpected failure; give support the reference ID |

Each error also has `recovery` guidance (`retryable`, and `retry_after_seconds` when the service reported one)
and, once it has been recorded, a `reference_id`.

**Handle errors by class or code, never by message text.** Messages are written for people and may change. Every
message says what happened and then what to do next, whether it is a class's default message or a more specific
one.

```python
from gllm_core.exceptions import BaseError
from gllm_evals import evaluate
from gllm_evals.exceptions import EvalConfigurationError, EvalInputError

try:
    result = await evaluate(data="data.csv", evaluators=[evaluator])
except EvalInputError as error:           # the caller can fix the data
    print(error.code, error.message)
except EvalConfigurationError as error:   # the caller can fix the setup
    print(error.code, error.message)
except BaseError as error:                # any other normalized failure
    if error.recovery.get("retryable"):
        ...  # retry later
    raise
```

**A failed row does not raise.** When an evaluator or metric fails on a row, the run continues and that result
carries an `error` record (an `ErrorInfo` model) instead of a score, with `code`, `message` (safe to show), `reference_id`,
`error_type`, `origin`, `recovery`, `component`, `operation` and `children` (the failures it summarizes). A failed
evaluator's result is keyed by the evaluator's name, like a successful one. Use `is_failed_result` to tell a
failure from a valid low score:

```python
from gllm_evals.exceptions import is_failed_result

for row in result.results:
    for evaluator_result in row:
        for name, output in evaluator_result.items():
            if is_failed_result(output):
                print(name, output.error.code, output.error.reference_id)
```

Result fields, messages and log records never contain row content, prompts, file paths, URLs, credentials or the
upstream error text. Handled failures are logged once on the `gllm_evals.errors` logger with the code in
`extra["error_code"]`; the original exception stays available as `__cause__` on a raised error. Direct Pydantic
model construction (for example `LLMTestCase(...)`) still raises `pydantic.ValidationError`.

See `examples/example_error_handling.py` for a runnable walkthrough.

## Architecture

### Core Components

#### 1. Evaluators
- **`BaseEvaluator`**: Abstract base class for all evaluators - extend for any evaluation scenario
- **`GEvalGenerationEvaluator`**: Production-ready GEval-backed evaluator for text generation quality with rule-based scoring

#### 2. Metrics
- **`BaseMetric`**: Abstract base class for metrics - create custom metrics for any domain
- **`LMBasedMetric`**: Generic LM-powered metric evaluation with customizable prompts

#### 3. Datasets
- **`BaseDataset`**: Abstract base class for datasets - support any data format
- **`DictDataset`**: Simple dictionary-based dataset implementation

### 4. Runner
- **`Runner`**: Runner class for batch evaluation

# Metrics
Below is a list of metrics that are currently supported by the SDK.

| Metric | Description | Type | Score Range |
|--------|-------------|------|-------------|
| LMBasedMetric | An all purpose metric that can be used to evaluate any metric that can be expressed as a LM prompt | LM-based | - |
| DeepEvalGEvalMetric | A versatile evaluation metric framework that can be used to create custom evaluation metrics with configurable criteria, evaluation steps, and rubrics | LM-based | - |
| GEvalCompletenessMetric | A metric that can be used to evaluate the completeness of the generated output | DeepEval GEval | 1-3 |
| GEvalRedundancyMetric | A metric that can be used to evaluate the redundancy of the generated output | DeepEval GEval | 1-3 |
| GEvalGroundednessMetric | A metric that can be used to evaluate the groundedness of the generated output | DeepEval GEval | 1-3 |
| GEvalLanguageConsistencyMetric | A metric that can be used to evaluate language consistency between query and generated response | DeepEval GEval | 0-1 |
| GEvalRefusalMetric | A metric that can be used to evaluate refusal behavior from query and expected response | DeepEval GEval | 0-1 |
| GEvalRefusalAlignmentMetric | A metric that can be used to evaluate refusal alignment between expected and generated responses | DeepEval GEval | 0-1 |

# Evaluators
Below is a list of evaluators that are currently supported by the SDK.

| Evaluator | Description | Type |
|--------|-------------|------|
| GEvalGenerationEvaluator | An evaluator that can be used to evaluate the quality of the generated output | LLM-based |

# Datasets
Below is a list of datasets that are currently supported by the SDK.

| Dataset | Description |
|--------|-------------|
| DictDataset | A dataset that loads data from a dictionary |
| HuggingFaceDataset |  A dataset that loads data from a HuggingFace dataset |
