Metadata-Version: 2.5
Name: gaussia
Version: 1.2.0
Summary: Gaussia - AI evaluation framework for measuring fairness, quality, and safety of AI models and assistants
Project-URL: Homepage, https://github.com/gaussia-labs/pygaussia
Project-URL: Repository, https://github.com/gaussia-labs/pygaussia.git
Project-URL: Bug Tracker, https://github.com/gaussia-labs/pygaussia/issues
Project-URL: Changelog, https://github.com/gaussia-labs/pygaussia/releases
Author: Gaussia Labs
Maintainer-email: Alex Fiorenza <alexfiorenza2012@gmail.com>
License: MIT
License-File: LICENSE
Keywords: ai,assistant,bias,evaluation,fairness,llm,metrics,ml,nlp,toxicity
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: jinja2>=3.1.0
Requires-Dist: langchain-core>=1.2.11
Requires-Dist: langchain>=1.2
Requires-Dist: loguru>=0.7.3
Requires-Dist: optuna>=4.7.0
Requires-Dist: pydantic<3.0.0,>=2.0.0
Requires-Dist: scipy>=1.14.1
Requires-Dist: tqdm>=4.65.0
Requires-Dist: transformers>=4.35.0
Provides-Extra: agentic
Provides-Extra: all
Requires-Dist: accelerate>=0.25.0; extra == 'all'
Requires-Dist: hdbscan>=0.8.41; extra == 'all'
Requires-Dist: interpreto>=0.1.0; extra == 'all'
Requires-Dist: nltk>=3.8.0; extra == 'all'
Requires-Dist: numba>=0.57.0; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: optuna>=3.0.0; extra == 'all'
Requires-Dist: pandas>=2.0.0; extra == 'all'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'all'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'all'
Requires-Dist: torch>=2.0.0; extra == 'all'
Requires-Dist: transformers>=4.35.0; extra == 'all'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'all'
Provides-Extra: bestof
Provides-Extra: bias
Requires-Dist: torch>=2.0.0; extra == 'bias'
Provides-Extra: context
Provides-Extra: conversational
Provides-Extra: evalhub
Requires-Dist: accelerate>=0.25.0; extra == 'evalhub'
Requires-Dist: eval-hub-sdk[adapter]<0.2.0,>=0.1.5; extra == 'evalhub'
Requires-Dist: hdbscan>=0.8.41; extra == 'evalhub'
Requires-Dist: nltk>=3.8.0; extra == 'evalhub'
Requires-Dist: numba>=0.57.0; extra == 'evalhub'
Requires-Dist: numpy>=1.24.0; extra == 'evalhub'
Requires-Dist: pandas>=2.0.0; extra == 'evalhub'
Requires-Dist: requests>=2.33.0; extra == 'evalhub'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'evalhub'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'evalhub'
Requires-Dist: torch>=2.0.0; extra == 'evalhub'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'evalhub'
Provides-Extra: explainability
Requires-Dist: interpreto>=0.1.0; extra == 'explainability'
Requires-Dist: torch>=2.0.0; extra == 'explainability'
Requires-Dist: transformers>=4.35.0; extra == 'explainability'
Provides-Extra: generators
Provides-Extra: humanity
Requires-Dist: numpy>=1.24.0; extra == 'humanity'
Requires-Dist: pandas>=2.0.0; extra == 'humanity'
Provides-Extra: metrics
Requires-Dist: accelerate>=0.25.0; extra == 'metrics'
Requires-Dist: hdbscan>=0.8.41; extra == 'metrics'
Requires-Dist: nltk>=3.8.0; extra == 'metrics'
Requires-Dist: numba>=0.57.0; extra == 'metrics'
Requires-Dist: numpy>=1.24.0; extra == 'metrics'
Requires-Dist: pandas>=2.0.0; extra == 'metrics'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'metrics'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'metrics'
Requires-Dist: torch>=2.0.0; extra == 'metrics'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'metrics'
Provides-Extra: privacy-huggingface
Requires-Dist: torch>=2.1; extra == 'privacy-huggingface'
Requires-Dist: transformers>=4.40; extra == 'privacy-huggingface'
Provides-Extra: privacy-presidio
Requires-Dist: presidio-analyzer>=2.2; extra == 'privacy-presidio'
Requires-Dist: spacy>=3.7; extra == 'privacy-presidio'
Provides-Extra: prompt-optimizer
Requires-Dist: optuna>=3.0.0; extra == 'prompt-optimizer'
Provides-Extra: regulatory
Requires-Dist: accelerate>=0.25.0; extra == 'regulatory'
Requires-Dist: torch>=2.0.0; extra == 'regulatory'
Provides-Extra: roastme
Requires-Dist: networkx>=3.0; extra == 'roastme'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'roastme'
Requires-Dist: torch>=2.0.0; extra == 'roastme'
Provides-Extra: roastme-rl
Requires-Dist: accelerate>=0.25.0; extra == 'roastme-rl'
Requires-Dist: networkx>=3.0; extra == 'roastme-rl'
Requires-Dist: peft>=0.10.0; extra == 'roastme-rl'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'roastme-rl'
Requires-Dist: torch>=2.0.0; extra == 'roastme-rl'
Requires-Dist: trl>=0.8.0; extra == 'roastme-rl'
Provides-Extra: role-adherence
Provides-Extra: toxicity
Requires-Dist: hdbscan>=0.8.41; extra == 'toxicity'
Requires-Dist: nltk>=3.8.0; extra == 'toxicity'
Requires-Dist: numba>=0.57.0; extra == 'toxicity'
Requires-Dist: numpy>=1.24.0; extra == 'toxicity'
Requires-Dist: pandas>=2.0.0; extra == 'toxicity'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'toxicity'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'toxicity'
Requires-Dist: torch>=2.0.0; extra == 'toxicity'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'toxicity'
Provides-Extra: vision
Requires-Dist: numpy>=1.24.0; extra == 'vision'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'vision'
Requires-Dist: torch>=2.0.0; extra == 'vision'
Description-Content-Type: text/markdown

# Gaussia

[![PyPI version](https://img.shields.io/pypi/v/gaussia)](https://pypi.org/project/gaussia/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/gaussia)](https://pypi.org/project/gaussia/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/gaussia)](https://pypi.org/project/gaussia/)
[![PyPI - License](https://img.shields.io/pypi/l/gaussia)](https://pypi.org/project/gaussia/)

AI evaluation framework for measuring fairness, quality, and safety of AI models and assistants.

## Installation

```bash
pip install gaussia
```

With specific metric dependencies:

```bash
pip install gaussia[toxicity]              # Toxicity analysis
pip install gaussia[bias]                  # Bias detection
pip install gaussia[privacy-presidio]      # Privacy with the Presidio detector
pip install gaussia[privacy-huggingface]   # Privacy with a HuggingFace NER detector
pip install gaussia[evalhub]               # EvalHub provider adapter
pip install gaussia[metrics]               # All metrics
pip install gaussia[all]                   # Everything
```

Roast Me ships its own extras, deliberately outside `metrics` and `all`, so whoever only profiles an
assistant pays for neither retrieval nor training:

```bash
pip install gaussia[roastme]      # The three corpus-reading probe engines
pip install gaussia[roastme-rl]   # The above, plus the reinforcement-learning search
```

Most metrics judge with a LangChain-compatible chat model, which you install separately:
`langchain-openai`, `langchain-anthropic`, `langchain-google-genai`, `langchain-groq`,
`langchain-ollama`.

## Quick Start

```python
from gaussia import Retriever, Dataset, Batch
from gaussia.metrics.context import Context
from langchain_openai import ChatOpenAI

# 1. Define your data source
class MyRetriever(Retriever):
    def load_dataset(self) -> list[Dataset]:
        return [
            Dataset(
                session_id="session-1",
                assistant_id="assistant-1",
                language="en",
                context="France is a country in Western Europe.",
                conversation=[
                    Batch(
                        qa_id="q1",
                        query="Where is France?",
                        assistant="France is located in Western Europe.",
                        ground_truth_assistant="France is a country in Western Europe.",
                    )
                ],
            )
        ]

# 2. Run a metric. `run` takes the retriever *class* and instantiates it for you.
metrics = Context.run(MyRetriever, model=ChatOpenAI(model="gpt-4o-mini", temperature=0.0))
```

Metrics are imported from their own modules — `gaussia.metrics` declares the surface but imports
nothing, so no metric drags in another's optional dependencies.

## Metrics

| Metric | Description | Install extra |
|--------|-------------|---------------|
| **Context** | Evaluates response alignment with provided context | — |
| **Conversational** | Dialogue quality via Grice's maxims (memory, language, quality, quantity, relation, manner) | — |
| **BestOf** | King-of-the-hill tournament comparison of multiple assistants | — |
| **Agentic** | Agent evaluation with pass@K and tool correctness | — |
| **RoleAdherence** | Whether the assistant stays inside its defined role, scored from judge first-token logprobs | — |
| **Toxicity** | Cluster-based toxicity profiling with demographic and sentiment analysis | `[toxicity]` |
| **Bias** | Bias detection across protected attributes using guardians | `[bias]` |
| **Privacy** | Domain-adjusted detection score rating one PII/PHI detector's fitness for a regulated domain | `[privacy-presidio]` / `[privacy-huggingface]` |
| **PrivacyRanker** | The same score across several detectors, ranked, to decide which one to ship | `[privacy-presidio]` / `[privacy-huggingface]` |
| **Humanity** | Emotion, empathy, and human-like quality analysis | `[humanity]` |
| **Regulatory** | Compliance evaluation against regulatory documents | `[regulatory]` |
| **VisionSimilarity** | VLM description comparison via semantic similarity | `[vision]` |
| **VisionHallucination** | Hallucination detection in VLM outputs | `[vision]` |

## Features

### Guardians

Pluggable bias detection backends. The guardian is passed as a class, like the retriever:

```python
from gaussia.guardians import IBMGranite, LLamaGuard
from gaussia.metrics.bias import Bias

metrics = Bias.run(MyRetriever, guardian=IBMGranite)
```

### PII Detectors

Pluggable detection backends behind the `PIIDetector` contract, the same way guardians sit behind
`Guardian`. A detector carries the expert `[0, 1]` scalars that place it in your domain, so it is
passed as an instance:

```python
from gaussia.detectors.presidio import PresidioDetector
from gaussia.metrics.privacy import Privacy
from gaussia.schemas.privacy import PrivacyDomainConfig

domain = PrivacyDomainConfig(
    classes=frozenset({"email_address", "phone_number"}),
    criticality_weights={"email_address": 0.6, "phone_number": 0.4},
    fn_severity_weights={"email_address": 0.7, "phone_number": 0.3},
    regulatory_framework="GDPR",
)
detector = PresidioDetector(name="presidio", domain_fit=0.9, regulatory_fit=0.8)

metrics = Privacy.run(MyRetriever, detector=detector, domain_config=domain)
```

`PrivacyRanker` takes `detectors=[...]` instead and ranks them under the same domain config.

### Role Adherence

Whether the assistant stayed in the role it was given, scored per turn and aggregated per session.
The judge derives a calibrated `[0, 1]` score from first-token logprobs, so it needs a provider that
exposes them — `StructuredOutputJudgeStrategy` is the fallback for providers that do not:

```python
from gaussia.metrics.role_adherence import RoleAdherence, LLMJudgeStrategy
from langchain_openai import ChatOpenAI

strategy = LLMJudgeStrategy(model=ChatOpenAI(model="gpt-4o-mini"))
metrics = RoleAdherence.run(MyRetriever, scoring_strategy=strategy)
```

### Statistical Modes

Choose between frequentist and Bayesian aggregation:

```python
from gaussia import FrequentistMode, BayesianMode
from gaussia.metrics.context import Context

metrics = Context.run(MyRetriever, model=judge, statistical_mode=FrequentistMode())
metrics = Context.run(MyRetriever, model=judge, statistical_mode=BayesianMode())
```

### Synthetic Data Generation

Generate evaluation datasets from documents:

```python
from gaussia.generators import BaseGenerator, create_markdown_loader
from langchain_openai import ChatOpenAI

generator = BaseGenerator(model=ChatOpenAI(model="gpt-4o-mini"))
loader = create_markdown_loader()

datasets = await generator.generate_dataset(
    context_loader=loader,
    source="./docs/knowledge_base.md",
    assistant_id="my-assistant",
)
```

### Roast Me

Adversarial evaluation as a search problem: profile an assistant's weaknesses from tagged probes,
then look for the *categories* of realistic question that break it reproducibly. A generator
subsystem, not a metric — nothing subclasses `Gaussia`. What enters the metric pipeline is the Roast
Dataset it emits, which existing metrics consume unchanged.

```python
from gaussia.generators.roastme import ProbeLibrary, Profiler, to_dataset

probes = ProbeLibrary(engines).generate(documents, catalogue)
result = Profiler(contract=contract, target=your_adapter).profile(probes)

dataset = to_dataset(
    probes,
    result.outcomes,
    session_id="roast-run-1",
    assistant_id="support-assistant",
    context="Roast Me run over the policy knowledge base",
)
```

> Every number Roast Me produces is a judge-only measurement: one language model's estimate of
> whether another one misbehaved. No grader here has been calibrated against human labels, so read a
> violation rate as evidence to look at, never as a measured error rate.

### Explainability

Token-level attribution analysis. The method is a class, not a string:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from gaussia.explainability import AttributionExplainer, Lime

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

explainer = AttributionExplainer(model, tokenizer)
result = explainer.explain(
    prompt=tokenizer.apply_chat_template([{"role": "user", "content": "What is gravity?"}], tokenize=False),
    target="Gravity is the force of attraction between objects.",
    method=Lime,
)
print(result.get_top_k(5))
```

### Prompt Optimization

Optimize prompts using evolutionary and multi-objective strategies:

```python
from gaussia.prompt_optimizer import GEPAOptimizer, MIPROv2Optimizer
```

### EvalHub Provider

Run Gaussia as an EvalHub BYOF provider:

```bash
python -m gaussia.integrations.evalhub.adapter
```

## Documentation

Full documentation available at [docs.gaussia.ai](https://docs.gaussia.ai).

## Requirements

- Python >= 3.11

## License

MIT
