Metadata-Version: 2.4
Name: trustmodel
Version: 0.3.1
Summary: Official Python SDK for TrustModel AI evaluation platform
Project-URL: Homepage, https://trustmodel.ai
Project-URL: Documentation, https://docs.trustmodel.ai/sdk/python
Project-URL: Repository, https://github.com/trustmodel/trustmodel-python-sdk
Project-URL: Bug Tracker, https://github.com/trustmodel/trustmodel-python-sdk/issues
Author-email: TrustModel <support@trustmodel.ai>
License: Proprietary - TrustModel Python SDK License
License-File: LICENSE
Keywords: ai,api,evaluation,sdk,trustmodel
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Requires-Dist: pydantic<3.0.0,>=2.0.0
Requires-Dist: python-dateutil>=2.8.0
Requires-Dist: requests<3.0.0,>=2.25.0
Requires-Dist: typing-extensions>=4.0.0
Provides-Extra: dev
Requires-Dist: black>=23.0.0; extra == 'dev'
Requires-Dist: bump2version>=1.0.1; extra == 'dev'
Requires-Dist: importlib-metadata>=4.0.0; (python_version < '3.8') and extra == 'dev'
Requires-Dist: isort>=5.12.0; extra == 'dev'
Requires-Dist: mypy>=1.5.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Requires-Dist: types-importlib-metadata>=4.0.0; (python_version < '3.8') and extra == 'dev'
Requires-Dist: types-requests>=2.25.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: sphinx-autodoc-typehints>=1.24.0; extra == 'docs'
Requires-Dist: sphinx-rtd-theme>=1.3.0; extra == 'docs'
Requires-Dist: sphinx>=7.0.0; extra == 'docs'
Description-Content-Type: text/markdown

<p align="center">
  <a href="https://www.trustmodel.ai">
    <img src="https://www.trustmodel.ai/assets/trustmodel-wordmark-CfuXSOoK.svg" alt="TrustModel" width="400">
  </a>
</p>

<p align="center">
  <strong>Official Python SDK for the TrustModel AI evaluation platform</strong>
</p>

<p align="center">
  <a href="https://www.trustmodel.ai">Website</a> •
  <a href="https://docs.trustmodel.ai/sdk/python">Documentation</a> •
  <a href="https://app.trustmodel.ai">Dashboard</a>
</p>

<p align="center">
  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.9+-blue.svg" alt="Python 3.9+"></a>
  <a href="https://pypi.org/project/trustmodel/"><img src="https://badge.fury.io/py/trustmodel.svg" alt="PyPI version"></a>
</p>

---

Evaluate AI models for safety, bias, and performance with a simple, intuitive interface.

## Features

- 🚀 **Simple Interface**: Easy-to-use client for all TrustModel operations
- 🔒 **Secure**: API key authentication with built-in validation
- 🎯 **Type Safe**: Full type hints for excellent IDE support
- 🔄 **Reliable**: Automatic retries and comprehensive error handling
- 📊 **Comprehensive**: Support for all evaluation types and configurations
- 🌍 **Framework Agnostic**: Works with any Python framework or standalone scripts

## Installation

```bash
pip install trustmodel
```

## Prerequisites

Before using the SDK, you **must** complete the following setup in the [TrustModel Dashboard](https://app.trustmodel.ai/app/keys):

### 1. Create an API Key (Required)

You need a TrustModel API key to authenticate all SDK requests:

1. Go to [Keys & Webhooks](https://app.trustmodel.ai/app/keys) in the dashboard
2. Click "Create API Key"
3. Copy your new API key (starts with `tm-`)
4. Store it securely - you won't be able to see it again

### 2. Configure Webhooks (Required)

To receive notifications when evaluations complete or fail, you must configure webhooks:

1. Go to [Keys & Webhooks](https://app.trustmodel.ai/app/keys) in the dashboard
2. Click "Create Webhook"
3. Enter your webhook endpoint URL
4. Select the events you want to receive
5. Save your webhook configuration

> **Important:** Without configuring both an API key and webhooks in the webapp, you cannot run evaluations. The API will return an error if these are not set up.

## Quick Start

```python
import trustmodel

# Initialize the client
client = trustmodel.TrustModelClient(api_key="tm-your-api-key-here")

# List available models
models, api_sources = client.models.list()
print(f"Found {len(models)} models available")

# Create an evaluation
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    categories=["safety", "bias", "performance"]
)

print(f"Evaluation created with ID: {evaluation.id}")
print(f"Status: {evaluation.status}")

# You'll receive a webhook notification when the evaluation completes
# Then retrieve the results:
completed_evaluation = client.evaluations.get(evaluation.id)
print(f"Overall score: {completed_evaluation.overall_score}")

# Check your credit balance
credits = client.credits.get_balance()
print(f"Credits remaining: {credits.credits_remaining}")
```

## Authentication

Get your API key from the [TrustModel Dashboard](https://app.trustmodel.ai/app/keys) and use it to initialize the client:

```python
import trustmodel

client = trustmodel.TrustModelClient(api_key="tm-your-api-key-here")
```

For production applications, store your API key securely using environment variables:

```python
import os
import trustmodel

api_key = os.getenv("TRUSTMODEL_API_KEY")
client = trustmodel.TrustModelClient(api_key=api_key)
```

## Evaluation Modes

TrustModel supports three ways to evaluate AI models:

| Mode | Use Case | API Key Required |
|------|----------|------------------|
| **Platform Key** | Quick evaluations using TrustModel's API keys | No (uses TrustModel's keys) |
| **BYOK** | Use your own vendor API key for any model | Yes (your vendor API key) |
| **Custom Endpoint** | Evaluate private/self-hosted models | Yes (your endpoint's API key) |

### Getting Available Vendors

Use `client.config.get().vendors` to discover available vendors:

```python
config = client.config.get()

# Public vendors - for Platform Key and BYOK evaluations
public_vendors = config.vendors["public"]
for vendor in public_vendors:
    print(f"{vendor['identifier']}: {vendor['name']}")

# Custom vendors - for Custom Endpoint evaluations only
custom_vendors = config.vendors["custom"]
for vendor in custom_vendors:
    print(f"{vendor['identifier']}: {vendor['name']}")
```

| Vendor Type | Use With | Description |
|-------------|----------|-------------|
| `public` | Platform Key, BYOK | Vendors like OpenAI, Anthropic, Google AI for standard evaluations |
| `custom` | Custom Endpoint | Validators for self-hosted/private endpoints (OpenAI-compatible, Hugging Face, Azure AI, etc.) |

### Getting Available Models

Use `client.models.list()` to discover available models:

```python
# Get all available models and API source info
models, api_sources = client.models.list()

# List all models with their details
for model in models:
    print(f"Model: {model.name}")
    print(f"  Identifier: {model.model_identifier}")
    print(f"  Vendor: {model.vendor_identifier}")
    print(f"  Platform Key Available: {model.available_via_trust_model_key}")
    print(f"  BYOK Available: {model.available_via_byok}")

# Filter models by vendor
openai_models = [m for m in models if m.vendor_identifier == "openai"]

# Filter models available via platform key (no vendor API key needed)
platform_key_models = [m for m in models if m.available_via_trust_model_key]

# Use a model in evaluation
model = models[0]
evaluation = client.evaluations.create(
    model_identifier=model.model_identifier,
    vendor_identifier=model.vendor_identifier,
    categories=["safety", "bias"]
)
```

| Model Field | Type | Description |
|-------------|------|-------------|
| `name` | str | Human-readable model name |
| `model_identifier` | str | Identifier to use in API calls |
| `vendor_identifier` | str | Vendor identifier |
| `available_via_trust_model_key` | bool | Can evaluate without vendor API key |
| `available_via_byok` | bool | Previously used with your own API key |

### Platform Key (Default)

Use TrustModel's platform keys for quick evaluations. No vendor API key needed:

```python
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    categories=["safety", "bias"]
)
```

**Note:** Platform key availability varies by model. Check `model.available_via_trust_model_key` to see if a model supports this mode.

### BYOK (Bring Your Own Key)

Use your own vendor API key to evaluate any model. All vendors support BYOK:

```python
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    api_key="sk-your-openai-key",  # Your OpenAI API key
    categories=["safety", "bias"]
)
```

**How it works:**
1. You provide your vendor API key (e.g., OpenAI, Anthropic, Google)
2. TrustModel validates the key before creating the evaluation
3. If validation fails, a `ConnectionValidationError` is raised with details
4. Your key is securely stored and used for the evaluation

**Getting vendor API keys:**
- **OpenAI**: [platform.openai.com/api-keys](https://platform.openai.com/api-keys)
- **Anthropic**: [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys)
- **Google AI**: [aistudio.google.com/apikey](https://aistudio.google.com/apikey)

**Example with error handling:**

```python
from trustmodel import ConnectionValidationError, InsufficientCreditsError

try:
    evaluation = client.evaluations.create(
        model_identifier="gpt-4",
        vendor_identifier="openai",
        api_key="sk-your-openai-key",
        categories=["safety", "bias"]
    )
    print(f"Evaluation created: {evaluation.id}")
except ConnectionValidationError as e:
    # API key validation failed
    print(f"Invalid API key: {e.message}")
    if e.validation_details:
        print(f"Details: {e.validation_details}")
except InsufficientCreditsError as e:
    print(f"Need more credits: {e.credits_required} required")
```

### Custom Endpoint

Evaluate your own OpenAI-compatible API endpoint (Ollama, vLLM, LiteLLM, Azure AI, etc.):

```python
# Create evaluation for a custom endpoint
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="https://api.yourcompany.com/v1",
    api_key="your-api-key",
    model_identifier="your-model-id",
    vendor_identifier="openai",  # Determines which validator to use
    model_name="My Custom Model",  # Optional display name
    categories=["safety", "bias"]
)
```

**Available vendor identifiers for custom endpoints:**

Get the list programmatically with `client.config.get().vendors["custom"]`, or use one of these:

| Identifier | Use For |
|------------|---------|
| `openai` | OpenAI-compatible APIs (Ollama, vLLM, LiteLLM, etc.) - **default** |
| `huggingface` | Hugging Face Inference Endpoints |
| `azure_ai` | Azure AI / Azure OpenAI Service |
| `xai` | Google Vertex AI |
| `bedrock` | AWS Bedrock |

**Examples:**

```python
# Ollama endpoint (uses default "openai" validator)
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="http://localhost:11434/v1",
    api_key="ollama",  # Ollama doesn't require a real key
    model_identifier="llama3:8b"
)

# Azure AI endpoint
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="https://your-resource.openai.azure.com",
    api_key="your-azure-key",
    model_identifier="gpt-4",
    vendor_identifier="azure_ai"
)

# Hugging Face endpoint
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="https://api-inference.huggingface.co/models/your-model",
    api_key="hf_your_token",
    model_identifier="your-model",
    vendor_identifier="huggingface"
)
```

---

## Core Concepts

### Models

Discover available AI models:

```python
# List all available models
models, api_sources = client.models.list()

for model in models:
    print(f"Model: {model.name}")
    print(f"Vendor: {model.vendor_identifier}")
    print(f"Platform key available: {model.available_via_trust_model_key}")
    print(f"Previously used BYOK: {model.available_via_byok}")
    print("---")

# Get specific model
model = client.models.get("openai", "gpt-4")
print(f"Found model: {model.name}")
```

**Note:** `available_via_byok` indicates you have previously used BYOK for this vendor. All vendors support BYOK - you can use your own API key with any model.

### Evaluations

Create and manage AI model evaluations:

```python
# Platform key (default) - uses TrustModel's keys
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    categories=["safety", "bias"]
)

# BYOK - uses your own API key
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    api_key="sk-your-openai-key",
    categories=["safety", "bias"]
)

# Custom endpoint - your own API
evaluation = client.evaluations.create_custom_endpoint(
    api_endpoint="https://api.yourcompany.com/v1",
    api_key="your-api-key",
    model_identifier="custom-model-v1"
)
```

#### Managing Evaluations

```python
# List all evaluations
evaluations = client.evaluations.list()

# Filter by status
completed = client.evaluations.list(status="completed")

# Get detailed results
evaluation = client.evaluations.get(evaluation_id)
if evaluation.status == "completed":
    print(f"Overall Score: {evaluation.overall_score}")
    for score in evaluation.scores:
        print(f"{score.category}: {score.score:.2f}")

# Quick status check
status = client.evaluations.get_status(evaluation_id)
print(f"Progress: {status['completion_percentage']}%")
```

### Configuration

Discover available options for evaluations:

```python
# Get configuration options
config = client.config.get()

print("Available application types:")
for app_type in config.application_types:
    print(f"  {app_type['id']}: {app_type['name']}")

print("Available categories:")
for category in config.categories:
    print(f"  {category}")

print(f"Credits per category: {config.credits_per_category}")
```

### Credits Management

Monitor your API key usage:

```python
# Check credit balance
credits = client.credits.get_balance()

print(f"API Key: {credits.api_key_name}")
print(f"Credits Used: {credits.credits_used}")
print(f"Credits Remaining: {credits.credits_remaining}")
print(f"Credit Limit: {credits.credit_limit}")
print(f"Status: {credits.status}")
```

## Error Handling

The SDK provides specific exceptions for different error types:

```python
import trustmodel
from trustmodel import (
    AuthenticationError,
    ConnectionValidationError,
    InsufficientCreditsError,
    RateLimitError,
    ValidationError,
    APIError
)

try:
    client = trustmodel.TrustModelClient(api_key="tm-your-key")
    evaluation = client.evaluations.create(
        model_identifier="gpt-4",
        vendor_identifier="openai",
        api_key="sk-your-openai-key"  # BYOK
    )
except AuthenticationError:
    print("Invalid TrustModel API key")
except ConnectionValidationError as e:
    # BYOK or custom endpoint validation failed
    print(f"Vendor API key validation failed: {e.message}")
    if e.validation_details:
        status_code = e.validation_details.get("status_code")
        if status_code == 401:
            print("Check your vendor API key is valid and not expired")
        elif status_code == 404:
            print("Model not found - check the model identifier")
except InsufficientCreditsError as e:
    print(f"Need {e.credits_required} credits, but only {e.credits_remaining} remaining")
except RateLimitError:
    print("Rate limit exceeded, please wait")
except ValidationError as e:
    print(f"Invalid input: {e}")
except APIError as e:
    print(f"API error: {e.message} (status: {e.status_code})")
```

### Exception Reference

| Exception | When Raised |
|-----------|-------------|
| `AuthenticationError` | Invalid TrustModel API key |
| `ConnectionValidationError` | BYOK or custom endpoint API key validation failed |
| `InsufficientCreditsError` | Not enough credits for the evaluation |
| `RateLimitError` | Too many requests, need to wait |
| `ValidationError` | Invalid input parameters |
| `ModelNotFoundError` | Requested model doesn't exist |
| `EvaluationNotFoundError` | Requested evaluation doesn't exist |
| `APIError` | General API error (base class) |

## Webhook Notifications

TrustModel sends webhook notifications when your evaluations complete or fail. Configure your webhook endpoint in the [TrustModel Dashboard](https://app.trustmodel.ai/app/keys) to receive these events.

### Success Event: `sdk_report_evaluation_success`

Sent when an evaluation completes successfully:

```json
{
  "event_type": "sdk_report_evaluation_success",
  "timestamp": "2026-01-21T13:41:44.253319+00:00",
  "evaluation_run_id": 82,
  "model_identifier": "gpt-4",
  "status": "completed",
  "completion_percentage": 100,
  "overall_score": 65,
  "category_scores": [
    {
      "category_name": "Accuracy",
      "category_score": 100.0,
      "subcategories": [
        {
          "subcategory_name": "Citation & Source Accuracy",
          "subcategory_score": 100.0
        }
      ]
    }
  ]
}
```

### Failure Event: `sdk_report_evaluation_failed`

Sent when an evaluation fails:

```json
{
  "event_type": "sdk_report_evaluation_failed",
  "timestamp": "2026-01-21T12:38:18.349320+00:00",
  "evaluation_run_id": 78,
  "model_identifier": "gpt-4",
  "failed_phase": "evaluation",
  "failed_at": "2026-01-21T12:38:18.341673+00:00"
}
```

### Webhook Event Fields

| Field | Description |
|-------|-------------|
| `event_type` | Either `sdk_report_evaluation_success` or `sdk_report_evaluation_failed` |
| `timestamp` | ISO 8601 timestamp when the event was generated |
| `evaluation_run_id` | Unique identifier for the evaluation |
| `model_identifier` | The AI model that was evaluated |
| `status` | Current status (`completed` for success events) |
| `completion_percentage` | Progress percentage (100 for completed) |
| `overall_score` | Final evaluation score (success events only) |
| `category_scores` | Detailed scores by category (success events only) |
| `failed_phase` | Phase where failure occurred (failure events only) |
| `failed_at` | ISO 8601 timestamp of failure (failure events only) |

## Advanced Usage

### Context Manager

Use the client as a context manager for automatic cleanup:

```python
with trustmodel.TrustModelClient(api_key="tm-your-key") as client:
    evaluation = client.evaluations.create(
        model_identifier="gpt-4",
        vendor_identifier="openai"
    )
    # Client automatically closed when exiting context
```

### Custom Configuration

```python
# Custom base URL and timeouts
client = trustmodel.TrustModelClient(
    api_key="tm-your-key",
    base_url="https://api-staging.trustmodel.ai",  # Use staging environment
    timeout=120,  # 2 minute timeout
    max_retries=5  # More aggressive retrying
)
```

### Detailed Evaluation Configuration

```python
evaluation = client.evaluations.create(
    model_identifier="gpt-4",
    vendor_identifier="openai",
    categories=["safety", "bias", "performance"],

    # Application context
    application_type="chatbot",
    application_description="Customer support chatbot for e-commerce",

    # User personas
    user_personas=["external-customer", "technical-user"],

    # Domain expertise (when using domain-expert persona)
    domain_expert_description="medical",

    # Custom naming
    model_config_name="GPT-4 Production Eval 2024-01"
)
```

## Framework Integration

### FastAPI

```python
from fastapi import FastAPI, HTTPException
import trustmodel

app = FastAPI()
client = trustmodel.TrustModelClient(api_key="tm-your-key")

@app.post("/evaluate")
async def create_evaluation(model: str, vendor: str):
    try:
        evaluation = client.evaluations.create(
            model_identifier=model,
            vendor_identifier=vendor
        )
        return {"evaluation_id": evaluation.id, "status": evaluation.status}
    except trustmodel.InsufficientCreditsError:
        raise HTTPException(status_code=402, detail="Insufficient credits")
```

### Django

```python
# views.py
from django.http import JsonResponse
import trustmodel

def evaluate_model(request):
    client = trustmodel.TrustModelClient(api_key=settings.TRUSTMODEL_API_KEY)

    evaluation = client.evaluations.create(
        model_identifier=request.POST["model"],
        vendor_identifier=request.POST["vendor"]
    )

    return JsonResponse({
        "evaluation_id": evaluation.id,
        "status": evaluation.status
    })
```

### Flask

```python
from flask import Flask, request, jsonify
import trustmodel

app = Flask(__name__)
client = trustmodel.TrustModelClient(api_key="tm-your-key")

@app.route("/evaluate", methods=["POST"])
def evaluate():
    data = request.get_json()

    evaluation = client.evaluations.create(
        model_identifier=data["model"],
        vendor_identifier=data["vendor"]
    )

    return jsonify({
        "evaluation_id": evaluation.id,
        "status": evaluation.status
    })
```

## Requirements

- Python 3.9 or higher
- `requests` >= 2.25.0
- `pydantic` >= 2.0.0

## Support

- 📚 [Documentation](https://docs.trustmodel.ai/sdk/python)
- 🐛 [Issues](https://github.com/trustmodel/trustmodel-python-sdk/issues)
- 💬 [Support](mailto:support@trustmodel.ai)

## License

This project is licensed under a proprietary license - see the [LICENSE](LICENSE) file for details.

**Important**: This SDK is provided exclusively for use with TrustModel's official API services. Modification, redistribution, or reverse engineering is prohibited.