Metadata-Version: 2.4
Name: prodloop-observability-sdk
Version: 0.1.5
Summary: Python SDK for evaluating AI voice bot calls via Prodloop APIs.
Project-URL: Homepage, https://prodloop.com
Project-URL: Documentation, https://observability-sdk-docs.pages.dev/
Project-URL: Repository, https://github.com/prodloop/prodloop-observability-sdk
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: litellm
Requires-Dist: requests>=2.31.0
Provides-Extra: docs
Requires-Dist: mkdocs>=1.6.0; extra == "docs"
Requires-Dist: mkdocs-material>=9.5.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.25.0; extra == "docs"

# Prodloop Observability SDK

Python SDK to evaluate AI voice bot calls through the Prodloop evaluation service.

## Install

```bash
pip install prodloop-observability-sdk
```

## Quickstart

```python
from prodloop import ProdloopClient, EvaluationParameter

client = ProdloopClient(api_key="sk_live_...")

result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[
        EvaluationParameter.E2E_RESPONSE_TIME,
        EvaluationParameter.HALLUCINATION,
    ],
    thresholds={"e2e_response_time_max_ms": 800},
    input_prompt="Bot instructions used during this call...",
)

print(result)
```

## Extraction Validation

To validate extraction quality, pass both `extraction_schema` and `bot_captured_variables`:

```python
result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[EvaluationParameter.EXTRACTION_VARIABLES],
    extraction_schema={"customer_name": "string"},
    bot_captured_variables={"customer_name": "ram"},
)
```

Response includes:

- `extraction_variables`
- `extraction_validation`

## Hallucination Input Requirement

When requesting `hallucination`, pass the bot's original call prompt as `input_prompt`:

```python
result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[EvaluationParameter.HALLUCINATION],
    input_prompt="You are a polite admissions bot. Never invent course details.",
)
```

## Supported Parameters

- `e2e_response_time`
- `turn_by_turn_latency`
- `pause_profile`
- `audio_artifacts`
- `hallucination`
- `extraction_variables`
- `interruption_behavior`

## Parameter Purpose

- `e2e_response_time`: average response latency in milliseconds.
- `turn_by_turn_latency`: per-turn latency series with `turn_index` and `latency_ms`.
- `pause_profile`: deterministic pause summary (`pause_count`, `total_pause_time_ms`, `longest_pause_ms`).
- `audio_artifacts`: deterministic audio-signal checks (`clipping_ratio`, `dc_offset`, clipping/DC flags).
- `hallucination`: whether the bot produced fabricated or incorrect claims.
- `extraction_variables`: structured variable extraction from call audio.
- `interruption_behavior`: whether the bot handled interruptions gracefully.

Deterministic parameters are computed directly from the audio signal:
`e2e_response_time`, `turn_by_turn_latency`, `pause_profile`, `audio_artifacts`.

## Authentication

Pass your Prodloop API key in the SDK constructor.  
The SDK sends it as a `Bearer` token in the `Authorization` header.

## Errors

- `ValidationError`: invalid local inputs (file path, parameters, etc.)
- `APIError`: backend/API-level failures (`status_code`, `message`)

## Documentation Site

Live documentation: https://observability-sdk-docs.pages.dev/

## Prompt Simulation

The SDK can test a bot prompt without an audio file by simulating a text conversation between a tester LLM and your bot.

There are two modes:

- `self_simulation`: Prodloop backend runs the tester and bot conversation. You select the bot model route, but Prodloop-owned backend credentials are used. No bot credentials are sent from your code.
- `user_orchestrated`: Prodloop backend runs the tester and grader. Your SDK process runs the bot locally with your own credentials and sends only bot replies/latency back to Prodloop.

Simulation currently accepts exactly one parameter per request. To test multiple parameters, start one simulation per parameter. `max_turns` is configurable from `1` to `10`.

Discover currently enabled simulation parameters at runtime:

```python
params = client.get_simulation_parameters()
print(params["parameters"])  # [{"key": "hallucination"}, ...]
```

The response intentionally exposes only public parameter keys. Whether a parameter is LLM-judged or deterministic is backend implementation detail.

For prompt simulation, `turn_by_turn_latency` is the only timing parameter exposed. It is also emitted in every simulation result automatically at no extra LLM cost, even when you select another parameter such as `hallucination`. Older audio-evaluation parameters like `e2e_response_time` and `pause_profile` remain relevant for call-audio evaluation, but they are not separate prompt-simulation parameters.

### Self Simulation

Use the generic LiteLLM connector. The currently supported backend model routes are `vertex_ai/gemini-2.5-pro` for Gemini on Vertex AI and `azure/<deployment-name>` for Azure OpenAI deployments.

```python
import os
import time
from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins

client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])

start = client.simulate_prompt(
    simulation_mode=SimulationMode.SELF_SIMULATION,
    prompt="You are a concise support bot. Do not invent policy details.",
    parameters=[EvaluationParameter.HALLUCINATION],
    bot_llm=plugins.LiteLLM(
        model="vertex_ai/gemini-2.5-pro",
        temperature=0.2,
        max_tokens=512,
    ),
    max_turns=5,
    scenario="A customer reports a delayed order and asks about compensation.",
)

chat_id = start["chat_id"]
print("chat_id:", chat_id)

while True:
    result = client.get_simulation(chat_id)
    print(result["status"])
    if result["status"] in {"completed", "failed"}:
        print(result)
        break
    time.sleep(2)
```

Backend provider configuration is required for `self_simulation`. Examples:

- Vertex AI: `VERTEX_PROJECT`, `VERTEX_LOCATION`, and the deployed service account permissions.
- Azure OpenAI: `AZURE_API_BASE`, `AZURE_API_VERSION`, and `AZURE_API_KEY` or `AZURE_AD_TOKEN`. You can also derive the base URL from a resource name in your own deployment setup.

If the backend is not configured for the selected route, the API returns a safe user-facing error asking you to configure provider credentials or use `user_orchestrated` mode.

### User Orchestrated Simulation

Use `user_orchestrated` when the bot must run in your own process with your own credentials. Prodloop never receives your bot provider credentials.

```python
import os
from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins

bot = plugins.LiteLLMBot(
    model="azure/<deployment-name>",
    system_prompt="You are a concise support bot. Do not invent policy details.",
    options={"temperature": 0.2, "max_tokens": 256},
)

client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])

result = client.simulate_prompt(
    simulation_mode=SimulationMode.USER_ORCHESTRATED,
    prompt="You are a concise support bot. Do not invent policy details.",
    parameters=[EvaluationParameter.HALLUCINATION],
    max_turns=5,
    scenario="A customer pressures the bot to confirm a fake policy.",
    bot_turn_handler=bot,
)

print(result)
```

For `user_orchestrated`, configure bot credentials locally for either Vertex AI or Azure OpenAI. For Vertex AI, use ADC/service-account configuration and project/location. For Azure OpenAI, use your Azure endpoint, API version, and API key/token locally. The SDK sends only bot response text and bot LLM processing latency to Prodloop.

### Result Shape

Simulation responses include:

- `chat_id`: stable simulation identifier.
- `status`: `running`, `completed`, or `failed`.
- `turns`: tester message, bot response, `turn_id`, grading result, and per-turn timing.
- `final_result`: overall pass/fail, per-parameter result, summary, and prompt patch suggestions when a test fails.

If a parameter fails, `final_result.parameter_results[*].prompt_patch_lines` and `prompt_patch_location` provide copy-paste prompt guidance.



## Credit Balance

Check the credits left on the API key without running an evaluation:

```python
balance = client.get_credit_balance()
print(balance["credits_remaining"])

# Alias with the same response
balance = client.get_credits_remaining()
```

This endpoint authenticates the API key but does not debit credits.
