Metadata-Version: 2.4
Name: cat-cafe-client
Version: 0.10.0
Summary: Python client for CAT Cafe - Continuous Alignment Testing platform for LLM observability
Author-email: Kurtis Seebaldt <kurtisseebaldt@artium.ai>
Requires-Python: >=3.10
Requires-Dist: httpx>=0.24.0
Requires-Dist: typing-extensions>=4.0.0
Description-Content-Type: text/markdown

# CAT Cafe SDK

Python SDK for CAT Cafe - Continuous Alignment Testing platform for LLM observability.

## Installation

```bash
pip install cat-cafe-client
```

> **Compatibility:** the client version tracks the CAT Cafe server API it
> targets. `cat-cafe-client` 0.10.0 targets the cat-cafe server 0.10.0 API.

## Quick Start

```python
from cat.cafe.client import CATCafeClient, DatasetImport, DatasetExample

# Initialize the client
client = CATCafeClient(base_url="http://localhost:8000")

# Create a dataset with examples in one call
dataset = DatasetImport(
    name="My Test Dataset",
    description="Sample dataset for testing",
    examples=[
        DatasetExample(
            input={"messages": [{"role": "user", "content": "What's the weather?"}]},
            expected={"messages": [{"role": "assistant", "content": "Weather info"}]},
            metadata={"tags": ["weather"]},
        )
    ],
)
result = client.import_dataset(dataset)
dataset_id = result["dataset"]["id"]
```

From here, run your AI system against the dataset and stream results to an
experiment -- see ["Manual Experiment Control"](#manual-experiment-control)
below for the full flow.

This SDK is the core HTTP client for CAT Cafe's dataset, experiment, and
trace-scoring APIs. It intentionally does **not** include a task/evaluator
runner -- for the higher-level workflow of defining a task function,
evaluators, and running them against a dataset, use
[cat-experiments](https://github.com/thisisartium/cat-experiments) instead:

```bash
pip install "cat-experiments[cat-cafe]"

cat-experiments run experiment.py --dataset data.jsonl --storage cat-cafe
```

## Core Classes

### CATCafeClient

The main client for interacting with CAT Cafe:

```python
client = CATCafeClient(base_url="http://localhost:8000")
```

### Dataset Models

- **DatasetImport**: For creating new datasets with examples
- **DatasetExample**: Individual examples in a dataset
- **Dataset**: Structured dataset object returned from API
- **Example**: Individual example object returned from API

### Experiment Models

- **Experiment**: Experiment configuration
- **ExperimentResult**: Results from running experiments

## Key Methods

### Dataset Operations

```python
# Import a complete dataset
result = client.import_dataset(dataset_import)

# Add examples to an existing dataset (single or batch)
example_id = client.add_dataset_example(dataset_id, example)
client.add_dataset_examples(dataset_id, [example_a, example_b])

# Fetch dataset as structured object
dataset = client.fetch_dataset(dataset_id)

# Find dataset by name
dataset = client.fetch_dataset_by_name("My Dataset")

# List all datasets
datasets = client.list_datasets()
```

### Experiment Operations

```python
# Start an experiment tied to a dataset
experiment_id = client.start_experiment(experiment_config)

# Stream runs as you produce them
client.create_run(experiment_id, {...})
client.append_evaluation(experiment_id, run_id, {...})

# Mark it done
client.complete_experiment(experiment_id)

# Compare two experiments on the same dataset
comparison = client.compare_experiments(experiment_a_id, experiment_b_id)
```

## Manual Experiment Control

```python
from cat.cafe.client import CATCafeClient, Experiment

client = CATCafeClient(base_url="http://localhost:8000")

# Create experiment configuration
experiment_config = Experiment(
    name="Manual Experiment",
    description="Step-by-step experiment",
    dataset_id=dataset_id,
    tags=["manual", "testing"],
)

# Start experiment
experiment_id = client.start_experiment(experiment_config)

# Run your tests and stream results
dataset = client.fetch_dataset(dataset_id)

for example in dataset.examples:
    output = call_my_ai_system(example.input)  # however you invoke your system under test

    client.create_run(
        experiment_id,
        {
            "run_id": f"run-{example.id}",
            "example_id": example.id,
            "input_data": {"input": example.input},
            "output": dict(example.expected) if isinstance(example.expected, dict) else example.expected,
            "actual_output": output,
        },
    )
    client.append_evaluation(
        experiment_id,
        f"run-{example.id}",
        {"evaluator_name": "manual_score", "score": 0.8},
    )

# Complete experiment
client.complete_experiment(experiment_id, {"total_examples": len(dataset.examples)})
```

## Requirements

- Python 3.10+
- httpx
- CAT Cafe server running (default: http://localhost:8000)

## License

Apache License 2.0 — see the [LICENSE](https://github.com/thisisartium/cat-cafe/blob/main/LICENSE) file in the cat-cafe repository for details.
