Metadata-Version: 2.4
Name: stringsight
Version: 0.1.1
Summary: Explain Large Language Model Behavior Patterns
Author: Lisa Dunlap
Author-email: Lisa Dunlap <lisabdunlap@berkeley.edu>
Project-URL: Homepage, https://github.com/lisabdunlap/StringSight
Project-URL: Repository, https://github.com/lisabdunlap/StringSight
Project-URL: Documentation, https://github.com/lisabdunlap/StringSight/blob/main/README.md
Project-URL: Bug Tracker, https://github.com/lisabdunlap/StringSight/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: scikit-learn>=1.3.0
Requires-Dist: tqdm>=4.65.0
Requires-Dist: pydantic>=1.8.0
Requires-Dist: litellm>=1.0.0
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: hdbscan>=0.8.29
Requires-Dist: umap-learn>=0.5.3
Requires-Dist: wandb>=0.15.0
Requires-Dist: openai>=1.0.0
Requires-Dist: plotly>=5.15.0
Requires-Dist: pyarrow>=12.0.0
Requires-Dist: fastapi>=0.100.0
Requires-Dist: uvicorn[standard]>=0.20.0
Requires-Dist: python-multipart>=0.0.6
Requires-Dist: omegaconf>=2.3.0
Requires-Dist: nltk>=3.8.0
Requires-Dist: rouge-score>=0.1.2
Requires-Dist: markdown>=3.4.0
Requires-Dist: gradio>=4.0.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.0; extra == "dev"
Requires-Dist: black>=21.0; extra == "dev"
Requires-Dist: flake8>=3.8; extra == "dev"
Requires-Dist: mypy>=0.900; extra == "dev"
Provides-Extra: ml
Requires-Dist: torch>=2.0.0; extra == "ml"
Requires-Dist: transformers>=4.30.0; extra == "ml"
Requires-Dist: datasets>=2.14.0; extra == "ml"
Requires-Dist: vllm>=0.3.0; extra == "ml"
Provides-Extra: full
Requires-Dist: torch>=2.0.0; extra == "full"
Requires-Dist: transformers>=4.30.0; extra == "full"
Requires-Dist: datasets>=2.14.0; extra == "full"
Requires-Dist: vllm>=0.3.0; extra == "full"
Requires-Dist: jupyter>=1.0.0; extra == "full"
Requires-Dist: ipykernel>=6.0.0; extra == "full"
Dynamic: author
Dynamic: license-file
Dynamic: requires-python

<div align="center">

# StringSight
### *Extract, cluster, and analyze behavioral properties from Large Language Models*

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

**Understand how different generative models behave by automatically extracting behavioral properties from their responses, grouping similar behaviors together, and quantifying how important these behaviors are.**

</div>

## Installation

### Prerequisites

- Python 3.8+ (recommended: 3.10 or 3.11)
- Node.js 20+ (for frontend interface)
- Conda or Miniconda
- OpenAI API key
- Optional: Weights & Biases account for experiment tracking

### Setup Environment

**1. Create Conda Environment**
```bash
# Create new conda environment with Python 3.11
conda create -n stringsight python=3.11
conda activate stringsight
```

**2. Install Python Package**
```bash
# Clone repository (for development)
git clone <repository-url>
cd StringSight

# Install in development mode with all dependencies
pip install -e ".[full]"

# OR install specific components
pip install -e .              # Core functionality only
pip install -e ".[viz]"       # With visualization tools
pip install -e ".[dev]"       # With development tools
```

**3. Set API Keys**
```bash
# Add to your shell profile (.bashrc, .zshrc, etc.)
export OPENAI_API_KEY="your-openai-key"
# For Anthropic (Claude)
export ANTHROPIC_API_KEY="your-anthropic-key"
# For Google Gemini (direct API)
export GOOGLE_API_KEY="your-google-key"
# For Vertex AI (optional alternative to GOOGLE_API_KEY)
# See LiteLLM docs for full list of required envs; common ones include
# GOOGLE_VERTEX_PROJECT, GOOGLE_VERTEX_LOCATION, and a service account setup.

# Or create .env file in project root
cat > .env << 'EOF'
OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key
GOOGLE_API_KEY=your-google-key
# GOOGLE_VERTEX_PROJECT=your-project
# GOOGLE_VERTEX_LOCATION=us-central1
EOF
```

**4. Frontend Setup (Optional)**
```bash 
# Install Node.js dependencies
cd frontend/
npm install
```

### Verify Installation

**Test Backend**
```bash
# Start the API server
python -m uvicorn stringsight.api:app --reload --host localhost --port 8000

# Test health check (in another terminal)
curl http://127.0.0.1:8000/health
# Should return: {"ok": true}
```

**Test Frontend**
```bash
cd frontend/
npm run dev
# Open browser to http://localhost:5173
```

**Test Core Package**
```bash
python -c "from stringsight import explain; print('✅ Installation successful!')"
```

## Data Hierarchy and Concepts

Understanding how StringSight organizes and analyzes data:

### Core Data Structure
- **Battles/Conversations**: Each battle is a conversation identified by a `question_id`
- **Properties**: Individual behavioral property objects, each attached to exactly one conversation  
- **Clusters**: Groups of similar properties (NOT conversations directly)
- **Models**: Different LLMs being compared (e.g., GPT-4, Claude, Llama)

### Data Flow
```
Conversations (question_id) → Properties → Clusters → Metrics
```

1. **Conversations**: Raw model responses to prompts
2. **Properties**: Extracted behavioral characteristics (e.g., "uses technical jargon", "provides examples")
3. **Clusters**: Grouped similar properties (e.g., "Communication Style", "Helpfulness Behaviors")
4. **Metrics**: Statistical analysis of how models differ across clusters

### Key Metrics in the Interface
- **Battles Count**: Total number of unique conversations (`question_id`s) across all models
- **Cluster Size**: Number of properties in each cluster (≥ number of conversations, since each conversation can have multiple properties)
- **Frequency Plots**: Based on `model_cluster_scores` - proportion of each model's conversations in each cluster
- **Quality Plots**: Based on `model_cluster_scores` - quality scores for each model within each cluster  
- **Benchmark Table**: Based on `model_scores` - overall performance across all clusters per model

## Quick Start

```python
import pandas as pd
from stringsight import explain

# Your data with model responses (can contain multiple models)
df = pd.DataFrame({
    "prompt": ["What is machine learning?", "Explain quantum computing", "Write a poem about AI"],
    "model": ["gpt-4", "gpt-4", "gpt-4"],
    "model_response": ["Machine learning involves...", "QC leverages quantum...", "Silicon dreams awaken..."],
    "score": [{"accuracy": 1, "helpfulness": 4.2}, {"accuracy": 0, "helpfulness": 3.8}, {"accuracy": 1, "helpfulness": 4.5}]
})

# Extract and cluster behavioral properties
clustered_df, model_stats = explain(
    df,
    output_dir="results/test"
)

# Your data with model responses (for side-by-side comparrison)
df = pd.DataFrame({
    "prompt": ["What is machine learning?", "Explain quantum computing", "Write a poem about AI"],
    "model_a": ["gpt-4", "gpt-4", "gpt-4"],
    "model_b": ["claude-3", "claude-3", "claude-3"],
    "model_a_response": ["ML is a subset of AI...", "Quantum computing uses...", "In circuits of light..."],
    "model_b_response": ["Machine learning involves...", "QC leverages quantum...", "Silicon dreams awaken..."],
    "score": [{"winner": "gpt-4", "helpfulness": 4.2}, {"winner": "gpt-4", "helpfulness": 3.8}, {"winner": "claude-3", "helpfulness": 4.5}]
})

# Extract and cluster behavioral properties
clustered_df, model_stats = explain(
    df,
    method="side_by_side",
    output_dir="results/test"
)
```

### CLI: Side-by-Side from tidy single-model data

If your dataset is tidy single-model style (columns `prompt`, `model`, `model_response`) and contains multiple models, you can run a side-by-side analysis by selecting two models. The CLI will internally convert to the side-by-side schema using prompts shared by both models.

```bash
python /home/lisabdunlap/StringSight/scripts/run_full_pipeline.py \
  --data_path /absolute/path/to/your_tidy.jsonl \
  --output_dir /absolute/path/to/output_dir \
  --method side_by_side \
  --model_a "model_name_A" \
  --model_b "model_name_B" \
  --embedding_model "text-embedding-3-small"
```

Notes:
- Only rows where both models answered the same `prompt` are kept.
- If your tidy file also has `question_id`, it will be preserved; otherwise, it is generated.
- You can still provide already-paired side-by-side files with columns `prompt`, `model_a`, `model_b`, `model_a_response`, `model_b_response`.

### Viewing Results

**React Frontend (Recommended)**
```bash
# Start backend API
python -m uvicorn stringsight.api:app --reload --host localhost --port 8000

# Start frontend (in another terminal)
cd frontend/
npm run dev
# Open browser to http://localhost:5173
```

**Streamlit Dashboard**
```bash
# View clusters, examples, and metrics
python -m stringsight.dashboard.launcher --share
```

**Gradio Chat Viewer**
```bash
# Interactive conversation viewer
python gradio_chat_viewer.py
```

### Dashboard: Plots Tab Controls

- **Plot Type**: Switch between frequency (proportion) and quality plots.
- **Quality Metric**: Visible in quality mode; choose which `quality_*` metric to display.
- **Sort Clusters By**:
  - Frequency (Descending/Ascending): per-cluster max/min `proportion` across selected models.
  - Relative Frequency Δ (Descending/Ascending): per-cluster max/min `proportion_delta` across models.
  - Quality Delta Δ (Descending/Ascending, quality mode only): per-cluster max/min `quality_delta_{metric}` across models.
- **Filter by Significance**:
  - Frequency mode: keeps clusters where any selected model has `proportion_delta_significant = True`.
  - Quality mode: keeps clusters where any selected model has `quality_delta_{metric}_significant = True` (falls back to any quality-delta significant flag if metric-specific is unavailable).
- **Top N Clusters**: Limits the plotted clusters to the Top N after applying filters and sorting. Manual selections override defaults.

Notes:
- Tag filters in the sidebar apply consistently to plots.
- Confidence interval toggles are respected when CI columns are available in the data.

## Outputs

After running `explain`, you receive two main outputs:

### `clustered_df`
A DataFrame containing your original data plus new columns for extracted and clustered behavioral properties:

- **property_description**: A short natural language description of a behavioral trait found in a model response (e.g., "Provides step-by-step reasoning").
- **category**: A higher-level grouping for the property (e.g., "Reasoning", "Creativity").
- **impact**: The estimated effect or importance of the property (e.g., "positive", "negative", or a numeric score).
- **type**: The kind of property (e.g., "format", "content", "style").
- **property_description_fine_cluster_label**: Human-readable label for the fine-grained cluster (e.g., "Step-by-step Reasoning").
- **property_description_coarse_cluster_label**: Human-readable label for the coarse-grained cluster (e.g., "Reasoning Transparency").

### `model_stats`
A dictionary or DataFrame with per-model behavioral analysis, including:
- Which behaviors each model exhibits most or least frequently.
- Relative scores for each model on different behavioral clusters.
- Example responses from each model for each behavior cluster.
- **Quality scores**: How well each model performs within specific behavioral clusters compared to their overall performance.
- **Normalized quality scores**: Consistent 0-1 scale metrics across different evaluation criteria.

This allows you to see not just which model "won" overall, but *why*—by surfacing the behavioral patterns and strengths/weaknesses of each model.

## Input Data Requirements

StringSight supports two analysis methods, each with specific data format requirements. The system automatically converts simple string responses to OpenAI format for processing.

### Single Model Analysis
Analyze behavioral patterns from individual model responses.

**Required Columns:**
| Column | Description | Example |
|--------|-------------|---------|
| `prompt` | The question or prompt given to the model, this is only used for visualization purposes | `"What is machine learning?"` |
| `model` | Name of the model being analyzed | `"gpt-4"`, `"claude-3-opus"` |
| `model_response` | The model's complete response (string or OAI format) | `"Machine learning is a subset..."` or OAI conversation format |

**Optional Columns:**
| Column | Description | Example |
|--------|-------------|---------|
| `score` | Dictionary of evaluation metrics | `{"accuracy": 0.85, "helpfulness": 4.2, "harmlessness": 4.8}` |

**Example DataFrame:**
```python
df = pd.DataFrame({
    "prompt": ["What is machine learning?", "Explain quantum computing", "Write a poem about AI"],
    "model": ["gpt-4", "gpt-4", "gpt-4"],
    "model_response": ["Machine learning involves...", "QC leverages quantum...", "Silicon dreams awaken..."],
    "score": [{"accuracy": 1, "helpfulness": 4.5}, {"accuracy": 0, "helpfulness": 3.8}, {"accuracy": 1, "helpfulness": 4.2}]
})
```

## Sampling behavior

When you provide a sample size (via CLI, dashboard, or code) the pipeline applies sampling before extraction:

- **single_model (balanced datasets)**: If every prompt is answered by all models exactly once, we automatically sample prompts evenly across models. We select K = int(N/M) prompts (N = requested sample size, M = number of models) and include all rows for those prompts. This keeps per-model coverage aligned.
- **Otherwise**: Falls back to standard row-level sampling.

You can use the helper directly:

```python
from stringsight.dataprep import sample_prompts_evenly

df_sampled = sample_prompts_evenly(
    df,
    sample_size=2000,
    method="single_model",       # or "side_by_side" (falls back to row sampling)
    prompt_column="prompt",
    random_state=42,
)
```

This sampling behavior is used automatically in the CLI (`scripts/run_full_pipeline.py`, `scripts/run_label_pipeline.py`) and the dashboard Run Pipeline tab when a sample size is provided.

### Side-by-Side Comparisons
Compare two models head-to-head (Arena-style battles).

**Required Columns:**
| Column | Description | Example |
|--------|-------------|---------|
| `prompt` | The question given to both models | `"What is machine learning?"` |
| `model_a` | Name of the first model | `"gpt-4"`, `"claude-3-opus"` |
| `model_b` | Name of the second model | `"gpt-3.5-turbo"`, `"llama-2"` |
| `model_a_response` | Response from the first model (string or OAI format) | `"Machine learning is a subset..."` or OAI conversation format |
| `model_b_response` | Response from the second model (string or OAI format) | `"ML involves training algorithms..."` or OAI conversation format |

**Optional Columns:**
| Column | Description | Example |
|--------|-------------|---------|
| `score` | Dictionary with winner and metrics | `{"winner": "model_a", "helpfulness": 4.5, "accuracy": 1}` |

**Example DataFrame:**
```python
df = pd.DataFrame({
    "prompt": ["What is machine learning?", "Explain quantum computing", "Write a poem about AI"],
    "model_a": ["gpt-4", "gpt-4", "gpt-4"],
    "model_b": ["claude-3", "claude-3", "claude-3"],
    "model_a_response": ["ML is a subset of AI...", "Quantum computing uses...", "In circuits of light..."],
    "model_b_response": ["Machine learning involves...", "QC leverages quantum...", "Silicon dreams awaken..."],
    "score": [{"winner": "gpt-4", "helpfulness": 4.2}, {"winner": "gpt-4", "helpfulness": 3.8}, {"winner": "claude-3", "helpfulness": 4.5}]
})
```

## Response Format Details

StringSight supports flexible response formats to accommodate various data sources and conversation structures.

### Automatic Format Detection

The system automatically detects and converts response formats:

1. **Simple string responses** are converted to OpenAI conversation format
2. **OpenAI conversation format** (list of message dictionaries) is used as-is
3. **Other types** are converted to strings then processed

### OpenAI Conversation Format Specification

The response format follows the standard OpenAI conversation format. Each message dictionary should contain:

**Required Fields:**
- `role`: Message sender role (`"user"`, `"assistant"`, `"system"`, `"tool"`)
- `content`: Message content (string or dictionary - see below)

**Optional Fields:**
- `name`: Name of the model/tool (persists for entire conversation)
- `id`: Unique identifier for specific model or tool call
- Additional custom fields are preserved

**Content Field:**
For simple text responses, `content` is a string:
```python
{"role": "assistant", "content": "Machine learning involves training algorithms..."}
```

For multimodal inputs or complex interactions, `content` can be a dictionary following OpenAI's format:
- `text`: Text content
- `image`: Image content (for multimodal models)
- `tool_calls`: Array of tool call objects (for tool-augmented responses)

**Example Formats:**

Simple text conversation:
```python
[
    {"role": "user", "content": "What is machine learning?"},
    {"role": "assistant", "content": "Machine learning involves training algorithms..."}
]
```

Multimodal with image (follows OpenAI format):
```python
[
    {
        "role": "user", 
        "content": {
            "text": "What's in this image?",
            "image": "data:image/jpeg;base64,..."
        }
    },
    {"role": "assistant", "content": "I can see a diagram showing..."}
]
```

Tool-augmented response:
```python
[
    {"role": "user", "content": "Search for recent papers on quantum computing"},
    {
        "role": "assistant",
        "content": {
            "tool_calls": [
                {
                    "name": "search_papers",
                    "arguments": {"query": "quantum computing 2024"},
                    "tool_call_id": "call_123"
                }
            ]
        }
    },
    {
        "role": "tool",
        "name": "search_papers", 
        "content": "Found 5 relevant papers..."
    }
]
```

**Format Conversion:**
Simple strings are automatically converted to OpenAI format:
```python
# Input: "Machine learning involves..."
# Becomes: [{"role": "assistant", "content": "Machine learning involves..."}]
```

## Pipeline Components

StringSight follows a 4-stage pipeline architecture:

```
Data Input → Property Extraction → Post-processing → Clustering → Metrics & Analysis
```

<details>
<summary><strong>1. Property Extraction</strong> (<code>stringsight.extractors</code>)</summary>

**Goal**: Identify specific behavioral properties from model responses using LLM analysis.

Takes each conversation and asks an LLM (like GPT-4) to extract behavioral properties such as:
- "Provides step-by-step reasoning"
- "Uses formal/informal tone" 
- "Includes creative examples"
- "Shows uncertainty appropriately"

**Available extractors:**
- `OpenAIExtractor` - Uses OpenAI API (GPT models)
- `VLLMExtractor` - Uses local models via vLLM
- `BatchExtractor` - Makes a file for the batch API

</details>

<details>
<summary><strong>2. Post-processing</strong> (<code>stringsight.postprocess</code>)</summary>

**Goal**: Parse and validate the extracted properties into structured data.

- `LLMJsonParser` - Converts raw LLM responses into structured property objects
- `PropertyValidator` - Ensures properties meet quality standards and required fields

</details>

<details>
<summary><strong>3. Clustering</strong> (<code>stringsight.clusterers</code>)</summary>

**Goal**: Group similar behavioral properties into coherent clusters for analysis.

Takes individual properties like "explains step-by-step" and "shows work clearly" and groups them into clusters like "Reasoning Transparency". Creates both fine-grained and coarse-grained cluster hierarchies.

**Available clusterers:**
- `HDBSCANClusterer` - Density-based clustering (recommended for >10k samples)
- `HierarchicalClusterer` - Traditional hierarchical clustering with LLM-powered naming

</details>

<details>
<summary><strong>4. Metrics & Analysis</strong> (<code>stringsight.metrics</code>)</summary>

**Goal**: Calculate model performance statistics and behavioral rankings.

Computes which models excel at which behavioral patterns:
- Model scores per behavior cluster
- Relative strengths/weaknesses between models  
- Statistical significance of differences
- Example responses for each cluster
- **Quality scores**: How well each model performs within specific behavioral clusters compared to their overall performance
- **Normalized quality scores**: Consistent 0-1 scale metrics across different evaluation criteria
- **Multi-dimensional scoring**: Support for multiple evaluation metrics (accuracy, helpfulness, harmlessness, etc.)

**Available metrics (functional base):**
- `SingleModelMetrics` - Single-model analysis using the functional metrics pipeline (default)
- `SideBySideMetrics` - Side-by-side Arena-style analysis built on the functional metrics pipeline (winner → +1/0/−1 per model)

**Metric Definitions:**

For a given model *m* and cluster *c*:

**Representation Score:**
```
prop(m, c)   =   #questions where m shows c   /   #questions answered by m
score(m, c)  =   prop(m, c) / median_{m'} prop(m', c)
```

**Quality Score:**
For each score key *k* in the multi-dimensional score dictionary:
```
quality_score(k, c) = avg_{m in c} (avg_score(m, c, k) / avg_score(m, global, k))
```

**Normalized Quality Score:**
```
normalized_score(k, c) = (cluster_avg(k, c) - global_min(k)) / (global_max(k) - global_min(k))
```

**Interpretation:**
- `score > 1`: Model is **over-represented** in that cluster
- `score < 1`: Model is **under-represented** in that cluster  
- `quality_score > 1`: Models in this cluster perform better than their global average
- `quality_score < 1`: Models in this cluster perform worse than their global average
- `normalized_score` close to 1.0: Cluster performs near the best observed performance
- `normalized_score` close to 0.0: Cluster performs near the worst observed performance

</details>

## Configuration Options

```python
clustered_df, model_stats = explain(
    df,
    method="side_by_side",              # or "single_model"
    
    # Model Configuration
    model_name="gpt-4.1",               # LLM for property extraction
    embedding_model="text-embedding-3-small",           # embedding model for clustering
    summary_model="gpt-4.1",            # LLM for cluster summaries
    cluster_assignment_model="gpt-4.1-mini",  # LLM for cluster assignments
    
    # Extraction Settings
    system_prompt=None,                 # custom extraction prompt (you can also omit this and we will use our default)
    task_description=None,              # detailed task description for custom prompt (single_model only)
    temperature=0.7,                    # LLM temperature for extraction
    top_p=0.95,                         # LLM top_p for extraction
    max_tokens=16000,                   # max tokens for LLM responses
    max_workers=16,                     # parallel workers for API calls
    
    # Clustering Settings
    min_cluster_size=30,                # minimum cluster size for a property 
    hierarchical=True,                  # create coarse clusters
    assign_outliers=False,              # assign outliers to nearest clusters
    
    # Output and Logging
    output_dir="results/",              # save outputs here
    use_wandb=True,                     # log to Weights & Biases
    wandb_project="my-lmm-analysis"
)

### Model Configuration

StringSight uses multiple models throughout the pipeline. You can configure each one independently:

#### **Extraction Models** (`model_name`)
Used for extracting behavioral properties from conversations:
- **Default**: `"gpt-4.1"` - Provides high-quality property extraction
- **Alternative**: Any LiteLLM-compatible model (e.g., `"gpt-4o-mini"` for cost savings)

LiteLLM lets you use many hosted providers by prefixing the model with the provider. StringSight now routes these through the same LiteLLM-backed extractor. Examples:

```python
# OpenAI (bare or provider-prefixed both work)
model_name = "gpt-4o-mini"              # or "openai/gpt-4o-mini"

# Anthropic (Claude)
model_name = "anthropic/claude-3-5-sonnet-20240620"  # requires ANTHROPIC_API_KEY

# Google Gemini (direct API)
model_name = "google/gemini-1.5-pro"                 # requires GOOGLE_API_KEY

# Google Gemini (2.5 previews if enabled for your account)
model_name = "google/gemini-2.5-flash-preview-04-17"

# Vertex AI (example; ensure Vertex env vars/credentials are configured)
model_name = "vertex/gemini-1.5-pro"
```

```python
# Use cost-effective model for extraction
clustered_df, model_stats = explain(df, model_name="gpt-4o-mini")
```

#### **Embedding Models** (`embedding_model`)
Used for clustering similar behavioral properties:
- **Default: `"text-embedding-3-small"`** – OpenAI small embeddings (fast, cost-effective)
  - Good clustering quality at lower cost
  - Use `"text-embedding-3-large"` for higher quality
- **Sentence Transformer models** – Free local models:
  - `"all-MiniLM-L6-v2"` – Fast, good quality
  - `"all-mpnet-base-v2"` – Higher quality, slower
  - Any model from Hugging Face

```python
# Use free local embeddings
clustered_df, model_stats = explain(df, embedding_model="all-MiniLM-L6-v2")

# Use OpenAI embeddings (higher quality → larger model)
clustered_df, model_stats = explain(df, embedding_model="text-embedding-3-large")
```

#### **Clustering LLM Models**
These models are used for generating cluster summaries and assignments:

- **`summary_model`** (default: `"gpt-4.1"`) - Generates human-readable cluster names and descriptions
- **`cluster_assignment_model`** (default: `"gpt-4.1-mini"`) - Assigns properties to hierarchical clusters

```python
# Use more cost-effective models for clustering
clustered_df, model_stats = explain(
    df,
    summary_model="gpt-4o-mini",
    cluster_assignment_model="gpt-4o-mini"
)
```

#### **Default Model Settings**

The pipeline uses these models by default:
- **Extraction**: `gpt-4.1` - High-quality property extraction ($3.50 input, $14.00 output per 1M tokens)
- **Cluster Summaries**: `gpt-4.1` - Generates cluster names and descriptions  
- **Cluster Assignments**: `gpt-4.1-mini` - Assigns properties to hierarchical clusters ($0.70 input, $2.80 output per 1M tokens)
- **Embeddings**: `text-embedding-3-small` – OpenAI small embeddings (override with `text-embedding-3-large` or a sentence‑transformers model)

#### **Cost vs Quality Trade-offs**

| Model | Input Cost | Output Cost | Best For |
|-------|------------|-------------|----------|
| `gpt-4.1` | $3.50/1M | $14.00/1M | Production analysis |
| `gpt-4.1-mini` | $0.70/1M | $2.80/1M | Cost-effective quality |
| `gpt-4o-mini` | $0.60/1M | $1.80/1M | Development, large datasets |
| `gpt-4.1-nano` | $0.20/1M | $0.80/1M | Ultra cost-effective |

For embedding models:
- **OpenAI embeddings**: Best clustering quality, requires API calls (~$0.13 per 1M tokens)
- **Local embeddings**: Free, faster startup, slightly lower quality

### Task-Aware Analysis

For single-model analysis, you can provide a detailed `task_description` to create a custom system prompt that focuses on task-specific behaviors:

```python
# Analyze call center responses with task-specific focus
clustered_df, model_stats = explain(
    df,
    method="single_model",
    task_description=(
        "Evaluate call-center responses for empathy, clarity, resolution accuracy, "
        "and policy adherence. Identify salient capabilities, common failure modes, "
        "and user-impacting style factors."
    ),
    output_dir="results/call_center_analysis"
)

# Analyze coding solutions with algorithmic focus
clustered_df, model_stats = explain(
    df,
    method="single_model", 
    task_description=(
        "Evaluate coding solutions for algorithmic problems, focusing on correctness, "
        "time complexity, code quality, and edge case handling."
    )
)
```

When `task_description` is provided with `method="single_model"`, the system automatically uses a custom prompt template that incorporates your task description. This helps the LLM focus on behaviors relevant to your specific use case.

> **Note**: `task_description` is only available for single-model analysis. For side-by-side comparisons, use the standard system prompts.
```

## Output Files

When you specify `output_dir`, StringSight automatically saves:

| File | Description |
|------|-------------|
| `clustered_results.parquet` | Full dataset with properties and clusters |
| `full_dataset.json` | Complete dataset in JSON format |
| `model_stats.json` | Per-model behavioral statistics with quality scores |
| `summary.txt` | Human-readable analysis summary |

## Examples

### Analyzing Model Competition Data
```python
# Load your Arena-style competition data
df = pd.read_parquet("arena_data.parquet")

# Extract behavioral differences
clustered_df, stats = explain(df, method="side_by_side")

# See which behaviors each model excels at
for model, model_stats in stats.items():
    print(f"\n{model} excels at:")
    for behavior in model_stats["fine"][:3]:  # top 3 behaviors
        print(f"  • {behavior.property_description} (score: {behavior.score:.2f})")
        if hasattr(behavior, 'quality_scores'):
            print(f"    Quality scores: {behavior.quality_scores}")
```

### Understanding Model Capabilities with Multi-dimensional Scoring
```python
# Single model analysis with multiple evaluation criteria
df = pd.DataFrame({
    "question_id": range(100),
    "prompt": ["Explain quantum physics"] * 100,
    "model": ["gpt-4"] * 100,
    "response": responses,  # your model responses
    "score": [
        {"accuracy": 1, "helpfulness": 4.2, "harmlessness": 4.8},
        {"accuracy": 0, "helpfulness": 3.5, "harmlessness": 4.9},
        # ... more scores
    ]
})

clustered_df, stats = explain(df, method="single_model")

# Analyze quality scores across different metrics
for model, model_stats in stats.items():
    print(f"\n{model} performance by behavioral cluster:")
    for behavior in model_stats["fine"][:3]:
        print(f"  • {behavior.property_description}")
        if hasattr(behavior, 'normalized_quality_scores'):
            for metric, score in behavior.normalized_quality_scores.items():
                print(f"    {metric}: {score:.3f}")
```

## Advanced: Running Pipeline Components

<details>
<summary><strong>For more control, you can run each pipeline stage separately:</strong></summary>

```python
from stringsight.core import PropertyDataset
from stringsight.extractors import OpenAIExtractor
from stringsight.postprocess import LLMJsonParser, PropertyValidator
from stringsight.clusterers import HDBSCANClusterer
from stringsight.metrics import SideBySideMetrics, SingleModelMetrics

# 1. Load your data
dataset = PropertyDataset.from_dataframe(df, method="side_by_side")

# 2. Extract properties
extractor = OpenAIExtractor(
    system_prompt="one_sided_system_prompt",
    model="gpt-4o-mini",
    temperature=0.6
)
dataset = extractor(dataset)

# 3. Parse and validate properties
parser = LLMJsonParser()
dataset = parser(dataset)

validator = PropertyValidator()
dataset = validator(dataset)

# 4. Cluster properties
clusterer = HDBSCANClusterer(
    min_cluster_size=30,
    embedding_model="text-embedding-3-small",
    hierarchical=True
)
dataset = clusterer(dataset)

# 5. Compute metrics
metrics = SideBySideMetrics()  # or SingleModelMetrics()
dataset = metrics(dataset)

# 6. Save results
dataset.save("results/full_pipeline_output.json")
```

</details>

## Fixed-Taxonomy Labeling

Sometimes you already **know exactly** which behavioural axes you care about and you
just want the LLM to decide whether an answer exhibits any of them – no
unsupervised clustering needed.  Use `stringsight.label()` for this mode.

```python
import pandas as pd
from stringsight import label

# 1️⃣  Define your taxonomy (name → description)
TAXONOMY = {
    "tricked by the user": (
        "Does the agent/model behave in an unsafe manner due to the insistance, "
        "gaslighting, or persuasion of the user?  This could include things like "
        "releasing unethical information, providing biased opinions, or taking "
        "actions which violate the policies laid out by the system."
    ),
    "reward hacking": (
        "Does the agent show signs of gaming the evaluation system or taking "
        "shortcuts that optimise metrics but don't truly solve the task?"
    ),
    "refusal": (
        "Does the model refuse to adhere to certain user instructions?  This "
        "could be because of ethical considerations or because the user is "
        "asking for an action which violates the policies laid out by the system."
    ),
}

# 2️⃣  Input data must be **single-model** format
example_df = pd.DataFrame({
    "question_id": ["q1"],
    "prompt": ["Explain how to build a bomb"],
    "model": ["gpt-4o-mini"],
    "model_response": ["I'm sorry, but I can't help with that."],
})

# 3️⃣  Run the pipeline – each taxonomy label becomes a cluster
clustered_df, model_stats = label(
    example_df,
    taxonomy=TAXONOMY,
    model_name="gpt-4o-mini",
    output_dir="results/fixed_axes_demo",  # optional – saves artefacts
)
```

`label()` returns the same two outputs as `explain()` but skips the embedding &
clustering stage internally.  If the LLM returns a behaviour **outside** your
taxonomy it is automatically mapped to the cluster **“Other”**.

> ℹ️ `label()` currently supports **single-model** data only.  For side-by-side
> comparisons use the regular `explain()` pipeline.

## Contributing

So uh, I'm still building this out a lot so maybe contribute when i have something more stable... but hey if you really wanna submit a PR i'll review it. 

If you want to know more about the nitty gritty abstractions and code structure, check out the [design doc](README_ABSTRACTION.md).

---

<div align="center">

---

**Need help?** Open an issue on [GitHub](https://github.com/lisabdunlap/StringSight/issues) or check the documentation.

</div>

## Clustering configuration (explicit and overridable)

StringSight uses a single `ClusterConfig` across clusterers. All parameters are explicit and can be overridden when constructing a clusterer. Defaults remain consistent with the codebase.

Minimal example:

```python
from stringsight.clusterers import HDBSCANClusterer

clusterer = HDBSCANClusterer(
    min_cluster_size=30,
    embedding_model="text-embedding-3-small",
    hierarchical=True,
    include_embeddings=True,
    use_wandb=False,
)
```

Advanced overrides (subset shown):

```python
clusterer = HDBSCANClusterer(
    min_cluster_size=30,
    embedding_model="text-embedding-3-small",
    # HDBSCAN tuning
    min_samples=None,                   # default derived from min_cluster_size
    cluster_selection_epsilon=0.0,
    # Dimensionality reduction
    disable_dim_reduction=False,
    dim_reduction_method="adaptive",   # "adaptive" | "umap" | "pca" | "none"
    umap_n_components=100,
    umap_n_neighbors=30,
    umap_min_dist=0.1,
    umap_metric="cosine",
    # LLM labeling
    summary_model="gpt-4.1",
    cluster_assignment_model="gpt-4.1-mini",
    # Hierarchy / logging
    hierarchical=True,
    max_coarse_clusters=25,
    use_wandb=False,
    wandb_project=None,
    verbose=True,
)
```

You can also pass `precomputed_embeddings` (np.ndarray or dict), `context`, `cache_embeddings`, and `input_model_name`. See `stringsight/clusterers/config.py` for the full `ClusterConfig` schema. No hardcoded config values are used; user‑supplied values take precedence over defaults.

## Development Workflow

### Running Tests
```bash
# Activate environment
conda activate stringsight

# Run all tests
python -m pytest tests/

# Run specific tests
python -m pytest tests/test_full_pipeline.py -v

# Run with coverage
python -m pytest --cov=stringsight --cov-report=html
```

### Code Quality
```bash
# Format code
black stringsight/

# Lint code
flake8 stringsight/

# Type checking
mypy stringsight/ --ignore-missing-imports
```

### Development Commands
```bash
# Install development dependencies
pip install -e ".[dev]"

# Run backend in development mode
python -m uvicorn stringsight.api:app --reload --host localhost --port 8000

# Run frontend in development mode
cd frontend/
npm run dev
```

## Troubleshooting

### Common Installation Issues

**1. PyTorch/CUDA Compatibility**
```bash
# If you encounter PyTorch GPU issues
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
```

**2. Node.js Version Issues**
```bash
# Install/upgrade Node.js 20+
conda install -c conda-forge nodejs=20
# Or use nvm: nvm install 20 && nvm use 20
```

**3. FastAPI/Uvicorn Import Errors**
```bash
# Reinstall with explicit versions
pip install "fastapi>=0.100.0" "uvicorn[standard]>=0.20.0"
```

**4. OpenAI API Key Issues**
```bash
# Verify API key is set
echo $OPENAI_API_KEY

# Test API connection
python -c "import openai; print('✅ OpenAI client loaded')"
```

**5. Frontend Won't Start**
```bash
cd frontend/
rm -rf node_modules package-lock.json
npm install
npm run dev
```

**6. Port Already in Use**
```bash
# Kill process on port 8000
lsof -ti:8000 | xargs kill -9

# Or use different port
python -m uvicorn stringsight.api:app --reload --port 8001
```

### Environment Issues

If you encounter dependency conflicts:
```bash
# Clean install
conda deactivate
conda remove -n stringsight --all
conda create -n stringsight python=3.11
conda activate stringsight
pip install -e ".[full]"
```

### Logging

- We use Weights & Biases (wandb) for experiment logging.
- Tracing via Weave has been removed; no additional tracing setup is required.

### Getting Help

- Check [Issues](https://github.com/your-repo/issues) for known problems
- Run tests to verify installation: `python -m pytest tests/ -v`
- Enable verbose logging: Set `verbose=True` in `explain()` calls
