Metadata-Version: 2.4
Name: llm-judge-panel
Version: 0.1.0
Summary: A multi-provider LLM-as-a-Judge consensus panel with chained criteria for concept matching validation.
Project-URL: Homepage, https://github.com/sjmeis/llm-judge-panel
Project-URL: Documentation, https://github.com/sjmeis/llm-judge-panel
Project-URL: Repository, https://github.com/sjmeis/llm-judge-panel
Project-URL: Issues, https://github.com/sjmeis/llm-judge-panel/issues
Author-email: Stephen Meisenbacher <sjmeis@gtgd.com>
License: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: anthropic>=0.3.0
Requires-Dist: google-genai>=0.1.0
Requires-Dist: openai>=1.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: together>=1.0.0
Description-Content-Type: text/markdown

# llm-judge-panel

A flexible, concurrent framework for evaluating concept matching in text datasets, using an LLM-as-a-Judge voting consensus. Chain multiple model providers and evaluate text rows across custom voting criteria simultaneously.

## Features
 - **Two-stage Workflow**: Run Stage 1 (Concept Verification) and Stage 2 (Candidate Matching) independently or combined.
 - **Parallel Multi-Criteria Evaluation**: Evaluate your custom voting rules (e.g., strict and majority agreements) simultaneously in a single run.
 - **Granular Outputs**: View aggregated execution metrics instantly and export row-level details directly to JSON, dictionaries, or pandas.
 - **Async Execution with Live Progress Tracking**: Leverages `asyncio` to execute judge pools concurrently, with real-time progress tracking.

## Installation
It's simple!

```python
pip install llm-judge-panel
```

## Getting Started

### 1. Configure Environment Variables
Store your respective API keys in your environment variables, namely for the LLM provider(s) you would like to use. We currently support OpenAI, Anthropic, Google Gemini, and Together AI.

```bash
export OPENAI_API_KEY="your-key"
export ANTHROPIC_API_KEY="your-key"
export GEMINI_KEY="your-key"
export TOGETHER_API_KEY="your-key"
```

### 2. Define Your Concept
"Concepts" are a quantity of interest. These can be anything!

```python
from llm_judge_panel import Concept

skill_concept = Concept(
        name="skill",
        definition="The ability to perform a specific task and apply knowledge in a work context."
    )
```

### 3. Define the Voting Criterion
Decide how you would like to resolve the individual LLM votes. You can choose one or multiple (current `strict`/unanimous, `majority`, and `at_least_m`).

```python
from llm_judge_panel import VotingCriterion

evaluation_strategies = {
        "strict": VotingCriterion(rule="strict"),
        "majority": VotingCriterion(rule="majority")
    }
```

### 4. Set Up the Panel and Run!
A panel consists of one or more LLM judges. Instantiate the panel with something like the following, and run on your data:

```python
from llm_judge_panel import Panel
from llm_judge_panel.providers import *

async with Panel(criteria=evaluation_strategies) as panel:
    panel.add_judge(OpenAIProvider("gpt-4o-mini"))
    panel.add_judge(AnthropicProvider("claude-sonnet-4-5"))

    # tuples of the form (extracted sentence, match candidate)
    dataset = [
        ("Seeking an expert in distributed Go microservices.", "Go language"),
        ("...", "...")
    ]

    run_result = await panel.judge_full_pipeline(skill_concept, dataset)

    df = run_result.to_dataframe()
    df.to_csv("results.csv", index=False)
```

#### Optional: Separate Stages
The function `judge_full_pipeline` runs both Stage 1 and 2 validation in order. If you would like only to run on of these, do the following:

```python
# setup as before

run_result = await panel.judge_stage1(skill_concept, LIST_OF_LABELED_TEXT_TUPLES) # i.e., (TEXT, 1/0)
# OR
run_result = await panel.judge_stage2(skill_concept, LIST_OF_TUPLES) # i.e., like the above example
```

## Contributing
For more providers or more flexible functionality, feel free to open up an issue!