Metadata-Version: 2.4
Name: llm-probe
Version: 0.1.2
Summary: Probe LLM models for tool-calling support, think block behavior, and response quality across providers.
Author-email: Z7Lab <Z7Lab@pm.me>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Z7Lab/LLM-probe
Project-URL: Issues, https://github.com/Z7Lab/LLM-probe/issues
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Requires-Dist: pyyaml>=6.0
Requires-Dist: click>=8.0
Dynamic: license-file

# llm-probe

Test whether LLM models actually work before you build on them. Discovers models from cloud and local providers, runs each through tool-calling, system prompt adherence, and response quality checks, then outputs a verified model list your project can consume.

Picking a model for agents or tool-calling workflows? Running Hugging Face models through Ollama, vLLM, or any OpenAI-compatible endpoint? Most models can generate text, but structured tool calls, enum constraints, and system prompt adherence are different — llm-probe shows you exactly where each model holds up and where it falls short for your use case.

## Install

```bash
pip install llm-probe
```

## Quick Start

Run from your project directory — results are written to `./verified/` relative to where you run it.

```bash
cd ~/dev/projects/my-project

# Set your API key
export VENICE_API_KEY=your-key-here

# Discover available models
llm-probe discover venice

# Test a specific model
llm-probe test venice --model mistral-small-3-2-24b-instruct

# Test all models from a provider
llm-probe test venice

# View results
llm-probe report
```

Ollama needs no API key but requires a running instance. If it's not on localhost:

```bash
llm-probe --config config.yaml discover ollama
llm-probe --config config.yaml test ollama --model qwen3:8b --timeout 180
```

## Providers

| Provider | Discovery | Auth | Default URL |
|----------|-----------|------|-------------|
| **venice** | `/v1/models` API | `VENICE_API_KEY` env var | `https://api.venice.ai/api/v1` |
| **ollama** | `/api/tags` endpoint | None | `http://localhost:11434` |
| **anthropic** | `/v1/models` API | `ANTHROPIC_API_KEY` env var | `https://api.anthropic.com` |
| **openai** | `/v1/models` API | `OPENAI_API_KEY` env var | `https://api.openai.com/v1` |

All providers except Anthropic use the OpenAI-compatible chat completions API. Anthropic uses the Messages API with `tool_use` content blocks.

### Custom OpenAI-Compatible Providers

Any service that speaks the OpenAI chat completions API can be used without writing code. Just add it to `config.yaml`:

```yaml
providers:
  lmstudio:
    api_base: http://localhost:1234/v1
  openrouter:
    api_base: https://openrouter.ai/api/v1
    api_key_env: OPENROUTER_API_KEY
  text-gen-webui:
    api_base: http://192.168.2.50:5000/v1
```

Then use it like any built-in provider:

```bash
llm-probe discover lmstudio
llm-probe test lmstudio --model my-local-model
```

Custom providers get automatic model discovery via `/models`, tool-call testing via `/chat/completions`, and local timeout defaults if the URL points to a private IP or localhost.

## Tests

Each model is tested for specific capabilities. Tests run N times (default 3) with a pass threshold (default 80%) to catch intermittent failures. See [TESTING.md](TESTING.md) for detailed test documentation.

| Test | What it checks |
|------|---------------|
| `tool_call_basic` | Send one tool + an obvious prompt. Does the model return a structured `tool_calls` response (not text)? |
| `tool_call_enum` | Send a tool with enum-constrained parameters. Does the model respect the constraint? |
| `tool_call_multi` | Send three tools. Does the model pick the correct one? |
| `think_block_leak` | Does internal reasoning (`<think>`, `Thinking Process:`, etc.) leak into the response content? |
| `empty_response` | Does the model produce empty, degenerate (`!!!!`), or garbage output? |
| `system_prompt` | Given a system prompt that overrides user intent, does the model follow it? Tests if the model respects system prompt instructions over user messages. |
| `response_time` | Records response latency as metadata. Always passes. |

Run specific tests with `--tests`:

```bash
llm-probe test venice --model mistral-small-3-2-24b-instruct --tests tool_call_basic,think_block_leak
```

## CLI Reference

```bash
# Global option: --config / -c (optional, defaults work for all providers)
llm-probe --config config.yaml <command>

# Discover available models from a provider
llm-probe discover <provider>

# Test models
llm-probe test <provider>                          # all models
llm-probe test <provider> --model <name>           # one model
llm-probe test <provider> --tests <test1,test2>    # specific tests
llm-probe test <provider> --runs 5 --threshold 0.8 # consistency testing
llm-probe test <provider> --timeout 180            # for slow local models

# View results
llm-probe report                # all providers
llm-probe report <provider>     # one provider
```

## Example Output

```
[1/1] Testing mistral-small-3-2-24b-instruct...
  PASS 6/7 tests passed (avg 1358ms)

Results written to verified/venice.yaml

  + New models: mistral-small-3-2-24b-instruct
```

Re-running compares against previous results and reports regressions:

```
  REGRESSIONS:
    qwen3-5-9b: pass -> fail
  IMPROVEMENTS:
    llama-3.3-70b: intermittent -> pass
```

## Configuration

Configuration is optional. Defaults exist for all providers. Create a `config.yaml` to override:

```yaml
providers:
  venice:
    api_base: https://api.venice.ai/api/v1
    api_key_env: VENICE_API_KEY             # env var name
  ollama:
    api_base: http://192.168.2.189:11434    # override for non-localhost
  anthropic:
    api_key_env: ANTHROPIC_API_KEY
    api_key_secret: ~/secrets/anthropic_key # file path (checked if env var is empty)

tests:
  timeout_cloud: 30        # seconds per request for cloud providers
  timeout_local: 180       # seconds per request for local models (CPU inference)
  runs_per_model: 3        # test each model N times for consistency
  consistency_threshold: 0.8  # must pass 80% of runs to be "pass"

output:
  dir: ./verified          # where to write result YAML files
```

API keys resolve in order: `api_key_env` (environment variable) > `api_key_secret` (file path). Providers that don't need keys (Ollama) skip both.

## Output Format

Results are written to `./verified/<provider>.yaml`:

```yaml
provider: venice
last_run: "2026-03-31"
models:
  - id: mistral-small-3-2-24b-instruct
    tool_call: pass             # pass, intermittent, or fail
    think_blocks: none          # none, xml, plain_text
    avg_response_ms: 1358
    tests:
      tool_call_basic: {passed: true, pass_rate: "3/3"}
      tool_call_enum: {passed: true, pass_rate: "3/3"}
      tool_call_multi: {passed: true, pass_rate: "3/3"}
      think_block_leak: {passed: true, pass_rate: "3/3"}
      empty_response: {passed: true, pass_rate: "3/3"}
      system_prompt: {passed: false, pass_rate: "0/3"}
      response_time: {passed: true, pass_rate: "3/3"}
    last_tested: "2026-03-31"
```

Other projects can import these YAML files to build model allowlists for tool-calling features.
