Metadata-Version: 2.4
Name: probelock
Version: 0.1.0
Summary: A capability lockfile for local models. Catch silent regressions when you swap a model, quant, or runtime.
Project-URL: Homepage, https://github.com/kelkalot/probelock
Project-URL: Repository, https://github.com/kelkalot/probelock
Project-URL: Issues, https://github.com/kelkalot/probelock/issues
Author: Michael A. Riegler
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ci,evals,llama-cpp,llm,local-models,mlx,ollama,quantization,regression-testing,tool-calling
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: jsonschema>=4.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Provides-Extra: anyllm
Requires-Dist: any-llm-sdk>=0.2; (python_version >= '3.11') and extra == 'anyllm'
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: litellm
Requires-Dist: litellm>=1.0; extra == 'litellm'
Description-Content-Type: text/markdown

# probelock

**A capability lockfile for local models.** It records what a model does on a set
of tool-calling and output checks, and fails CI when a model/quant/runtime swap
lowers a score.

```
llama-3.1-8b @ Q8_0 (ollama)  →  llama-3.1-8b @ Q4_K_M (ollama)
Capability            Baseline   Candidate     Δ   Status
arg_validity              1.00        0.67  -0.33  REGRESSION
arity_robustness          1.00        0.67  -0.33  REGRESSION
format_adherence          1.00        1.00  +0.00  ok
needle_in_tools           1.00        0.33  -0.67  REGRESSION
no_hallucinated_tool      1.00        0.67  -0.33  REGRESSION
required_args             1.00        1.00  +0.00  ok
structured_output         1.00        0.33  -0.67  REGRESSION
tool_discrimination       1.00        0.33  -0.67  REGRESSION
tool_permission           1.00        0.67  -0.33  REGRESSION
tool_restraint            1.00        0.67  -0.33  REGRESSION
tool_selection            1.00        0.67  -0.33  REGRESSION

FAIL — capabilities regressed or removed: arg_validity, arity_robustness,
needle_in_tools, no_hallucinated_tool, structured_output, tool_discrimination,
tool_permission, tool_restraint, tool_selection
```

Here the Q4 quant scores 0.33–0.67 on several capabilities where Q8 scored 1.00.
`probelock gate` exits non-zero when a capability drops past the threshold.

## How it differs from promptfoo

> **promptfoo is a test framework you author. probelock is a lockfile you commit.**

1. **Probes are derived from your tool schemas.** Point it at the OpenAI-style
   tool definitions your agent already ships, and it generates a fixed,
   reproducible battery of capability checks. You write no test cases.
2. **No LLM judge.** Every probe is scored by code: JSON-schema validation, exact
   match, or a tool-name check. Run it twice on the same model and the numbers
   match. (promptfoo relies on assertions you write and often on model-graded
   evals, which vary across runs.)
3. **It compares a model against its own baseline,** across a model/quant/runtime
   swap, rather than producing an absolute leaderboard. You only ever compare like
   with like, on your box, with your tools, so the "benchmarks are
   gameable/hardware-dependent" objection doesn't apply.

## Install & run (only needs [uv](https://docs.astral.sh/uv/))

Run it without installing, or install it into the current environment:

```bash
uvx probelock --help          # run the latest release
pip install probelock         # or install it
```

To run an unreleased revision straight from git:

```bash
uvx --from git+https://github.com/kelkalot/probelock probelock --help
```

The examples below use `uv run` from a checkout of this repo. No model is required
for the demo — a deterministic `SimulatedClient` stands in for two quant levels of
the same model:

```bash
# from the probelock/ project dir
uv run probelock derive --tools examples/agent_tools.json          # see the probe battery
uv run probelock probe  --tools examples/agent_tools.json --simulate fixtures/profile_q8.json -o q8.lock
uv run probelock probe  --tools examples/agent_tools.json --simulate fixtures/profile_q4.json -o q4.lock
uv run probelock diff   q8.lock q4.lock
uv run probelock gate   --baseline q8.lock --candidate q4.lock     # exits non-zero
```

Against a local model, swap `--simulate` for an OpenAI-compatible endpoint:

```bash
uv run probelock probe --tools examples/agent_tools.json \
    --endpoint http://localhost:11434/v1 --model llama3.1:8b-instruct-q4_K_M \
    --quant Q4_K_M --runtime ollama --timeout 120 -o q4.lock
```

A probe the model rejects (e.g. "model does not support tools") or that times out
scores **0** for that capability and the run continues, so a model that can't
tool-call still produces a lockfile. An unreachable server, a 404 (wrong model or
URL), or a run where every probe fails aborts the run, so a misconfiguration never
becomes a poisoned all-zeros baseline.

## Providers & frameworks

probelock speaks one protocol — OpenAI `/v1/chat/completions` with OpenAI-style
tools — so anything that exposes it works with `--endpoint`. For providers that
don't (Anthropic, Gemini, …), route through a unified SDK with `--via`. Every path
is deterministic; none of them put an LLM in the loop.

| You have… | Use |
|-----------|-----|
| Ollama, **vLLM**, **llama.cpp** server, LM Studio, HF TGI, OpenAI, OpenRouter, Together… | `--endpoint <url>/v1 --model <name>` (vLLM needs `--enable-auto-tool-choice`; llama.cpp needs `--jinja`) |
| **Anthropic / Gemini / Mistral / Bedrock / …** (any-llm) | `--via anyllm --model anthropic/claude-3-5-sonnet` |
| Any of 100+ providers (litellm SDK) | `--via litellm --model anthropic/claude-3-5-sonnet` |
| A running **LiteLLM proxy** | `--endpoint http://litellm:4000/v1 --model <name>` (no extra) |
| In-process HF `transformers` / MLX (no server) | not yet — add a small `Client` adapter |

```bash
pip install 'probelock[anyllm]'   # or 'probelock[litellm]'
probelock probe --tools tools.json --via anyllm --model mistral/mistral-large-latest \
    --samples 5 --temperature 0.7 -o candidate.lock
```

`--via` clients reuse the same caching, sampling, and error semantics as
`--endpoint`; they're thin adapters over each SDK's OpenAI-shaped response. Add a
new backend by implementing the tiny `Client` protocol — that's the only seam.

### Recorded demo (Ollama)

[`demo/`](demo/) has runs against a local Ollama server: a committed `qwen3.5:9b`
baseline vs a `gemma3:1b` candidate (which does not support tool-calling). See
[`demo/DEMO.md`](demo/DEMO.md) for the transcript, or replay it:

```bash
asciinema play demo/probelock-demo.cast   # or: bash demo/demo.sh
```

The tool-calling capabilities drop `1.00 → 0.00` and the gate exits non-zero.
`tool_restraint`, `tool_permission`, and `no_hallucinated_tool` stay `1.00` (a
model that can't call tools can't misuse one), and `gemma3:1b` scores `1.00` on
`format_adherence` vs `0.50` for `qwen3.5:9b`. The diff is per-capability.

Also committed: `qwen3.5:9b` vs `lfm2.5-thinking:1.2b`:

```bash
uv run probelock diff demo/qwen3.5-9b.lock demo/lfm2.5-thinking.lock
```

The 1.2B model matches `qwen3.5:9b` on tool selection, discrimination,
`needle_in_tools`, `arg_validity`, `required_args`, and the three safety probes;
`structured_output` and `arity_robustness` drop `1.00 → 0.33`.

## The capabilities (all scored deterministically)

| Capability            | What it checks                                            | Scorer |
|-----------------------|-----------------------------------------------------------|--------|
| `tool_selection`      | Calls the right tool for the task                         | tool-name match |
| `tool_discrimination` | Calls the right tool **and no other** (picks precisely)   | tool-name set |
| `needle_in_tools`     | Finds the right tool when many (15+) are offered          | tool-name match |
| `arg_validity`        | Emitted args validate against the tool's JSON schema      | `jsonschema` |
| `required_args`       | All required args present and non-empty                   | key presence |
| `arity_robustness`    | Fills **every** parameter (required + optional) when asked | all-present |
| `structured_output`   | Emits schema-valid JSON on demand (no tools, no fences)   | parse + `jsonschema` |
| `tool_restraint`      | Does **not** call a tool for a task that needs none (over-trigger) | no tool call |
| `tool_permission`     | Does **not** call a tool it was explicitly forbidden to use | forbidden tool absent |
| `no_hallucinated_tool`| Does **not** fabricate a call to a tool that wasn't offered | called ⊆ offered |
| `format_adherence`    | Follows an exact output constraint                        | exact match |

Three are **negative** probes (a higher score means the bad behavior didn't
happen): `tool_restraint` (over-triggering), `tool_permission` (calling a forbidden
tool), and `no_hallucinated_tool` (fabricating a tool). All probes are derived from
the tool schemas, not hand-authored.

## Architecture

```
tool schemas ──▶ derive probes ──▶ Client ──▶ ResponseMessage ──▶ deterministic scorer ──▶ Lockfile
 (your agent)    (zero authoring)  (model)    (the only model      (no LLM judge)          (commit it)
                                              -touching part)
                                                                    Lockfile + Lockfile ──▶ diff / gate
```

The only nondeterministic part is the `Client`; everything else is pure, so the
same inputs produce the same lockfile and the same diff. At temperature 0 the
client caches identical requests, so the probes that share one request (the tool
checks for a given tool) hit the network once. The `SimulatedClient` crafts correct or
incorrect responses that the real scorers grade, so the scoring path runs even
with no model present.

## Sampling & noisy gates

With one sample per probe, a capability backed by 3 tools quantizes to
`{0, 0.33, 0.67, 1.0}` — a single flip moves it 0.33, far past the default 0.05
gate. So:

- **`probe --samples N [--temperature T]`** runs each probe N times and records the
  pass-*rate* (raise the temperature for sampling variance).
- **`gate --confidence 0.95`** only fails on a drop that is statistically
  significant for the recorded trial count (a one-sided two-proportion test).
  Sub-significant drops are shown as **`noisy ↓`** and do **not** fail the gate;
  raise `--samples` to confirm or clear them.

A total collapse (e.g. `1.00 → 0.00`) is significant even at low N; a single-flip
`1.00 → 0.67` over 3 trials is `noisy` until you raise `--samples`.

## In CI

`probelock init` scaffolds a `probelock.tools.json` and a
`.github/workflows/probelock.yml` to start from. Commit a baseline lockfile, then
gate each candidate:

```yaml
- run: uvx probelock probe
       --tools tools.json --endpoint $LLM_URL --model $MODEL --samples 5 --temperature 0.7 -o candidate.lock
- run: uvx probelock gate
       --baseline probelock.lock --candidate candidate.lock --max-drop 0.05 --confidence 0.95
```

A composite GitHub Action ([`action.yml`](action.yml)) wraps those two steps. To
show the result on a pull request, render the diff as Markdown (or `--format html`
for a self-contained page):

```bash
probelock diff probelock.lock candidate.lock --format markdown >> "$GITHUB_STEP_SUMMARY"
```

## Roadmap

- A `--json-mode` `structured_output` probe (`response_format`) beside the strict prompt path.
- Trend view: compare a capability across more than two lockfiles (a quant ladder Q8→Q5→Q4→Q3).
- In-process backends (HF `transformers` / MLX) via a small `Client` adapter, no server required.

## License

Apache-2.0 — see [LICENSE](LICENSE).

## Acknowledgements

Built with [Claude Code](https://claude.com/claude-code).
