Metadata-Version: 2.4
Name: poolprobe
Version: 1.0.0
Summary: A doctor for OpenAI-compatible LLM gateways: probe the pool, classify every failure, and only then talk about pruning.
License: MIT License
        
        Copyright (c) 2026 ameobius
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://gitlab.com/ameobius-ai/poolprobe
Project-URL: Source, https://gitlab.com/ameobius-ai/poolprobe
Project-URL: Issues, https://gitlab.com/ameobius-ai/poolprobe/-/issues
Keywords: llm,gateway,openai,cliproxy,healthcheck,ops,fallback
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: cliproxy
Requires-Dist: PyYAML>=6.0; extra == "cliproxy"
Dynamic: license-file

<div align="center">

# poolprobe

**A doctor for OpenAI-compatible LLM gateways.**
Probe the pool, classify every failure, and only *then* talk about pruning.

[![python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![dependencies](https://img.shields.io/badge/dependencies-stdlib%20only-brightgreen.svg)](#install)
[![license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

</div>

---

A pool of routed models rots quietly. An upstream drops a model, a key hits an
account tier, a reasoning model starts spending its whole token budget on
thinking, a gateway route begins answering SSE to a non-streaming request. None
of that raises an exception anywhere. The agent just gets an empty reply, or a
slower one, or falls through to a worse model — and nobody notices for a week.

`poolprobe` looks at every model in the pool, tells you *why* each one is
failing, and refuses to delete anything until five separate guards agree it is
dead. It grew out of running a live [cliproxy] pool on a schedule and learning,
the hard way, how many "dead model" verdicts were really weather.

## The three rules

Everything in the tool obeys these. They are the reason it exists.

1. **A verdict is not a death sentence.** Every failure is classified into a
   kind — `ALIVE`, `ALIVE-STARVED`, `PERMANENT`, `TRANSIENT`, `CREDENTIAL`,
   `MODALITY`, `UNKNOWN`. Only `PERMANENT` (the upstream genuinely does not know
   this model) is ever a prune candidate. A 401 is a dead *key*, not a dead
   *model*; pruning on it deletes a good alias because a credential ran dry.

2. **One observation is not a verdict.** A measured pool produced **544**
   scheduled liveness runs over six weeks; **39%** went red, and *every single
   red was transient*. `poolprobe` burst-confirms by default — five attempts in
   one run, best answer wins — before it will call anything dead.

3. **A number without evidence is not reported.** Throughput comes from the
   gateway's own `usage` block, or it is marked as a lower bound (`>=`), or it
   renders as `-`. Never estimated from SSE chunk counts and dressed up as a
   measurement.

## Install

```bash
pip install poolprobe                 # stdlib engine, generic gateway lane
pip install 'poolprobe[cliproxy]'     # + PyYAML, to read a cliproxy-style config
```

No PyPI release yet? Install straight from the checkout:

```bash
git clone https://gitlab.com/ameobius-ai/poolprobe
cd poolprobe && pip install '.[cliproxy]'
```

The engine has **zero runtime dependencies**. PyYAML is optional and only needed
to attribute models to providers and to plan config edits from a gateway YAML.

## Quickstart

```bash
# 1. Is the gateway even up, and what does it publish? (no chat request sent)
poolprobe ping -u http://127.0.0.1:8317/v1 -k "$KEY"

# 2. Probe the whole pool, burst-confirmed, and keep a self-contained HTML report
poolprobe sweep -u http://127.0.0.1:8317/v1 -k "$KEY" --html out.html

# 3. Give it memory: fail streaks across runs are what make `prune` safe
poolprobe sweep -u ... -k "$KEY" --state ~/.cache/poolprobe.json

# 4. Only list what passed all five guards — nothing on disk changes without --apply
poolprobe prune -u ... -k "$KEY" --state ~/.cache/poolprobe.json \
                --adapter cliproxy --config ~/.config/gateway.yaml

# 5. Validate agent fallback chains against the lane rules that actually read them
poolprobe chains ~/.hermes/config.yaml --cross-check -u ... -k "$KEY"
```

`-u` also reads `POOLPROBE_BASE_URL`; `-k` reads `POOLPROBE_API_KEY` then
`OPENAI_API_KEY`; `--state` reads `POOLPROBE_STATE`. The bearer key is masked in
every argv echo — reports are meant to be pasted into issues.

### What a sweep looks like

```
poolprobe 1.0.0  ·  sweep
http://127.0.0.1:8317/v1 · generic adapter (read-only)

5 models  ·  4 alive  ·  0 starved  ·  1 dead  ·  0 prunable  ·  median 5.43s  ·  run #1

VERDICT       MODEL                PROVIDER   LAT      BURST  STREAK  KIND        NOTE
------------  -------------------  ---------  -------  -----  ------  ----------  --------------------
OK            gpt-5.4              -            2.74s  ✓         ok1  -           alive
OK            gemini-3.8-flash     -            6.12s  ✓         ok1  -           alive
OK            grok-4.6             -           27.3s   ✓         ok1  -           alive
ERR           suno-music           -            0.13s  ✗          x1  CREDENTIAL  {"error":{"message"…
OK            deepseek-v4-flash    -            4.73s  ✓         ok1  -           alive

prunable: none — nothing met all five guards (dead, PERMANENT, streak, unprotected, chat modality)

what to do
  CREDENTIAL (1): credential or account level: rotate/top up the key, never prune the model
```

`grok-4.6` answering `OK` here is the whole point: a prior single-shot check had
written it off as dead. The burst said otherwise.

## Commands

| Command | What it does | Sends a chat request? |
|---------|--------------|:---------------------:|
| `ping` | Reach the gateway, list the published catalog, break it down by provider. | no |
| `sweep [models…]` | Burst-confirmed liveness probe; one verdict + kind per model. | yes |
| `bench [models…]` | Latency and throughput; `--repeat N` reports the median, `--stream` measures time-to-first-token. | yes |
| `prune [models…]` | List models that passed all five guards and plan the config edit. `--apply` writes it (previous copy kept as `.bak`). | yes |
| `chains PATH` | Validate agent fallback chains against the lane rules that read them. `--cross-check` diffs rungs against the live catalog. | no |
| `selfcheck` | Offline assert suite — taxonomy, burst logic, prune guards, argv redaction, all three renderers. **This is what CI runs.** | no |

Common options: `--adapter {auto,cliproxy,generic}`, `--config PATH`,
`--threads N`, `--timeout S`, `--max-tokens N`, `--prompt TEXT`, `--state PATH`,
`--streak-min N`, `--protect MODEL` (repeatable), `--json PATH`, `--html PATH`,
`--color {auto,always,never}`, `-q/--quiet`.

`--max-tokens` is a **floor, not a cap**: a reasoning model spends it on thinking
and answers `EMPTY-LENGTH`, which `poolprobe` reads as *alive but starved*, never
as dead.

## Exit codes

A cron contract. Red is a finding, not noise.

| Code | Meaning |
|:----:|---------|
| `0` | Nothing dead — or nothing that met every guard. |
| `1` | Something dead (burst-confirmed), or a fallback chain is broken. |
| `2` | Usage error, unreadable config, or PyYAML missing for a config read. |
| `3` | The gateway could not be reached, or published an empty catalog. |
| `130` | Interrupted. |

## The five prune guards

`prune` will not name a model unless **all five** agree. A report that says
"prunable" while the prune cycle deletes something else is the worst outcome for
a tool like this, so the guards live in one function, checked in one place:

1. **dead verdict** — `EMPTY`, `ERR`, or `TIMEOUT` (`OK`/`EMPTY-LENGTH` never qualify);
2. **kind `PERMANENT`** — the upstream does not know this model;
3. **fail streak** — dead for at least `--streak-min` consecutive *runs* (needs `--state`);
4. **not protected** — absent from `--protect` (a namespaced alias's base name matches too);
5. **not a generation modality** — image/video/tts/music models legitimately fail a chat probe.

Without `--state` there is no memory, every streak reads `0`, and guard 3 can
never pass — so `prune` lists candidates but changes nothing. That is the safe
default.

## Reading a report

**Verdict** — what the probe saw:

`OK` · `EMPTY` (200, empty body) · `EMPTY-LENGTH` (empty because the token budget
ran out — alive) · `ERR` (non-2xx) · `TIMEOUT`.

**Kind** — what it means:

| Kind | Reading | Prunable? |
|------|---------|:---------:|
| `ALIVE` | Answered. | no |
| `ALIVE-STARVED` | Answered `EMPTY-LENGTH` — raise `--max-tokens`. | no |
| `PERMANENT` | Upstream does not know this model (`404`, "does not exist"). | **yes** |
| `TRANSIENT` | Timeout, `429`, `5xx` — weather. Burst-confirm before believing it. | no |
| `CREDENTIAL` | `401`/`403`, "invalid api key" — rotate the key, never the model. | no |
| `MODALITY` | Model lives on another endpoint (`/v1/images/…`), not chat. | no |
| `UNKNOWN` | None of the above; reported, never acted on. | no |

Three renderers, one report: the terminal table (ANSI, honours `NO_COLOR` /
`FORCE_COLOR` / `TERM=dumb` / isatty), a machine-readable `--json`, and a
self-contained `--html` you can attach to a post. They cannot disagree about
what a run recorded.

## Library use

```python
from poolprobe import Gateway, sweep, taxonomy

gw = Gateway("http://127.0.0.1:8317/v1", api_key="local")
for result in sweep(gw, ["gpt-5.4", "glm-5.3-flash"], attempts=5):
    print(result.model, result.verdict, result.kind, result.advice)
```

## Development

```bash
python -m poolprobe selfcheck   # the whole offline contract; CI runs exactly this
```

`selfcheck` needs no gateway, no network and no secrets. If it passes, the
taxonomy, the burst "best truth wins" logic, the five prune guards, the argv
redaction and all three renderers are behaving.

## License

MIT — see [LICENSE](LICENSE).

[cliproxy]: https://github.com/router-for-me/CLIProxyAPI
