Metadata-Version: 2.4
Name: lite-llm-doctor
Version: 0.1.2
Summary: LiteLLM / OpenAI-compatible gateway doctor CLI (ping, tokens, context bench)
Requires-Python: >=3.12
Requires-Dist: httpx>=0.28
Requires-Dist: python-dotenv>=1.0
Requires-Dist: tiktoken>=0.9
Requires-Dist: typer>=0.16
Description-Content-Type: text/markdown

# litellm-doctor

CLI `llm` for OpenAI-compatible gateways (LiteLLM and similar): list models, ping, local token estimates, needle context test, stepped context bench.

## Setup

```bash
git clone git@github.com:hewimetall/litellm-doctor.git
cd litellm-doctor
cp .env.example .env   # set OPENAI_API_KEY (+ optional OPENAI_BASE_URL)
uv sync
```

`.env`:

```env
OPENAI_BASE_URL=http://localhost:4000/v1
OPENAI_API_KEY=sk-...
```

Default base URL if unset: `http://localhost:4000/v1`.  
Auth: `Authorization: Bearer $OPENAI_API_KEY`.

Vendored tokenizer files ship in the repo under `tiktoken_cache/` (no download required for `--mode openai`).

## Commands

| Command | Purpose |
|---------|---------|
| `llm list` | `GET /models` |
| `llm info [MODEL]` | `GET /model/info` (max_in / max_out / mode / input types / key) |
| `llm ping MODEL` | single `chat/completions` |
| `llm pings` | parallel ping of all models (`-j`) |
| `llm tokens TEXT` | local token count |
| `llm context MODEL` | one needle test at a given size |
| `llm bench MODEL` | size sweep: est vs usage vs OK/FAIL |

### Examples

```bash
uv run llm list
uv run llm info
uv run llm info --json          # includes input_types + supports_* flags
uv run llm ping gpt-4o-mini
uv run llm pings -j 8 --timeout 5

uv run llm tokens "hello world"
uv run llm tokens - --mode openai < prompt.txt

uv run llm context my-model -n 32000
uv run llm context my-model -n 32000 --mode openai --no-random

uv run llm bench my-model
uv run llm bench my-model --start 1000 --stop 64000 --step 8000
uv run llm bench my-model --no-random --json
```

Common flags: `--timeout`, `--max-tokens`, `--json` (where supported).

## Tokenizer (`--mode`)

| Mode | How it counts | Network |
|------|---------------|---------|
| `opencode` (default) | `chars / 4` (OpenCode-style) | none |
| `openai` | local tiktoken + chatml overhead | none (vendored files only) |

Model → encoding for `--mode openai`: see `MODEL_ENCODING` in `src/helpers/tokenize.py`  
(default `cl100k_base`; `o200k_base` for gpt-4o / o3 / …). Unknown model names fall back to `cl100k_base`.

### Vendored encodings

Committed in-repo (safe to `git push`):

- `tiktoken_cache/cl100k_base.tiktoken`
- `tiktoken_cache/o200k_base.tiktoken`

Source of the pin script (optional refresh): GitHub `niieani/gpt-tokenizer` raw data — **not** `openaipublic.blob.core.windows.net`.

```bash
bash scripts/pin_tiktoken.sh   # refresh from GitHub if needed
```

Runtime loads BPE from disk (`load_tiktoken_bpe(path)`). `tiktoken.get_encoding()` is not used, so OpenAI CDN is never contacted.

## `context` / `bench`: prompt shape

Single `user` message:

```text
[bust=<hex>]          # only when --random (default)
SECRET=NEEDLE-42
<padding>
Find SECRET= in this message. Reply with the SECRET value only.
```

| Flag | Behavior |
|------|----------|
| `--random` (default) | `bust=` + random pad — avoids KV / prefix-cache hits |
| `--no-random` | no `bust=`, stable `lorem ipsum` — prefix may cache |

Step success: HTTP 200 and reply contains `NEEDLE-42`.

### `bench` defaults

- sizes: `1000 … 128000`, step `8000`
- `max_tokens=16`, `timeout=30`
- columns: `target`, `est`, `usage` (`prompt_tokens`), `ms`, `status`, `recall`
- summary: `max_ok_target`, `first_fail_target`, meta `max_input_tokens`

## Releases

Push a tag `v*` (e.g. `v0.1.0`) to run `.github/workflows/release.yml`:

| Artifact | Platforms |
|----------|-----------|
| `llm` binary (PyInstaller one-file) | `linux-x86_64`, `windows-x86_64`, `macos-x86_64`, `macos-aarch64` |
| Docker image | `ghcr.io/hewimetall/litellm-doctor` (Python runtime — not a packaged binary) |
| PyPI package | `pip install lite-llm-doctor` (sdist + wheel; version = tag without `v`) |

Manual `workflow_dispatch` builds artifacts / validates the image without publishing a GitHub Release, pushing to GHCR, or uploading to PyPI.

PyPI uses [Trusted Publishing](https://docs.pypi.org/trusted-publishers/) (OIDC) from this workflow — no API token in repo secrets. Register the publisher on PyPI with workflow `release.yml` and Environment name left empty (Any).

```bash
docker pull ghcr.io/hewimetall/litellm-doctor:latest
docker run --rm -e OPENAI_API_KEY -e OPENAI_BASE_URL ghcr.io/hewimetall/litellm-doctor:latest list
```

## Layout

```text
src/helpers/llm.py          # CLI
src/helpers/tokenize.py     # token estimates + needle_prompt
src/helpers/paths.py        # data/.env roots (dev, wheel, frozen)
scripts/pin_tiktoken.sh     # optional refresh from GitHub
tiktoken_cache/*.tiktoken   # vendored encodings (in git)
.github/workflows/release.yml
Dockerfile                  # GHCR runtime image
.env                        # secrets — gitignored
.env.example                # template
```
