Metadata-Version: 2.5
Name: jevify
Version: 0.3.0
Summary: Make a pretrained model you already have answer typed questions in one pass, and measure how well it does.
Project-URL: Homepage, https://github.com/fidecastro/jevify
Author: Felipe Infante de Castro
License-Expression: MIT
License-File: LICENSE
Keywords: decisions,evaluation,logprobs,openai-compatible,readout
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.12
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: starlette>=0.40
Requires-Dist: uvicorn>=0.30
Provides-Extra: encoder
Requires-Dist: safetensors>=0.4; extra == 'encoder'
Requires-Dist: torch>=2.4; extra == 'encoder'
Requires-Dist: transformers>=4.51; extra == 'encoder'
Provides-Extra: games
Requires-Dist: numpy>=1.26; extra == 'games'
Requires-Dist: vizdoom>=1.3; extra == 'games'
Description-Content-Type: text/markdown

# jevify

**Make any model you already have answer typed questions in one pass, the way
Jev does, and measure how well it does.**

[![CI](https://github.com/fidecastro/jevify/actions/workflows/ci.yml/badge.svg)](https://github.com/fidecastro/jevify/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/jevify.svg)](https://pypi.org/project/jevify/)
[![Licence: MIT](https://img.shields.io/badge/licence-MIT-blue.svg)](https://github.com/fidecastro/jevify/blob/main/LICENSE)
[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/)

You bring a model behind an OpenAI-compatible endpoint (vLLM, llama.cpp,
TensorFold, anything that returns logprobs for a reply token or for the
prompt) or a small model that runs in-process. jevify sends it
a **state** once, then asks **typed questions** about it: a choice among
options, a score on ordered levels, a yes/no with a probability. Each answer
is read straight off the model's next-token distribution, no generation, no
parsing, one request per question. The result comes back in the exact shape
of TypeSafe's Jev API, so the unmodified `typesafe-sdk` works against it.

jevify owns no weights and trains nothing. A **recipe** says how a model is
asked, a **probe** proves what the deployment can do, and a **scorecard**
says how it did, with the command, hashes and raw file that produced every
number.

```text
   state (text, images)          one warm request, cached prefix
        │
        ├── "Which team handles this?"  ──►  {billing: 0.94, support: 0.05, sales: 0.01}
        ├── "How urgent?"               ──►  {low: 0.05, medium: 0.10, high: 0.85}
        └── "Threatens to cancel."      ──►  p(true) = 0.95
                                              each ≈ 30–140 ms on a local GPU
```

## Quickstart

```sh
uv tool install jevify                # or: pipx install jevify
git clone https://github.com/fidecastro/jevify && cd jevify

# 1. point a recipe at your server (copy the nearest one under recipes/)
jevify probe recipes/gemma-4-e4b-it.llamacpp.yaml

# 2. ask
jevify ask recipes/gemma-4-e4b-it.llamacpp.yaml \
  --state "Customer: I was charged twice. Fix it today or I cancel." \
  --choice "dept:Which team should handle this?=billing,support,sales" \
  --score  "urgency:How urgent is this?=low,medium,high" \
  --noul   "cancel:The customer threatens to cancel." \
  [--image screenshot.png]

# 3. serve Jev's API and use the stock SDK
jevify serve recipes/gemma-4-e4b-it.llamacpp.yaml --port 8600
```

Any model behind an OpenAI-compatible server works, not only the ones under
`recipes/`: copy the nearest recipe, set `endpoint.base_url` and
`model.name`, and run the probe, which re-verifies the answer token ids for
that model and writes what it found into the file. vLLM and llama.cpp get
their dialect extras; any other server is driven through the plain OpenAI
surface. A server that refuses logprobs on the reply token but returns
vLLM's `prompt_logprobs`, such as TensorFold's native server with
GLM-5.3-Flash, is read through the `prompt_logprobs` rung on
`/v1/completions`: with a raw-mode recipe (the template written out with the
model's own control tokens), or with a chat-template recipe where the server
also takes `messages` on that route, as TensorFold's does. The chat form is
the one that carries images.

`serve` binds to `127.0.0.1`. To reach it from other machines pass
`--host 0.0.0.0`, and set `JEVIFY_API_KEY` first so that every request must
carry that bearer key; to leave it running, detach it and keep its log:

```sh
jevify serve recipes/<name>.yaml --host 0.0.0.0 --port 8600 > ~/jevify.log 2>&1 &
curl -s http://<this-host>:8600/health
```

Every answer carries Jev's fields (`choice`, `confidence`, `probabilities`,
`score`, `legend`, `noul`) plus an `x_jevify` block naming the readout method,
the cached tokens, the latency and the recipe hash that produced it.

From a clone, `make ask` and `make serve` wrap the two commands;
[`docs/playbook.md`](https://github.com/fidecastro/jevify/blob/main/docs/playbook.md)
walks through them and explains every answer field.

## How it works

1. **Recipe.** A hashed YAML file: the endpoint, the prompt template (chat
   messages or a raw prompt with the model's own control tokens), the answer
   token spellings, and the readout policy. The only way a model is ever
   asked.
2. **Probe.** `jevify probe` measures the deployment: dialect (vLLM,
   llama.cpp, generic), context, which logprob rungs it honours, whether
   thinking is really off, whether an image really adds tokens, how much of a
   warmed prefix a question reuses. The findings are written into the recipe.
3. **Warm, then ask.** The state goes once; each question is a short suffix
   that reuses the server's prefix cache. A 32k-token state costs one warm and
   then a few hundred milliseconds per question.
4. **Readout ladder.** Named-token logprobs, grammar with post-sampling
   probabilities, top-k, prompt logprobs (each answer token appended to the
   prompt and its logprob read back, for servers that give none on the reply
   token), equal bias, and a floored top-k as the last resort.
   Auto walks down the proven rungs; the answer names the rung that read it
   and whether it was degraded.
5. **Evidence.** `jevify eval` runs a frozen suite with controls (shuffled
   states, permuted options) and writes a scorecard. `jevify calibrate` fits
   temperatures and refuses tables it cannot defend.

## Model kinds

| Kind | Talks to | Proven live with |
|---|---|---|
| `endpoint` | Any OpenAI-compatible server that returns reply-token or prompt logprobs; dialect extras for vLLM and llama.cpp | DeepSeek-V4-Flash on vLLM; Ternary Bonsai 2 27B, Gemma 4 E4B and Qwen3-VL-Reranker 8B/2B on llama.cpp; text and images |
| `embedding` | A `/v1/embeddings` route, or a model in-process | Qwen3-Embedding-0.6B in-process |
| `encoder` | An NLI sequence-classification head in-process | facebook/bart-large-mnli in-process |
| `rerank` | A `/v1/rerank` route | Fakes only |

The in-process kinds need `uv tool install 'jevify[encoder]'`.

## Results

Four frozen suites, every recipe on each, one run each, latencies per
question on one machine on one day. Every row links to the scorecard that
records the command, the recipe hash, the suite hash and the raw file digest.

### Latency per question

The median of each suite, combined across the suites a model ran; one run
each. A question is one request that reads the answer off the next-token
probabilities, after the state was sent once and cached.

| Model | How it runs | Hardware | Latency per question |
|---|---|---|---|
| facebook/bart-large-mnli | in-process NLI classifier | RTX 4090 | 7–9 ms, avg 8 |
| Qwen3-Embedding-0.6B | in-process embedding, cosine | RTX 4090 | 8–10 ms, avg 9 |
| Gemma 4 E4B (Q8_0) | llama.cpp, chat | RTX 4090 | 28–37 ms, avg 32 |
| Qwen3-VL-Reranker-2B (Q8_0) | llama.cpp, yes/no readout | RTX 4090 | 47–64 ms, avg 53 |
| Qwen3-VL-Reranker-8B (Q8_0) | llama.cpp, yes/no readout | RTX 4090 | 68 ms (one suite) |
| Ternary Bonsai 2 27B (PQ2_0) | llama.cpp (PrismML fork), raw prompt | RTX 4090 | 97–113 ms, avg 102 |
| DeepSeek-V4-Flash | vLLM, chat | 2× DGX Spark, over LAN | 359–672 ms, avg 483 |

Image questions cost the llama.cpp models nothing extra, since the frame is
paid once in the warm; DeepSeek pays about half again on images. The two
in-process models are the fastest and the least accurate.

### Accuracy by suite

**policy-29**, the diagnostic suite carried over from the author's earlier experiments (choice only):

| Recipe | Kind | Accuracy | Median latency | Scorecard |
|---|---|---|---|---|
| `gemma-4-e4b-it.llamacpp` | endpoint | 29/29 | 28 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/gemma-4-e4b-it.llamacpp--policy-29.md) |
| `ternary-bonsai-2-27b.llamacpp` | endpoint | 29/29 | 99 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/ternary-bonsai-2-27b.llamacpp--policy-29.md) |
| `deepseek-v4-flash-vision-exp-keys.vllm` | endpoint | 28/29 | 428 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/deepseek-v4-flash-vision-exp-keys.vllm--policy-29.md) |
| `qwen3-vl-reranker-8b.llamacpp` | endpoint | 26/29 | 68 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/qwen3-vl-reranker-8b.llamacpp--policy-29.md) |
| `qwen3-vl-reranker-2b.llamacpp` | endpoint | 21/29 | 49 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/qwen3-vl-reranker-2b.llamacpp--policy-29.md) |
| `qwen3-embedding-0.6b.local` | embedding | 19/29 | 8 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/qwen3-embedding-0.6b.local--policy-29.md) |
| `bart-large-mnli.local` | encoder | 15/29 | 7 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/bart-large-mnli.local--policy-29.md) |

**policy-hard-52**, written for jevify before any model saw it: instruction
flips, negations, two-threshold policies, distractors, and long transcripts
with the decisive line buried (`tools/author_policy_hard.py`):

| Recipe | Accuracy | Flips | Thresholds | Long states | Scorecard |
|---|---|---|---|---|---|
| `deepseek-v4-flash-vision-exp-keys.vllm` | 47/52 | 6/9 | 16/18 | 12/12 | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/deepseek-v4-flash-vision-exp-keys.vllm--policy-hard-52.md) |
| `ternary-bonsai-2-27b.llamacpp` | 46/52 | 6/9 | 15/18 | 12/12 | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/ternary-bonsai-2-27b.llamacpp--policy-hard-52.md) |
| `gemma-4-e4b-it.llamacpp` | 45/52 | 5/9 | 16/18 | 12/12 | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/gemma-4-e4b-it.llamacpp--policy-hard-52.md) |
| `qwen3-vl-reranker-2b.llamacpp` | 34/52 | 3/9 | 9/18 | 12/12 | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/qwen3-vl-reranker-2b.llamacpp--policy-hard-52.md) |
| `bart-large-mnli.local` | 28/52 | 3/9 | 8/18 | 12/12 | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/bart-large-mnli.local--policy-hard-52.md) |
| `qwen3-embedding-0.6b.local` | 24/52 | 3/9 | 8/18 | 6/12 | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/qwen3-embedding-0.6b.local--policy-hard-52.md) |

Every generative model misses the same inverted instructions ("the action the
customer asked NOT to take"), a weakness the author's earlier experiments found across model
generations. Negations in the state and long transcripts are solved.

**doom-frames-41**, 164 questions over 41 ViZDoom screenshots at 640×480,
every label from the engine's labels buffer or a fixed expert rule
(`tools/record_doom_suite.py`; frames regenerate byte-identically and their
hashes are pinned in the manifest):

| Recipe | Enemy visible | Enemy count | Enemy side | Expert button | Median latency | Scorecard |
|---|---|---|---|---|---|---|
| `deepseek-v4-flash-vision-exp-keys.vllm` | 0.88 | 0.83 | 0.56 | 0.24 | 672 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/deepseek-v4-flash-vision-exp-keys.vllm--doom-frames-41.md) |
| `ternary-bonsai-2-27b.llamacpp` | 0.93 | 0.85 | 0.37 | 0.24 | 97 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/ternary-bonsai-2-27b.llamacpp--doom-frames-41.md) |
| `gemma-4-e4b-it.llamacpp` | 0.78 | 0.49 | 0.41 | 0.20 | 36 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/gemma-4-e4b-it.llamacpp--doom-frames-41.md) |
| `qwen3-vl-reranker-2b.llamacpp` | 0.10 | 0.56 | 0.41 | 0.10 | 64 ms | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/qwen3-vl-reranker-2b.llamacpp--doom-frames-41.md) |

Presence and count are perception; the expert button asks the model to agree
with a rule it is not told, so that column is a floor for a scripted policy,
not a skill score. Enemy side is where the vision models diverge most.

**long-state-24**, one decisive customer line at the start, middle or end of
a filler transcript of about 2k, 8k, 16k or 32k tokens
(`tools/author_long_state.py`). This is the suite the warm step exists for:
the state is sent once, and each question then costs only its own tokens.

| Recipe | 2k | 8k | 16k | 32k | Warm at 32k | Per question after warm | Scorecard |
|---|---|---|---|---|---|---|---|
| `deepseek-v4-flash-vision-exp-keys.vllm` | 6/6 | 6/6 | 6/6 | 6/6 | 20 s | 359 ms, 31,232 tokens cached | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/deepseek-v4-flash-vision-exp-keys.vllm--long-state-24.md) |
| `ternary-bonsai-2-27b.llamacpp` | 6/6 | 6/6 | 6/6 | 6/6 | 13 s | 136 ms, 33,946 tokens cached | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/ternary-bonsai-2-27b.llamacpp--long-state-24.md) |
| `gemma-4-e4b-it.llamacpp` | 6/6 | context | context | context | | 37 ms at 2k | [summary](https://github.com/fidecastro/jevify/blob/main/docs/evidence/gemma-4-e4b-it.llamacpp--long-state-24.md) |

"context" means the launch's slot context was shorter than the state and the
case recorded a context error, which is the finding for that launch (Gemma
was launched with two 8k slots; Bonsai first ran with two 16k slots and
failed 16k and 32k the same way, then with one 64k slot for the row above). No model that could read the state missed
the decisive line at any position.

These suites rank models; they do not measure calibration, which is why
`jevify calibrate` refused to write a table for any of them.

## Play Doom

`tools/play_doom.py` lets a recipe play ViZDoom from screenshots: one frame
per decision, one question per frame, the chosen button pressed, a video with
the decision and its latency burned in. `tools/record_doom_suite.py` records
the frozen perception suite the same way. Both need `uv sync --extra games`
and ffmpeg.

## Work with your agent

jevify is written to be operated through a coding agent. Point yours at
[`SKILL.md`](https://github.com/fidecastro/jevify/blob/main/SKILL.md): the workflow, recipe anatomy per kind, how to read
`x_jevify`, and what a result may claim. Typical asks:

- "Write a recipe for the model behind `http://host:8000/v1`, probe it, and
  tell me which readout rung it landed on and why."
- "Run `policy-hard-52` against this recipe with controls and read me the
  scorecard against the claims contract."
- "Serve this recipe and show me an unmodified typesafe-sdk call against it."

## Repository map

| Path | What it holds |
|---|---|
| [`jevify/`](https://github.com/fidecastro/jevify/blob/main/jevify) | The package: domain, ports, adapters (endpoint, embedding, encoder, rerank), recipes, evaluation, Jev API, CLI |
| [`recipes/`](https://github.com/fidecastro/jevify/blob/main/recipes) | One recipe per model deployment, with its probe results and launch line |
| [`suites/`](https://github.com/fidecastro/jevify/blob/main/suites) | Frozen, hashed suites and their manifests |
| [`docs/evidence/`](https://github.com/fidecastro/jevify/blob/main/docs/evidence) | Committed scorecards, one per recipe and suite |
| [`docs/00-invariants.md`](https://github.com/fidecastro/jevify/blob/main/docs/00-invariants.md) | What jevify is and refuses to be; the claims contract. Ranks above everything else |
| [`docs/adr/`](https://github.com/fidecastro/jevify/blob/main/docs/adr/README.md) | The decision ledger |
| [`tools/`](https://github.com/fidecastro/jevify/blob/main/tools) | Suite authoring, the Doom recorder and player |
| [`AGENTS.md`](https://github.com/fidecastro/jevify/blob/main/AGENTS.md) | The engineering principles that bind every change |
| [`SKILL.md`](https://github.com/fidecastro/jevify/blob/main/SKILL.md) | How an agent operates jevify |
| [`docs/playbook.md`](https://github.com/fidecastro/jevify/blob/main/docs/playbook.md) | How a person operates jevify: ask, read an answer, serve, add a model, fix common problems |
| [`Makefile`](https://github.com/fidecastro/jevify/blob/main/Makefile) | `make ask` and `make serve` shortcuts; `make help` lists the variables and recipes |

To work on jevify: `uv sync --group dev` (add `--extra encoder --extra
games` as needed) and `uv run pytest`. Live and in-process tests skip unless
you opt in.

## Not yet

Named so that nobody has to discover it: a direct-call control, so no
"faster than generation" claim exists yet; clearing Doom's `deadly_corridor`
(in `defend_the_center` Bonsai's perception plus a fixed rule scores 10 kills
in 16 seconds, but in the corridor even the rule fed the engine's own labels
dies before the armour, so the corridor needs a policy with memory, not
better perception); the rerank kind and the
embedding route against a live server; a marker-slot encoder layout; the
grammar rung on the PrismML llama.cpp fork, which accepts a grammar without
constraining the reported probabilities; the `prompt_logprobs` rung on any
server but TensorFold's native one with GLM-5.3-Flash, where it has been
probed and asked by hand (raw and chat recipes, text and images) but no
evaluation suite has been scored on it yet (that server skips its prefix
cache for such requests, so every question reads the whole state again,
images included, which the probe measures and notes); Windows. Choice menus on the endpoint kind are bounded by the
single-token identifier alphabet the probe verifies (26 on the DeepSeek
recipe); Jev's 255-option ceiling is not reachable there.

## Licence

[MIT](https://github.com/fidecastro/jevify/blob/main/LICENSE).
