Metadata-Version: 2.5
Name: jev-lite
Version: 0.0.9
Summary: OpenAI-style sync and async Pydantic client for TypeSafe Jev (System One).
Project-URL: Documentation, https://docs.typesafe.ai/
Project-URL: Local model, https://huggingface.co/dockhardman/d3-nano-onnx
Author-email: Allen Chou <f1470891079@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: jev,openai,opentelemetry,pydantic,system-one,typesafe
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: httpx>=0.23
Requires-Dist: opentelemetry-api>=1.20
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20
Requires-Dist: opentelemetry-sdk>=1.20
Requires-Dist: pydantic>=2
Provides-Extra: cache
Requires-Dist: diskcache>=5.6; extra == 'cache'
Provides-Extra: distill
Requires-Dist: datasets>=2; extra == 'distill'
Provides-Extra: local
Requires-Dist: numpy>=1.24; extra == 'local'
Requires-Dist: onnxruntime>=1.24; extra == 'local'
Requires-Dist: tokenizers>=0.15; extra == 'local'
Provides-Extra: ml
Requires-Dist: accelerate>=1.0; extra == 'ml'
Requires-Dist: pillow>=9; extra == 'ml'
Requires-Dist: torch>=2.1; extra == 'ml'
Requires-Dist: torchvision>=0.16; extra == 'ml'
Requires-Dist: transformers>=4.40; extra == 'ml'
Description-Content-Type: text/markdown

# jev-lite

OpenAI-style Python client for [TypeSafe Jev](https://typesafe.ai/) (System One).

Jev does not generate text. Send `state` plus typed questions; get structured
decisions with probabilities. `jev-lite` is the official TypeSafe HTTP client — sync
and async, same shape as the OpenAI Python SDK. `jev-lite@0.0.2` added Dialogflow-style
system entities; `jev-lite@0.0.3` adds a privacy filter and a reranker (all
three are previews, below); `jev-lite@0.0.4` adds images and Box answers
for Jev 1.14; `jev-lite@0.0.5` adds `LocalTypeSafe`, the same API answered by
an open model on your CPU with no API key; `jev-lite@0.0.6` adds
`OpenAIDecisions`, the same API answered by OpenAI's Decisions API;
`jev-lite@0.0.7` makes `LocalTypeSafe` read the state once per request, and
measures [latency by question count](#latency-by-question-count) for every
model on the board; `jev-lite@0.0.8` keeps the board to a core set of 15
models; `jev-lite@0.0.9` keeps prompts and results out of tracing spans by
default.

```bash
pip install jev-lite
# or: uv add jev-lite
```

Python 3.11+. Set `TYPESAFE_API_KEY`. Docs: <https://docs.typesafe.ai/>

## Usage

```python
from jev_lite import Choice, Noul, Score, TypeSafe

client = TypeSafe()  # TYPESAFE_API_KEY

result = client.system_one.create(
    state="I was charged twice. Please fix this ASAP.",
    questions={
        "billing": Noul(instructions="Is this ticket about billing?"),
        "tone": Choice(
            instructions="What is the customer's tone?",
            criteria={"calm": None, "frustrated": None, "angry": None},
        ),
        "urgency": Score(
            instructions="How urgent is this ticket?",
            criteria=["can wait", "this week", "today"],
        ),
    },
)

print(result.nouls["billing"].noul)
print(result.choices["tone"].choice)
print(result.scores["urgency"].score)
print(result.elapsed_ms, result.request_id, result.usage)
print(client.models.list())  # GET /v1/models
```

Async is the same surface with `await`:

```python
from jev_lite import AsyncTypeSafe, Noul

async with AsyncTypeSafe() as client:
    result = await client.system_one.create(
        state="I was charged twice. Please fix this ASAP.",
        questions={"billing": Noul(instructions="Is this ticket about billing?")},
    )
    models = await client.models.list()
```

`jev-latest` is an alias that moves: on 2026-10-21 it answers as
`jev-1.13.0`, and `jev-1.14.0` is opt-in. `client.models.list()` lists only
the aliases (`jev-latest`, `jev-preview`); `response.model` names the version
that answered. Pin a version with `TypeSafe(model="jev-1.14.0")` (or
`TYPESAFE_DEFAULT_MODEL`) when answers must repeat. Prefer one
`system_one.create` with many questions over a loop of calls. From 0.0.7 the
modules are pinned to `jev-1.14.0` and use the client's `model`. On the same
day, 1.14 matched or beat 1.13 on every module below, and scored lower on
hard semantic decisions (461 vs 481 of 504 on `semantic-decisions@v0.0.2`).

Re-running unchanged questions can replay answers from disk
(`pip install "jev-lite[cache]"`):

```python
client = TypeSafe(model="jev-1.14.0", cache="~/.cache/jev")  # or a diskcache.Cache
```

Only 200 responses are stored, keyed on the request body (which carries the
model id). Pin a version: an alias such as `jev-latest` can move to a new model
while the cache keeps replaying the old one, so a cached alias warns. Leave the
cache off when you measure repeat or order stability.

## Local decisions (CPU, no API key)

`LocalTypeSafe` answers the same `system_one.create` calls on this machine's
CPU with [`vllm-sr/d3-nano`](https://huggingface.co/vllm-sr/d3-nano) (2.2B,
Apache-2.0), exported to ONNX Runtime with int8 weights. No torch, no
transformers, no remote code.

```bash
pip install "jev-lite[local]"   # about 40 MB download, 150 MB installed
```

```python
from jev_lite import Choice, LocalTypeSafe

client = LocalTypeSafe()  # model "vllm-sr/d3-nano"
client.load()             # optional warm-up; the first run downloads 1.76 GiB
result = client.system_one.create(
    state="包裹顯示已送達，可是我家門口什麼都沒有。",
    questions={"route": Choice(criteria={"billing": "付款", "shipping": "配送"})},
)
print(result.choices["route"].choice)  # shipping
```

- **Weights:** [`dockhardman/d3-nano-onnx`](https://huggingface.co/dockhardman/d3-nano-onnx)
  at a pinned commit, each file checked against its sha256 and cached in
  `~/.cache/jev-lite` (`cache_dir=`); after that it runs offline.
- **Same API:** `AsyncLocalTypeSafe`, `models.list()`, `cache=` and the
  tracing spans work as for `TypeSafe`. Text only: an image or a `Box`
  question raises `BadRequestError`; a question over 8,192 tokens is
  refused, never truncated.
- **Quality:** 449/504 on the zh-Hant `semantic-decisions@v0.0.2` (hard 296/336), against
  481 for hosted `jev-1.13.0`, with the same choice as the original bf16
  model on all 504. The modules below are measured on hosted Jev only.
- **Speed:** one question at a time on 20 Arm cores: a median of 0.5 s on
  short questions and 1.1 s on the bank's prompts, about 5 s at 1,100 tokens.

## OpenAI Decisions

`OpenAIDecisions` sends the same `system_one.create` calls to OpenAI's
Decisions API (`POST /v1/decisions`). It needs no extra install; only the
client class changes.

```python
from jev_lite import Choice, OpenAIDecisions

client = OpenAIDecisions()  # OPENAI_API_KEY; model "gpt-6-luna"
result = client.system_one.create(
    state="包裹顯示已送達，可是我家門口什麼都沒有。",
    questions={"route": Choice(criteria={"billing": "付款", "shipping": "配送"})},
)
print(result.choices["route"].choice)  # shipping
```

- **Mapping:** `Noul` → predicate (its `true`/`false` texts are appended to
  the instructions), `Choice` → choice, and `Score` → score levels. A
  question without instructions gets a short default. `state` that is not a
  string is sent as compact JSON text.
- **Images:** an `Image` in `state` or `instructions` is sent as an image
  part. An image in `criteria` and a `Box` question raise `BadRequestError`
  before anything is sent, as do counts the API refuses (1–400 questions,
  2–255 options, 2–10 levels).
- **Same API:** `AsyncOpenAIDecisions`, `cache=`, typed errors, `request_id`,
  `usage` and the tracing spans work as for `TypeSafe`. `models.list()`
  returns the `luna` models.
- **Quality:** `gpt-6-luna` scored 442/504 on the zh-Hant
  `semantic-decisions@v0.0.2` (hard 275/336), against 481 for Jev
  `jev-1.13.0` and 449 for `LocalTypeSafe`. It changed its answer with option
  order on 20 of 252 rows.

## Images and Box (Jev 1.14, experimental)

Put an `Image` anywhere a value can go: in `state`, `instructions` or
`criteria`. The client uploads it as an asset and adds the
`TypeSafe-Danger-Zone-Images` header. A `Box` answer is where a target is,
as edges in fractions of the image; it needs `jev-1.14.0`.

```python
from pathlib import Path
from jev_lite import Box, Image, Noul, TypeSafe

with TypeSafe(model="jev-1.14.0") as client:
    result = client.system_one.create(
        state={"photo": Image(Path("kitchen.jpg"))},  # PNG/JPEG, sides 280-1568 px
        questions={
            "has_mug": Noul(instructions="Is there a coffee mug in `photo`?"),
            "mug": Box(instructions="the coffee mug in `photo`"),
        },
    )

mug = result.boxes["mug"]
print(mug.box.left, mug.box.top, mug.box.right, mug.box.bottom, mug.confidence)
```

A Box always returns a box, so gate it on the Noul. An image costs about
`2·⌈w/28⌉·⌈h/28⌉` input tokens. Measured limits (targets under ~40 px,
counting past five, text under 7 px); the benchmark is
`image-questions@v0.0.1`.

## System entities (preview)

`jev_lite.system_entities` extracts the 59 Dialogflow ES system entities
(`@sys.date-time`, `@sys.unit-currency`, `@sys.geo-city`, …) and fills
typed slots, in ten languages (en and zh-Hant primary). Code proposes
candidates, Jev decides (the client's `model`), code normalizes. **Preview:** the API
may change within `0.0.x`. Same install; no extra dependency.

```python
from datetime import datetime
from zoneinfo import ZoneInfo

from jev_lite import TypeSafe
from jev_lite.system_entities import extract

now = datetime(2026, 10, 1, 16, 0, tzinfo=ZoneInfo("Asia/Taipei"))
with TypeSafe() as client:
    result = extract(client, "Book a table for 4 next Friday at 7pm", ["date-time", "number"], now=now)
for match in result.matches:
    print(match.entity, match.span.text, match.value)  # number 4 …; date-time … 2026-10-09 19:00
```

Slot filling (`extract_slots`), async (`aextract`, `aextract_slots`) and the
value models are documented in the module's docstrings. Extraction is not a
security boundary, `probability` is not calibrated, and th, ms and other
languages outside the ten get few matches. On the held-out set
`system-entities-cases@v0.0.1` (301 cases, ten languages) `jev-lite@0.0.5` (unchanged in 0.0.6) is
98.9% exact on accepted values with 98.9% coverage on `jev-1.13.0`
(`jev-lite@0.0.4`: 98.9% on `jev-1.13.0`, 99.3% on `jev-1.14.0`).

## Privacy filter and reranker (preview)

`jev_lite.privacy_filter` and `jev_lite.reranker` ship in `jev-lite@0.0.3`
as previews: the APIs may change within `0.0.x`. Import them as modules
(`from jev_lite.privacy_filter import detect`, `from jev_lite.reranker import
rerank`). Both use the client's `model` and are pinned to `jev-1.14.0` from
0.0.7; the numbers are on `jev-1.13.0`, with the same-day 1.14 run beside them.

**Privacy filter**: private-span detection. Public multilingual gold
(ai4privacy OpenPII, Gretel finance, NVIDIA Nemotron-PII), 300 test texts,
exact-span precision / recall:

| Setup | P / R | Over-redaction | Speed |
| --- | --- | --- | --- |
| [openai/privacy-filter](https://huggingface.co/openai/privacy-filter), local GPU | 63.4 / 62.9 | 5.2% | ~20 ms per text |
| openai/privacy-filter proposes, Jev decides | 75.4 / 60.1 | 2.5% | p50 210 ms |
| Jev only, no GPU (broader proposals, per-label cuts) | 68.6 / 60.5 | 2.4% | p50 215 ms |
| Jev only, round-04 default | 80.4 / 57.6 | 1.5% | p50 210 ms |
| Jev only, before the person proposer | 78.3 / 63.6 | 1.6% | p50 209 ms |
| **Jev only, default `detect(client, text)`**, no GPU | **79.3 / 69.2** | **1.6%** | p50 ~0.7 s |
| Same default on `jev-1.14.0` (same-day 1.13: 79.7 / 69.7, 1.6%) | 82.4 / 68.2 | 1.3% | call p50 0.19 s |

The default uses per-label proposers and cuts fitted on dev, and finds
people with `system_entities` person spans plus a person detect Noul (person
recall 43.9 → 71.9), with an order-stable name boundary. On overlap it
scores 88.9 / 77.6; exact address recall is 0 because the gold splits an
address into components while the filter masks the whole address. It fails
most written robustness bars, so it is a preview. Hosted Jev sees the raw
text before masking.

**Reranker**: batched evidence Noul over a first-stage pool (balanced
batches of at most 30, independent of the caller's order), nDCG@10:

| Bank | First stage | Reranked | Query p50 | Same day, 1.13 → 1.14 |
| --- | --- | --- | --- | --- |
| SciFact (50 q) | .556 | .638 | 0.30 s | .635 → .634 |
| FiQA (50 q) | .219 | .346 | 0.26 s | .346 → .348 |
| zh, BM25 (50 q) | .526 | .587 | 0.30 s | .603 → .618 |
| zh, E5 (50 q) | .551 | .647 | 0.27 s | .640 → .656 |
| Mr. TyDi ko / ja / ar (30 q each) | .415 / .605 / .529 | .675 / .791 / .754 | 0.23–0.24 s | .687 / .791 / .741 → .675 / .808 / .741 |

A batch score is relative to the other passages in its request, not a
standalone probability. A passage that echoes the relevance criteria can be
lifted into the top 10; `rerank(..., check_echo=True)` (off by default)
re-checks the top-k and moves such passages down, at about +0.23 s.

## Tracing

Off by default. Opt in and pick the destination:

```python
from jev_lite import instrument

instrument(endpoint="http://127.0.0.1:4318")  # local Grafana LGTM
```

Or set `OTEL_EXPORTER_OTLP_ENDPOINT` and call `instrument()`, or install your
own `TracerProvider` first (Logfire, etc.). `jev-lite` then writes to that
provider even if you never call `instrument()`.

Spans hold metadata: model, token usage, latency, request id, URL without
credentials, question counts and ids, and the error type and status code.
Question ids are your own keys, so keep personal data out of them. From
`jev-lite@0.0.9` spans leave out prompts, results and error text, which may
hold personal data. To record those in plain text:

```python
instrument(endpoint="http://127.0.0.1:4318", capture_content=True)
```

With your own `TracerProvider`, call `instrument(capture_content=True)` with no
destination; naming one raises an error, because content would go to your
provider instead. The setting is process-wide and the last call wins.

Request images stay out of spans, unless a server's error text quotes them and
capture is on. Your own logging or spans around a call can still store an
exception's text, which may quote the request or the response.

## Upgrading from 0.0.1–0.0.5

- **Pin the model.** `jev-latest` moves to new versions, and answers and
  confidence shift with it. Use `TypeSafe(model="jev-1.14.0")` (0.0.7; it was
  `jev-1.13.0` before), and refit any confidence cut when you move.
  A cached client asking for an alias warns (0.0.6).
- **`Noul(true=..., false=...)`** works from 0.0.5. Before that, the
  shorthand was silently ignored; use `criteria=NoulCriteria(...)` there.
  Unknown question fields warn from 0.0.5.
- **Replay cache:** `TypeSafe(cache="dir")` (0.0.3, `jev-lite[cache]`)
  replaces hand-made caches. `jev_lite.eval` is research code, not a stable
  API.
- **Confidence** comes with every Choice and Score answer; do not recompute
  it. From 0.0.6, `TypeSafe`, `LocalTypeSafe` and `OpenAIDecisions` all
  report `(n·p_max − 1)/(n − 1)`.
- **Other backends behind the same calls:** `LocalTypeSafe` (0.0.5, CPU, no
  key) and `OpenAIDecisions` (0.0.6). A self-hosted server that speaks
  `/v1/systemone` works through `TypeSafe(base_url=...)`.
- **Modules** (`system_entities`, `privacy_filter`, `reranker`) use the
  client's model from 0.0.5. From 0.0.7 they are pinned to `jev-1.14.0`;
  their tables give 1.13 and 1.14 from the same day.

## Develop

```bash
uv sync --dev
uv run ruff check . && uv run pytest
```

The supported public API is the TypeSafe client. Optional local engines live
under `jev_lite.engine` (`pip install 'jev-lite[ml]'`).

## Semantic decisions v0.0.2

Dataset `semantic-decisions@v0.0.2`, versioned separately from the package.

252 frozen zh-Hant rows
in 18 families (36 easy, 48 medium, 168 hard), gold derived from visible text.
This natural zh-Hant v1 qualified freeze **is** v0.0.2, and it is the only
semantic-decision eval in this repo. Every model answers the same 504 locked
requests: each row in canonical and left-rotated option order. Charts are
percent correct. The families were seen during development before the
freeze, so this is not unseen confirmation.

From 0.0.8 the charts and tables show a core set of 15 models: each size
class (hosted, ≤1B, 2–3B, 4B, 9B, 26–35B) with its best model, plus models
from well-known makers even when they score lower. Models clearly behind
others of their size are no longer re-measured. The 0.0.7 README on PyPI
lists every model measured (45 scored, 53 timed).

[![All 504](https://quickchart.io/chart?version=3&w=640&h=430&bkg=white&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%22d3%2027B%22%2C%22AutoTrust%20v1.1%22%2C%22pplx%2027B%20v1.1%22%2C%22d3-flash%209B%22%2C%22d3-mini%204B%22%2C%22Jev%201.14.0%22%2C%22pplx%20flash%204B%22%2C%22d3-nano%202B%22%2C%22Nimble%209B%20v3%22%2C%22OpenAI%20gpt-6-luna%22%2C%22h2o%20lightning%204B%22%2C%22Clef%2027B%22%2C%22Clef-Flash%209B%22%2C%22d3-lite%200.8B%22%2C%22Strands%202B%22%5D%2C%22datasets%22%3A%5B%7B%22data%22%3A%5B98.8%2C98.4%2C96.2%2C95.2%2C94.6%2C91.5%2C89.5%2C89.1%2C89.1%2C87.7%2C86.7%2C80.2%2C76.2%2C70.0%2C69.8%5D%2C%22backgroundColor%22%3A%5B%22%23b91c1c%22%2C%22%23e879f9%22%2C%22%23334155%22%2C%22%23ef4444%22%2C%22%23f97316%22%2C%22%2314b8a6%22%2C%22%2364748b%22%2C%22%23f59e0b%22%2C%22%230e7490%22%2C%22%2310a37f%22%2C%22%23facc15%22%2C%22%23059669%22%2C%22%2310b981%22%2C%22%23fdba74%22%2C%22%2384cc16%22%5D%2C%22borderWidth%22%3A0%2C%22barPercentage%22%3A0.82%7D%5D%7D%2C%22options%22%3A%7B%22indexAxis%22%3A%22y%22%2C%22layout%22%3A%7B%22padding%22%3A%7B%22right%22%3A40%7D%7D%2C%22plugins%22%3A%7B%22legend%22%3A%7B%22display%22%3Afalse%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22All%20504%22%2C%22font%22%3A%7B%22size%22%3A16%7D%7D%2C%22datalabels%22%3A%7B%22display%22%3Atrue%2C%22anchor%22%3A%22end%22%2C%22align%22%3A%22end%22%2C%22clamp%22%3Atrue%2C%22color%22%3A%22%23111827%22%2C%22font%22%3A%7B%22size%22%3A11%7D%2C%22formatter%22%3Afunction%28v%29%7Breturn%20v.toFixed%281%29%7D%7D%7D%2C%22scales%22%3A%7B%22x%22%3A%7B%22min%22%3A0%2C%22max%22%3A100%2C%22ticks%22%3A%7B%22stepSize%22%3A20%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%25%22%7D%7D%2C%22y%22%3A%7B%22grid%22%3A%7B%22display%22%3Afalse%7D%7D%7D%7D%7D)](https://quickchart.io/chart?version=3&w=640&h=430&bkg=white&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%22d3%2027B%22%2C%22AutoTrust%20v1.1%22%2C%22pplx%2027B%20v1.1%22%2C%22d3-flash%209B%22%2C%22d3-mini%204B%22%2C%22Jev%201.14.0%22%2C%22pplx%20flash%204B%22%2C%22d3-nano%202B%22%2C%22Nimble%209B%20v3%22%2C%22OpenAI%20gpt-6-luna%22%2C%22h2o%20lightning%204B%22%2C%22Clef%2027B%22%2C%22Clef-Flash%209B%22%2C%22d3-lite%200.8B%22%2C%22Strands%202B%22%5D%2C%22datasets%22%3A%5B%7B%22data%22%3A%5B98.8%2C98.4%2C96.2%2C95.2%2C94.6%2C91.5%2C89.5%2C89.1%2C89.1%2C87.7%2C86.7%2C80.2%2C76.2%2C70.0%2C69.8%5D%2C%22backgroundColor%22%3A%5B%22%23b91c1c%22%2C%22%23e879f9%22%2C%22%23334155%22%2C%22%23ef4444%22%2C%22%23f97316%22%2C%22%2314b8a6%22%2C%22%2364748b%22%2C%22%23f59e0b%22%2C%22%230e7490%22%2C%22%2310a37f%22%2C%22%23facc15%22%2C%22%23059669%22%2C%22%2310b981%22%2C%22%23fdba74%22%2C%22%2384cc16%22%5D%2C%22borderWidth%22%3A0%2C%22barPercentage%22%3A0.82%7D%5D%7D%2C%22options%22%3A%7B%22indexAxis%22%3A%22y%22%2C%22layout%22%3A%7B%22padding%22%3A%7B%22right%22%3A40%7D%7D%2C%22plugins%22%3A%7B%22legend%22%3A%7B%22display%22%3Afalse%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22All%20504%22%2C%22font%22%3A%7B%22size%22%3A16%7D%7D%2C%22datalabels%22%3A%7B%22display%22%3Atrue%2C%22anchor%22%3A%22end%22%2C%22align%22%3A%22end%22%2C%22clamp%22%3Atrue%2C%22color%22%3A%22%23111827%22%2C%22font%22%3A%7B%22size%22%3A11%7D%2C%22formatter%22%3Afunction%28v%29%7Breturn%20v.toFixed%281%29%7D%7D%7D%2C%22scales%22%3A%7B%22x%22%3A%7B%22min%22%3A0%2C%22max%22%3A100%2C%22ticks%22%3A%7B%22stepSize%22%3A20%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%25%22%7D%7D%2C%22y%22%3A%7B%22grid%22%3A%7B%22display%22%3Afalse%7D%7D%7D%7D%7D)

[![Easy](https://quickchart.io/chart?version=3&w=640&h=430&bkg=white&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%22Strands%202B%22%2C%22Clef-Flash%209B%22%2C%22Clef%2027B%22%2C%22Jev%201.14.0%22%2C%22d3-mini%204B%22%2C%22d3-flash%209B%22%2C%22d3%2027B%22%2C%22AutoTrust%20v1.1%22%2C%22pplx%20flash%204B%22%2C%22h2o%20lightning%204B%22%2C%22pplx%2027B%20v1.1%22%2C%22Nimble%209B%20v3%22%2C%22OpenAI%20gpt-6-luna%22%2C%22d3-nano%202B%22%2C%22d3-lite%200.8B%22%5D%2C%22datasets%22%3A%5B%7B%22data%22%3A%5B100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C97.2%2C97.2%5D%2C%22backgroundColor%22%3A%5B%22%2384cc16%22%2C%22%2310b981%22%2C%22%23059669%22%2C%22%2314b8a6%22%2C%22%23f97316%22%2C%22%23ef4444%22%2C%22%23b91c1c%22%2C%22%23e879f9%22%2C%22%2364748b%22%2C%22%23facc15%22%2C%22%23334155%22%2C%22%230e7490%22%2C%22%2310a37f%22%2C%22%23f59e0b%22%2C%22%23fdba74%22%5D%2C%22borderWidth%22%3A0%2C%22barPercentage%22%3A0.82%7D%5D%7D%2C%22options%22%3A%7B%22indexAxis%22%3A%22y%22%2C%22layout%22%3A%7B%22padding%22%3A%7B%22right%22%3A40%7D%7D%2C%22plugins%22%3A%7B%22legend%22%3A%7B%22display%22%3Afalse%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22Easy%22%2C%22font%22%3A%7B%22size%22%3A16%7D%7D%2C%22datalabels%22%3A%7B%22display%22%3Atrue%2C%22anchor%22%3A%22end%22%2C%22align%22%3A%22end%22%2C%22clamp%22%3Atrue%2C%22color%22%3A%22%23111827%22%2C%22font%22%3A%7B%22size%22%3A11%7D%2C%22formatter%22%3Afunction%28v%29%7Breturn%20v.toFixed%281%29%7D%7D%7D%2C%22scales%22%3A%7B%22x%22%3A%7B%22min%22%3A0%2C%22max%22%3A100%2C%22ticks%22%3A%7B%22stepSize%22%3A20%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%25%22%7D%7D%2C%22y%22%3A%7B%22grid%22%3A%7B%22display%22%3Afalse%7D%7D%7D%7D%7D)](https://quickchart.io/chart?version=3&w=640&h=430&bkg=white&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%22Strands%202B%22%2C%22Clef-Flash%209B%22%2C%22Clef%2027B%22%2C%22Jev%201.14.0%22%2C%22d3-mini%204B%22%2C%22d3-flash%209B%22%2C%22d3%2027B%22%2C%22AutoTrust%20v1.1%22%2C%22pplx%20flash%204B%22%2C%22h2o%20lightning%204B%22%2C%22pplx%2027B%20v1.1%22%2C%22Nimble%209B%20v3%22%2C%22OpenAI%20gpt-6-luna%22%2C%22d3-nano%202B%22%2C%22d3-lite%200.8B%22%5D%2C%22datasets%22%3A%5B%7B%22data%22%3A%5B100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C97.2%2C97.2%5D%2C%22backgroundColor%22%3A%5B%22%2384cc16%22%2C%22%2310b981%22%2C%22%23059669%22%2C%22%2314b8a6%22%2C%22%23f97316%22%2C%22%23ef4444%22%2C%22%23b91c1c%22%2C%22%23e879f9%22%2C%22%2364748b%22%2C%22%23facc15%22%2C%22%23334155%22%2C%22%230e7490%22%2C%22%2310a37f%22%2C%22%23f59e0b%22%2C%22%23fdba74%22%5D%2C%22borderWidth%22%3A0%2C%22barPercentage%22%3A0.82%7D%5D%7D%2C%22options%22%3A%7B%22indexAxis%22%3A%22y%22%2C%22layout%22%3A%7B%22padding%22%3A%7B%22right%22%3A40%7D%7D%2C%22plugins%22%3A%7B%22legend%22%3A%7B%22display%22%3Afalse%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22Easy%22%2C%22font%22%3A%7B%22size%22%3A16%7D%7D%2C%22datalabels%22%3A%7B%22display%22%3Atrue%2C%22anchor%22%3A%22end%22%2C%22align%22%3A%22end%22%2C%22clamp%22%3Atrue%2C%22color%22%3A%22%23111827%22%2C%22font%22%3A%7B%22size%22%3A11%7D%2C%22formatter%22%3Afunction%28v%29%7Breturn%20v.toFixed%281%29%7D%7D%7D%2C%22scales%22%3A%7B%22x%22%3A%7B%22min%22%3A0%2C%22max%22%3A100%2C%22ticks%22%3A%7B%22stepSize%22%3A20%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%25%22%7D%7D%2C%22y%22%3A%7B%22grid%22%3A%7B%22display%22%3Afalse%7D%7D%7D%7D%7D)

[![Medium](https://quickchart.io/chart?version=3&w=640&h=430&bkg=white&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%22Jev%201.14.0%22%2C%22d3%2027B%22%2C%22AutoTrust%20v1.1%22%2C%22pplx%2027B%20v1.1%22%2C%22Nimble%209B%20v3%22%2C%22OpenAI%20gpt-6-luna%22%2C%22Clef%2027B%22%2C%22d3-mini%204B%22%2C%22d3-flash%209B%22%2C%22pplx%20flash%204B%22%2C%22h2o%20lightning%204B%22%2C%22Clef-Flash%209B%22%2C%22d3-nano%202B%22%2C%22d3-lite%200.8B%22%2C%22Strands%202B%22%5D%2C%22datasets%22%3A%5B%7B%22data%22%3A%5B100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C99.0%2C97.9%2C97.9%2C97.9%2C96.9%2C96.9%2C93.8%2C86.5%2C80.2%2C76.0%5D%2C%22backgroundColor%22%3A%5B%22%2314b8a6%22%2C%22%23b91c1c%22%2C%22%23e879f9%22%2C%22%23334155%22%2C%22%230e7490%22%2C%22%2310a37f%22%2C%22%23059669%22%2C%22%23f97316%22%2C%22%23ef4444%22%2C%22%2364748b%22%2C%22%23facc15%22%2C%22%2310b981%22%2C%22%23f59e0b%22%2C%22%23fdba74%22%2C%22%2384cc16%22%5D%2C%22borderWidth%22%3A0%2C%22barPercentage%22%3A0.82%7D%5D%7D%2C%22options%22%3A%7B%22indexAxis%22%3A%22y%22%2C%22layout%22%3A%7B%22padding%22%3A%7B%22right%22%3A40%7D%7D%2C%22plugins%22%3A%7B%22legend%22%3A%7B%22display%22%3Afalse%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22Medium%22%2C%22font%22%3A%7B%22size%22%3A16%7D%7D%2C%22datalabels%22%3A%7B%22display%22%3Atrue%2C%22anchor%22%3A%22end%22%2C%22align%22%3A%22end%22%2C%22clamp%22%3Atrue%2C%22color%22%3A%22%23111827%22%2C%22font%22%3A%7B%22size%22%3A11%7D%2C%22formatter%22%3Afunction%28v%29%7Breturn%20v.toFixed%281%29%7D%7D%7D%2C%22scales%22%3A%7B%22x%22%3A%7B%22min%22%3A0%2C%22max%22%3A100%2C%22ticks%22%3A%7B%22stepSize%22%3A20%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%25%22%7D%7D%2C%22y%22%3A%7B%22grid%22%3A%7B%22display%22%3Afalse%7D%7D%7D%7D%7D)](https://quickchart.io/chart?version=3&w=640&h=430&bkg=white&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%22Jev%201.14.0%22%2C%22d3%2027B%22%2C%22AutoTrust%20v1.1%22%2C%22pplx%2027B%20v1.1%22%2C%22Nimble%209B%20v3%22%2C%22OpenAI%20gpt-6-luna%22%2C%22Clef%2027B%22%2C%22d3-mini%204B%22%2C%22d3-flash%209B%22%2C%22pplx%20flash%204B%22%2C%22h2o%20lightning%204B%22%2C%22Clef-Flash%209B%22%2C%22d3-nano%202B%22%2C%22d3-lite%200.8B%22%2C%22Strands%202B%22%5D%2C%22datasets%22%3A%5B%7B%22data%22%3A%5B100.0%2C100.0%2C100.0%2C100.0%2C100.0%2C99.0%2C97.9%2C97.9%2C97.9%2C96.9%2C96.9%2C93.8%2C86.5%2C80.2%2C76.0%5D%2C%22backgroundColor%22%3A%5B%22%2314b8a6%22%2C%22%23b91c1c%22%2C%22%23e879f9%22%2C%22%23334155%22%2C%22%230e7490%22%2C%22%2310a37f%22%2C%22%23059669%22%2C%22%23f97316%22%2C%22%23ef4444%22%2C%22%2364748b%22%2C%22%23facc15%22%2C%22%2310b981%22%2C%22%23f59e0b%22%2C%22%23fdba74%22%2C%22%2384cc16%22%5D%2C%22borderWidth%22%3A0%2C%22barPercentage%22%3A0.82%7D%5D%7D%2C%22options%22%3A%7B%22indexAxis%22%3A%22y%22%2C%22layout%22%3A%7B%22padding%22%3A%7B%22right%22%3A40%7D%7D%2C%22plugins%22%3A%7B%22legend%22%3A%7B%22display%22%3Afalse%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22Medium%22%2C%22font%22%3A%7B%22size%22%3A16%7D%7D%2C%22datalabels%22%3A%7B%22display%22%3Atrue%2C%22anchor%22%3A%22end%22%2C%22align%22%3A%22end%22%2C%22clamp%22%3Atrue%2C%22color%22%3A%22%23111827%22%2C%22font%22%3A%7B%22size%22%3A11%7D%2C%22formatter%22%3Afunction%28v%29%7Breturn%20v.toFixed%281%29%7D%7D%7D%2C%22scales%22%3A%7B%22x%22%3A%7B%22min%22%3A0%2C%22max%22%3A100%2C%22ticks%22%3A%7B%22stepSize%22%3A20%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%25%22%7D%7D%2C%22y%22%3A%7B%22grid%22%3A%7B%22display%22%3Afalse%7D%7D%7D%7D%7D)

[![Hard](https://quickchart.io/chart?version=3&w=640&h=430&bkg=white&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%22d3%2027B%22%2C%22AutoTrust%20v1.1%22%2C%22pplx%2027B%20v1.1%22%2C%22d3-flash%209B%22%2C%22d3-mini%204B%22%2C%22d3-nano%202B%22%2C%22Jev%201.14.0%22%2C%22pplx%20flash%204B%22%2C%22Nimble%209B%20v3%22%2C%22OpenAI%20gpt-6-luna%22%2C%22h2o%20lightning%204B%22%2C%22Clef%2027B%22%2C%22Clef-Flash%209B%22%2C%22Strands%202B%22%2C%22d3-lite%200.8B%22%5D%2C%22datasets%22%3A%5B%7B%22data%22%3A%5B98.2%2C97.6%2C94.3%2C93.5%2C92.6%2C88.1%2C87.2%2C85.1%2C83.6%2C81.8%2C81.0%2C70.8%2C66.1%2C61.6%2C61.3%5D%2C%22backgroundColor%22%3A%5B%22%23b91c1c%22%2C%22%23e879f9%22%2C%22%23334155%22%2C%22%23ef4444%22%2C%22%23f97316%22%2C%22%23f59e0b%22%2C%22%2314b8a6%22%2C%22%2364748b%22%2C%22%230e7490%22%2C%22%2310a37f%22%2C%22%23facc15%22%2C%22%23059669%22%2C%22%2310b981%22%2C%22%2384cc16%22%2C%22%23fdba74%22%5D%2C%22borderWidth%22%3A0%2C%22barPercentage%22%3A0.82%7D%5D%7D%2C%22options%22%3A%7B%22indexAxis%22%3A%22y%22%2C%22layout%22%3A%7B%22padding%22%3A%7B%22right%22%3A40%7D%7D%2C%22plugins%22%3A%7B%22legend%22%3A%7B%22display%22%3Afalse%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22Hard%22%2C%22font%22%3A%7B%22size%22%3A16%7D%7D%2C%22datalabels%22%3A%7B%22display%22%3Atrue%2C%22anchor%22%3A%22end%22%2C%22align%22%3A%22end%22%2C%22clamp%22%3Atrue%2C%22color%22%3A%22%23111827%22%2C%22font%22%3A%7B%22size%22%3A11%7D%2C%22formatter%22%3Afunction%28v%29%7Breturn%20v.toFixed%281%29%7D%7D%7D%2C%22scales%22%3A%7B%22x%22%3A%7B%22min%22%3A0%2C%22max%22%3A100%2C%22ticks%22%3A%7B%22stepSize%22%3A20%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%25%22%7D%7D%2C%22y%22%3A%7B%22grid%22%3A%7B%22display%22%3Afalse%7D%7D%7D%7D%7D)](https://quickchart.io/chart?version=3&w=640&h=430&bkg=white&c=%7B%22type%22%3A%22bar%22%2C%22data%22%3A%7B%22labels%22%3A%5B%22d3%2027B%22%2C%22AutoTrust%20v1.1%22%2C%22pplx%2027B%20v1.1%22%2C%22d3-flash%209B%22%2C%22d3-mini%204B%22%2C%22d3-nano%202B%22%2C%22Jev%201.14.0%22%2C%22pplx%20flash%204B%22%2C%22Nimble%209B%20v3%22%2C%22OpenAI%20gpt-6-luna%22%2C%22h2o%20lightning%204B%22%2C%22Clef%2027B%22%2C%22Clef-Flash%209B%22%2C%22Strands%202B%22%2C%22d3-lite%200.8B%22%5D%2C%22datasets%22%3A%5B%7B%22data%22%3A%5B98.2%2C97.6%2C94.3%2C93.5%2C92.6%2C88.1%2C87.2%2C85.1%2C83.6%2C81.8%2C81.0%2C70.8%2C66.1%2C61.6%2C61.3%5D%2C%22backgroundColor%22%3A%5B%22%23b91c1c%22%2C%22%23e879f9%22%2C%22%23334155%22%2C%22%23ef4444%22%2C%22%23f97316%22%2C%22%23f59e0b%22%2C%22%2314b8a6%22%2C%22%2364748b%22%2C%22%230e7490%22%2C%22%2310a37f%22%2C%22%23facc15%22%2C%22%23059669%22%2C%22%2310b981%22%2C%22%2384cc16%22%2C%22%23fdba74%22%5D%2C%22borderWidth%22%3A0%2C%22barPercentage%22%3A0.82%7D%5D%7D%2C%22options%22%3A%7B%22indexAxis%22%3A%22y%22%2C%22layout%22%3A%7B%22padding%22%3A%7B%22right%22%3A40%7D%7D%2C%22plugins%22%3A%7B%22legend%22%3A%7B%22display%22%3Afalse%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22Hard%22%2C%22font%22%3A%7B%22size%22%3A16%7D%7D%2C%22datalabels%22%3A%7B%22display%22%3Atrue%2C%22anchor%22%3A%22end%22%2C%22align%22%3A%22end%22%2C%22clamp%22%3Atrue%2C%22color%22%3A%22%23111827%22%2C%22font%22%3A%7B%22size%22%3A11%7D%2C%22formatter%22%3Afunction%28v%29%7Breturn%20v.toFixed%281%29%7D%7D%7D%2C%22scales%22%3A%7B%22x%22%3A%7B%22min%22%3A0%2C%22max%22%3A100%2C%22ticks%22%3A%7B%22stepSize%22%3A20%7D%2C%22title%22%3A%7B%22display%22%3Atrue%2C%22text%22%3A%22%25%22%7D%7D%2C%22y%22%3A%7B%22grid%22%3A%7B%22display%22%3Afalse%7D%7D%7D%7D%7D)

| Model | Easy /72 | Medium /96 | Hard /336 | All /504 | Hard priority reversals /29 |
| --- | ---: | ---: | ---: | ---: | ---: |
| d3 27B | 72 | 96 | 330 | 498 | 29 |
| AutoTrust JEV-27B v1.1 | 72 | 96 | 328 | 496 | 29 |
| pplx-decider v1.1 27B | 72 | 96 | 317 | 485 | 28 |
| d3-flash 9B | 72 | 94 | 314 | 480 | 28 |
| d3-mini 4B | 72 | 94 | 311 | 477 | 28 |
| Jev `jev-1.14.0` | 72 | 96 | 293 | 461 | 24 |
| pplx-decider-flash v1.2 4B | 72 | 93 | 286 | 451 | 21 |
| d3-nano 2B | 70 | 83 | 296 | 449 | 25 |
| Bespoke Nimble 9B v3 | 72 | 96 | 281 | 449 | 29 |
| OpenAI `gpt-6-luna` (hosted) | 72 | 95 | 275 | 442 | 23 |
| h2o-lightning-4b | 72 | 93 | 272 | 437 | 14 |
| Clef 27B | 72 | 94 | 238 | 404 | 4 |
| Clef-Flash 9B | 72 | 90 | 222 | 384 | 3 |
| d3-lite 0.8B | 70 | 77 | 206 | 353 | 6 |
| Strands Decider 2B | 72 | 73 | 207 | 352 | 1 |

Easy and medium are regression and "insufficient information" controls.
Hard carries the difference: a priority reversal counts only if both ends
of a same-case pair are right in both orders. Always answering B already
scores 194/336 on hard. All 22,176 first attempts (44 models × 504) are
valid, with no errors. Adapters differ, so these are task-level scores, not
weight comparisons. Jev `jev-1.14.0` scores 20 fewer than `jev-1.13.0`, all
on hard (293 vs 313; a repeat gave 462 in all). From 0.0.7 jev-lite pins
`jev-1.14.0` anyway (owner decision): pin `jev-1.13.0` yourself while it is
served if hard text decisions matter most. Self-host pick:
[d3-nano](https://huggingface.co/vllm-sr/d3-nano) (2.2B, Apache-2.0, 449/504),
which `LocalTypeSafe` runs (above). This bank ships in the `jev-lite`
0.0.2–0.0.4 sdists, so models released after 2026-10-03 may have seen it.

Earlier banks (`semantic-decisions@v0.0.1`, the controlled slices, r2) are
retired. v0.0.1 survives only at package tag `v0.0.1`, and none of their
scores is pooled with v0.0.2. The project redistributes no measured model's
weights, except the Apache-2.0 d3-nano export that `LocalTypeSafe`
downloads, which carries its LICENSE and NOTICE.

## Latency by question count

One request, one state, N questions (1 → 255), on one machine (NVIDIA GB10,
local models on its GPU, `LocalTypeSafe` on its CPU; hosted rows over the
network). Speed depends on the hardware, so read the growth, not the
milliseconds. The state is 24 joined bank policies (about 2,000 tokens);
p50 of three uncached requests (each opens with a fresh request number, so
no server reuses an earlier prefix). A backend that scores one question per call
gets them one after another, as its client would. "cut" means a smaller N
already passed 45 s or ran out of memory (local models get at most 70% of the
GPU memory). Questions are numbered, so no two in a request are the same.
The bar chart has every core model: what each added question costs, as a share
of a one-question request, from 1 to the largest N it reached (green hosted,
blue state read once, grey state read per question; 100% means N questions
take N times as long). The line chart follows a few of them over N; the
table gives the milliseconds.

*State read* is inferred from the growth: a model that reads the state once
per question takes about N times as long for N questions. Hosted rows also
ran 384 and 512 questions (Jev 1.14: 458 and 528 ms); Jev answered 768
questions (about 37k tokens) with HTTP 400.

[![Cost of each added question](https://quickchart.io/chart?version=3&w=640&h=586&bkg=white&c={%22type%22:%22bar%22,%22data%22:{%22labels%22:[%22Jev%201.14.0%22,%22OpenAI%20gpt-6-luna%22,%22LocalTypeSafe%200.0.7%20CPU%22,%22d3%2027B%20state%20once%22,%22d3-flash%20state%20once%22,%22d3-mini%20state%20once%22,%22d3-nano%20state%20once%22,%22Clef%2027B%22,%22Clef-Flash%209B%22,%22d3-lite%20state%20once%22,%22Strands%202B%22,%22pplx%20flash%204B%22,%22Nimble%209B%20v3%22,%22h2o%20lightning%204B%22,%22AutoTrust%20v1.1%22,%22d3%2027B%22,%22d3-flash%209B%22,%22pplx%2027B%20v1.1%22,%22d3-mini%204B%22,%22d3-nano%202B%22,%22d3-lite%200.8B%22],%22datasets%22:[{%22data%22:[0.4,0.86,4.1,4.1,4.2,4.5,4.6,4.9,5.1,5.3,7.5,10.0,17.0,87.0,90.0,97.0,99.0,100.0,100.0,110.0,120.0],%22backgroundColor%22:[%22%230d9488%22,%22%230d9488%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22]}]},%22options%22:{%22indexAxis%22:%22y%22,%22layout%22:{%22padding%22:{%22right%22:40}},%22plugins%22:{%22legend%22:{%22display%22:false},%22title%22:{%22display%22:true,%22text%22:%22Cost%20of%20each%20added%20question%22,%22font%22:{%22size%22:16}},%22datalabels%22:{%22display%22:true,%22anchor%22:%22end%22,%22align%22:%22end%22,%22color%22:%22%23111827%22,%22font%22:{%22size%22:11},%22formatter%22:function%28v%29{return%20v%2B%27%25%27}}},%22scales%22:{%22x%22:{%22type%22:%22logarithmic%22,%22min%22:0.1,%22max%22:300,%22title%22:{%22display%22:true,%22text%22:%22%25%20of%20one%20question%27s%20latency,%20per%20added%20question%22}},%22y%22:{%22grid%22:{%22display%22:false}}}}})](https://quickchart.io/chart?version=3&w=640&h=586&bkg=white&c={%22type%22:%22bar%22,%22data%22:{%22labels%22:[%22Jev%201.14.0%22,%22OpenAI%20gpt-6-luna%22,%22LocalTypeSafe%200.0.7%20CPU%22,%22d3%2027B%20state%20once%22,%22d3-flash%20state%20once%22,%22d3-mini%20state%20once%22,%22d3-nano%20state%20once%22,%22Clef%2027B%22,%22Clef-Flash%209B%22,%22d3-lite%20state%20once%22,%22Strands%202B%22,%22pplx%20flash%204B%22,%22Nimble%209B%20v3%22,%22h2o%20lightning%204B%22,%22AutoTrust%20v1.1%22,%22d3%2027B%22,%22d3-flash%209B%22,%22pplx%2027B%20v1.1%22,%22d3-mini%204B%22,%22d3-nano%202B%22,%22d3-lite%200.8B%22],%22datasets%22:[{%22data%22:[0.4,0.86,4.1,4.1,4.2,4.5,4.6,4.9,5.1,5.3,7.5,10.0,17.0,87.0,90.0,97.0,99.0,100.0,100.0,110.0,120.0],%22backgroundColor%22:[%22%230d9488%22,%22%230d9488%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%232563eb%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22,%22%239ca3af%22]}]},%22options%22:{%22indexAxis%22:%22y%22,%22layout%22:{%22padding%22:{%22right%22:40}},%22plugins%22:{%22legend%22:{%22display%22:false},%22title%22:{%22display%22:true,%22text%22:%22Cost%20of%20each%20added%20question%22,%22font%22:{%22size%22:16}},%22datalabels%22:{%22display%22:true,%22anchor%22:%22end%22,%22align%22:%22end%22,%22color%22:%22%23111827%22,%22font%22:{%22size%22:11},%22formatter%22:function%28v%29{return%20v%2B%27%25%27}}},%22scales%22:{%22x%22:{%22type%22:%22logarithmic%22,%22min%22:0.1,%22max%22:300,%22title%22:{%22display%22:true,%22text%22:%22%25%20of%20one%20question%27s%20latency,%20per%20added%20question%22}},%22y%22:{%22grid%22:{%22display%22:false}}}}})

[![Latency growth from one question](https://quickchart.io/chart?version=3&w=900&h=560&bkg=white&c={%22type%22:%22line%22,%22data%22:{%22datasets%22:[{%22label%22:%22Jev%201.14.0%22,%22data%22:[[1,1.0],[2,1.0],[4,1.0],[8,1.0],[16,1.0],[32,1.1],[64,1.2],[128,1.6],[255,1.9],[384,2.6],[512,3.0]],%22borderColor%22:%22%230d9488%22,%22backgroundColor%22:%22%230d9488%22},{%22label%22:%22OpenAI%20gpt-6-luna%22,%22data%22:[[1,1.0],[2,1.1],[4,1.1],[8,1.4],[16,1.4],[32,1.7],[64,2.3],[128,2.9],[255,3.5],[384,4.3]],%22borderColor%22:%22%23f59e0b%22,%22backgroundColor%22:%22%23f59e0b%22},{%22label%22:%22LocalTypeSafe%200.0.7%20CPU%22,%22data%22:[[1,1.0],[2,1.0],[4,1.1],[8,1.3],[16,1.6],[32,2.3],[64,3.5],[128,6.1],[255,11]],%22borderColor%22:%22%2394a3b8%22,%22backgroundColor%22:%22%2394a3b8%22},{%22label%22:%22d3%2027B%22,%22data%22:[[1,1.0],[2,1.9],[4,3.9],[8,11],[16,16],[32,31]],%22borderColor%22:%22%231e293b%22,%22backgroundColor%22:%22%231e293b%22},{%22label%22:%22d3%2027B%20state%20once%22,%22data%22:[[1,1.0],[2,1.0],[4,1.1],[8,1.2],[16,1.5],[32,2.2],[64,3.4],[128,6.4],[255,11]],%22borderColor%22:%22%23c084fc%22,%22backgroundColor%22:%22%23c084fc%22},{%22label%22:%22pplx%20flash%204B%22,%22data%22:[[1,1.0],[2,1.0],[4,1.1],[8,1.2],[16,1.5],[32,2.0],[64,3.2],[128,6.0],[255,26]],%22borderColor%22:%22%237c3aed%22,%22backgroundColor%22:%22%237c3aed%22},{%22label%22:%22Strands%202B%22,%22data%22:[[1,1.0],[2,1.1],[4,1.1],[8,1.3],[16,1.7],[32,2.5],[64,5.0],[128,10],[255,20]],%22borderColor%22:%22%2316a34a%22,%22backgroundColor%22:%22%2316a34a%22},{%22label%22:%22Clef-Flash%209B%22,%22data%22:[[1,1.0],[2,1.0],[4,1.2],[8,1.3],[16,1.7],[32,2.6],[64,4.1],[128,7.5]],%22borderColor%22:%22%23dc2626%22,%22backgroundColor%22:%22%23dc2626%22},{%22label%22:%22Nimble%209B%20v3%22,%22data%22:[[1,1.0],[2,1.1],[4,1.2],[8,1.5],[16,1.9],[32,3.2],[64,6.0],[128,14],[255,44]],%22borderColor%22:%22%232563eb%22,%22backgroundColor%22:%22%232563eb%22},{%22label%22:%22h2o%20lightning%204B%22,%22data%22:[[1,1.0],[2,1.9],[4,3.8],[8,7.6],[16,14],[32,29],[64,57],[128,112],[255,223]],%22borderColor%22:%22%23ec4899%22,%22backgroundColor%22:%22%23ec4899%22}]},%22options%22:{%22elements%22:{%22point%22:{%22radius%22:2},%22line%22:{%22borderWidth%22:1.5,%22fill%22:false}},%22plugins%22:{%22title%22:{%22display%22:true,%22text%22:%22Latency%20growth%20from%20one%20question%22,%22font%22:{%22size%22:16}},%22legend%22:{%22position%22:%22right%22,%22labels%22:{%22boxWidth%22:10,%22font%22:{%22size%22:10}}}},%22scales%22:{%22x%22:{%22type%22:%22logarithmic%22,%22title%22:{%22display%22:true,%22text%22:%22questions%20in%20one%20request%22}},%22y%22:{%22type%22:%22logarithmic%22,%22title%22:{%22display%22:true,%22text%22:%22times%20the%201-question%20latency%22}}}}})](https://quickchart.io/chart?version=3&w=900&h=560&bkg=white&c={%22type%22:%22line%22,%22data%22:{%22datasets%22:[{%22label%22:%22Jev%201.14.0%22,%22data%22:[[1,1.0],[2,1.0],[4,1.0],[8,1.0],[16,1.0],[32,1.1],[64,1.2],[128,1.6],[255,1.9],[384,2.6],[512,3.0]],%22borderColor%22:%22%230d9488%22,%22backgroundColor%22:%22%230d9488%22},{%22label%22:%22OpenAI%20gpt-6-luna%22,%22data%22:[[1,1.0],[2,1.1],[4,1.1],[8,1.4],[16,1.4],[32,1.7],[64,2.3],[128,2.9],[255,3.5],[384,4.3]],%22borderColor%22:%22%23f59e0b%22,%22backgroundColor%22:%22%23f59e0b%22},{%22label%22:%22LocalTypeSafe%200.0.7%20CPU%22,%22data%22:[[1,1.0],[2,1.0],[4,1.1],[8,1.3],[16,1.6],[32,2.3],[64,3.5],[128,6.1],[255,11]],%22borderColor%22:%22%2394a3b8%22,%22backgroundColor%22:%22%2394a3b8%22},{%22label%22:%22d3%2027B%22,%22data%22:[[1,1.0],[2,1.9],[4,3.9],[8,11],[16,16],[32,31]],%22borderColor%22:%22%231e293b%22,%22backgroundColor%22:%22%231e293b%22},{%22label%22:%22d3%2027B%20state%20once%22,%22data%22:[[1,1.0],[2,1.0],[4,1.1],[8,1.2],[16,1.5],[32,2.2],[64,3.4],[128,6.4],[255,11]],%22borderColor%22:%22%23c084fc%22,%22backgroundColor%22:%22%23c084fc%22},{%22label%22:%22pplx%20flash%204B%22,%22data%22:[[1,1.0],[2,1.0],[4,1.1],[8,1.2],[16,1.5],[32,2.0],[64,3.2],[128,6.0],[255,26]],%22borderColor%22:%22%237c3aed%22,%22backgroundColor%22:%22%237c3aed%22},{%22label%22:%22Strands%202B%22,%22data%22:[[1,1.0],[2,1.1],[4,1.1],[8,1.3],[16,1.7],[32,2.5],[64,5.0],[128,10],[255,20]],%22borderColor%22:%22%2316a34a%22,%22backgroundColor%22:%22%2316a34a%22},{%22label%22:%22Clef-Flash%209B%22,%22data%22:[[1,1.0],[2,1.0],[4,1.2],[8,1.3],[16,1.7],[32,2.6],[64,4.1],[128,7.5]],%22borderColor%22:%22%23dc2626%22,%22backgroundColor%22:%22%23dc2626%22},{%22label%22:%22Nimble%209B%20v3%22,%22data%22:[[1,1.0],[2,1.1],[4,1.2],[8,1.5],[16,1.9],[32,3.2],[64,6.0],[128,14],[255,44]],%22borderColor%22:%22%232563eb%22,%22backgroundColor%22:%22%232563eb%22},{%22label%22:%22h2o%20lightning%204B%22,%22data%22:[[1,1.0],[2,1.9],[4,3.8],[8,7.6],[16,14],[32,29],[64,57],[128,112],[255,223]],%22borderColor%22:%22%23ec4899%22,%22backgroundColor%22:%22%23ec4899%22}]},%22options%22:{%22elements%22:{%22point%22:{%22radius%22:2},%22line%22:{%22borderWidth%22:1.5,%22fill%22:false}},%22plugins%22:{%22title%22:{%22display%22:true,%22text%22:%22Latency%20growth%20from%20one%20question%22,%22font%22:{%22size%22:16}},%22legend%22:{%22position%22:%22right%22,%22labels%22:{%22boxWidth%22:10,%22font%22:{%22size%22:10}}}},%22scales%22:{%22x%22:{%22type%22:%22logarithmic%22,%22title%22:{%22display%22:true,%22text%22:%22questions%20in%20one%20request%22}},%22y%22:{%22type%22:%22logarithmic%22,%22title%22:{%22display%22:true,%22text%22:%22times%20the%201-question%20latency%22}}}}})

| Model | State read | 1 | 16 | 64 | 255 | 255 vs 1 | Largest N | Per added question |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Jev `jev-1.14.0` | hosted | 174 | 179 | 214 | 335 | 1.9x | 512 | 0.4% |
| OpenAI `gpt-6-luna` (hosted) | hosted | 222 | 306 | 511 | 770 | 3.5x | 384 | 0.86% |
| `LocalTypeSafe` 0.0.7 (CPU) | once | 4,627 | 7,484 | 16,184 | 52,292 | 11.3x | 255 | 4.1% |
| d3 27B, state once | once | 2,360 | 3,489 | 8,097 | 26,822 | 11.4x | 255 | 4.1% |
| d3-flash 9B, state once | once | 744 | 1,132 | 2,712 | 8,616 | 11.6x | 255 | 4.2% |
| d3-mini 4B, state once | once | 506 | 788 | 1,918 | 6,221 | 12.3x | 255 | 4.5% |
| d3-nano 2B, state once | once | 216 | 327 | 836 | 2,720 | 12.6x | 255 | 4.6% |
| Clef 27B | once | 2,154 | 3,606 | 8,824 | cut | — | 128 | 4.9% |
| Clef-Flash 9B | once | 710 | 1,224 | 2,898 | cut | — | 128 | 5.1% |
| d3-lite 0.8B, state once | once | 140 | 227 | 602 | 2,009 | 14.4x | 255 | 5.3% |
| Strands Decider 2B | once | 357 | 596 | 1,792 | 7,184 | 20.1x | 255 | 7.5% |
| pplx-decider-flash v1.2 4B | once | 464 | 699 | 1,481 | 12,304 | 26.5x | 255 | 10% |
| Bespoke Nimble 9B v3 | once | 1,078 | 2,102 | 6,468 | 47,206 | 43.8x | 255 | 17% |
| h2o-lightning-4b | per question | 274 | 3,921 | 15,466 | 60,958 | 222.7x | 255 | 87% |
| AutoTrust JEV-27B v1.1 | per question | 1,519 | 22,456 | 88,019 | cut | — | 64 | 90% |
| d3 27B | per question | 2,140 | 33,333 | cut | cut | — | 32 | 97% |
| d3-flash 9B | per question | 664 | 10,503 | 41,828 | cut | — | 128 | 99% |
| pplx-decider v1.1 27B | per question | 2,139 | 34,239 | cut | cut | — | 32 | 100% |
| d3-mini 4B | per question | 458 | 7,398 | 29,616 | cut | — | 128 | 101% |
| d3-nano 2B | per question | 194 | 3,334 | 13,379 | 53,317 | 274.1x | 255 | 108% |
| d3-lite 0.8B | per question | 124 | 2,287 | 9,133 | 36,344 | 293.6x | 255 | 115% |

"State once" rows serve the d3 checkpoints with the state read once and the
questions batched on it; in d3's noncausal layers the state then no longer
sees the questions. The score on `semantic-decisions@v0.0.2` (/504) changes
only for some sizes: nano 449 → 449, lite 353 → 342, mini 477 → 459, 27B
498 → 496 (flash: 67 → 67 on a 72-slot sample). jev-lite 0.0.7 ships only
the nano one, as `LocalTypeSafe`.

## License

Apache-2.0
