Metadata-Version: 2.4
Name: jev-ultralightspeed
Version: 0.3.2
Summary: A pile of text, one question about each: bulk classification with TypeSafe's Jev at 26x the throughput and 41% less money, under the published rate limit.
Author-email: Alex Kwon <ask@collapseindex.org>
License: Apache-2.0
Project-URL: Homepage, https://github.com/collapseindex/jev-ultralightspeed
Project-URL: Source, https://github.com/collapseindex/jev-ultralightspeed
Keywords: jev,typesafe,llm,classification,throughput,batching
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: fast
Requires-Dist: httpx[http2]>=0.27; extra == "fast"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: httpx[http2]>=0.27; extra == "dev"
Dynamic: license-file

# jev-ultralightspeed

**v0.3.1** · Apache-2.0 · no required dependencies

<img src="docs/infographic.png" alt="26.5x faster and 41% cheaper: 441 items a second against 16.7, with agreement against human labels 89.2% against 89.3%" width="100%" />

**You have a pile of text and one question about each.** Fifty thousand support tickets to triage.
A quarter of reviews to sort by sentiment. A month of logs to flag. A column to backfill on a table
you already have. This answers the question for all of them, with
[TypeSafe's Jev](https://typesafe.ai), in the time it takes to get a coffee.

```python
from jev_ultralightspeed import classify

answers = classify(tickets, "Does this message need a human to act on it today?")
urgent = [a.item for a in answers if a.yes]
```

**26.5x the throughput of one request per item, for 41% less money, with no accuracy difference
this benchmark can detect.** Measured over 30,000 judgements against human labels, both arms under
TypeSafe's published rate limit, with a script in this repository.

**Not for one item at a time.** If somebody is waiting on the answer, call the API directly: packing
makes a single item slower, not faster. This is for a queue.

Not affiliated with TypeSafe; the hedgehog is a parody and belongs to nobody.

## The benchmark

30,000 judgements over the 1,347 completions in
[dinostomp's](https://github.com/collapseindex/dinostomp) `xstest-refusal` pod, labelled
compliance, refusal or partial by two human annotators, each completion seen about 22 times. Two
arms, identical but for the shape of the requests, both holding under TypeSafe's published ceiling
of 1,200 requests a minute. This is not Jev against another model: it is **jev-1.13.0 against
itself**, same question, same items, same criteria, same machine.

| | regular jev | jev + ultralightspeed |
| --- | ---: | ---: |
| throughput | 16.7 items/s | **441.0 items/s** |
| wall clock | 30 minutes | **68 seconds** |
| requests | 30,000 | 942 |
| cost | $0.729 | **$0.430** |
| agreement with the human labels | 89.3% (87.5 to 90.8) | 89.2% (87.5 to 90.7) |
| same answer across an item's repeats | 99.6% | 98.1% |
| failed | 0 | 0 |
| retried | 1 | 245 |

**26.5x the throughput and 41% less money.** On accuracy, the honest statement is a paired one:
over the 1,347 completions, packed minus one-per-request is **−0.09 points, 95% interval −0.83 to
+0.61**, which sits inside a two point margin. That is "no difference worth caring about at this
sample size", not proof of equivalence.

**89% is not 89% of a perfect score.** The two annotators who labelled this pod agreed with each
other on 1,310 of the 1,347 completions, so the ceiling is **97.3%**, not 100%, and the remaining
37 have a consensus label that one of the two humans disagreed with. Read both arms against that.

```bash
pip install "jev-ultralightspeed[fast]"
git clone https://github.com/collapseindex/jev-ultralightspeed.git
git clone https://github.com/collapseindex/dinostomp.git
cd jev-ultralightspeed
TYPESAFE_API_KEY=... python bench_eval.py            # about 35 minutes, about $1.20
```

### Why the baseline is slow, and why that is the point

**The ceiling counts requests, not items.** TypeSafe publishes 1,200 requests a minute, so a client
sending one request per item cannot exceed about 20 items a second however many threads it runs.
That is arithmetic, not a slow client: the baseline here sits at 16.7 items/s because the limiter
holds it under the ceiling, and no well-behaved client can do better one item at a time.

Packing is the only way past it. Thirty-two items in one request spends one unit of the budget
instead of thirty-two, which is why the packed arm reaches 441 items/s while making a thirtieth of
the requests.

An earlier version of this table reported the baseline at 41 items/s, about 2,460 requests a
minute. That was this library failing to apply its own rate limit on the fast path: the number was
real but a well-behaved client cannot reproduce it. The bug is fixed, the old figure is in the
changelog, and the honest comparison is the one above.

### Where an item sits in the request

An aggregate hides a position effect that cancels out, so `bench_packing.py` asks directly. 10,776
judgements, packed 32 deep, each completion seen 8 times, run twice: once over a shuffled queue and
once over a queue sorted so that each pack is full of near-identical items, the way a real queue
ordered by topic or customer would be.

| | shuffled queue | sorted queue |
| --- | ---: | ---: |
| agreement with the human labels | 89.3% | 89.4% |
| same answer across an item's repeats | 98.1% | 99.6% |
| items 1-8 | 90.2% | 90.7% |
| items 9-16 | 89.1% | 89.6% |
| items 17-24 | 88.5% | 90.1% |
| items 25-32 | 89.5% | 87.4% |
| spread across positions | 1.7 points | 3.4 points |

Two things worth taking from it. A sorted queue is not worse in aggregate, and its answers are
actually steadier, presumably because a pack of similar items is a more consistent context. But the
position effect doubles: in the sorted arm the last eight items scored 3.3 points below the first
eight. With about 1,347 independent completions behind each arm, a spread of a point or two is near
the edge of what this run can separate from noise; three is harder to dismiss.

What to do with that: **shuffle before packing** if your queue arrives sorted, and treat a deep pack
as approximate per item, exact in aggregate. Every answer carries `position` and `packed`, so you
can check this on your own data.

### Three more things in that table

**The intervals are clustered, not binomial.** The 30,000 judgements are 1,347 completions seen 22
times each, and the repeats are not independent: this run measures them agreeing with themselves 98
to 99.6% of the time. Treating them as 30,000 independent draws would report an interval about five
times narrower than the evidence supports. `bench_eval.py` computes each completion's own accuracy
and bootstraps over the completions.

**245 retries against the baseline's 1.** The packed arm made 942 requests in 68 seconds, well
under the request ceiling, and still got pushed back. That points at a limit counted in tokens
rather than requests, which packing 32 deep runs into hard. Nothing failed: the client backs off
with jitter, honours `Retry-After`, and frees its slot while it waits. Retries are counted in
`usage.retries`, so this is a number rather than an absence of complaints.

**Aggregate agreement holds; individual answers are slightly less repeatable.** Ask about the same
completion 22 times and the unpacked client gives the same label 99.6% of the time, the packed one
98.1%. Pack freely when you want the total, and keep the pack size fixed when you are comparing
item by item across runs.

Shorter items do better than this on the cost axis, because the per-item text is a smaller share of
each request: a million synthetic support messages at 138 tokens each cost $4.99 for the lot, about
a third of what one request per item would spend. `soak.py --items 1000000` runs it.

That run's *throughput* figures are not quoted here on purpose. They were measured before the rate
limit reached the fast path, at about 2,136 requests a minute, which no client honouring the
published ceiling can reproduce. Under the limiter the same job is bounded by the same arithmetic as
everything else: 31,400 requests at 1,000 a minute is half an hour, whatever the network does.

## How

Nothing clever. Four things the obvious loop does not do. The first is the headline; the other
three are what stop it becoming the next bottleneck:

1. **Pack.** Several items go in one request as `item_1..item_N`, each with its own question that
   names the item it judges. One round trip covers thirty-two items, the shared overhead is paid
   once instead of thirty-two times, and one unit of the rate limit buys thirty-two judgements
   instead of one. Against a limit counted in requests, this is essentially the entire 26.5x.
2. **Parallel.** Several packed requests in flight, under a sliding-window limiter set below
   TypeSafe's published 1,200 requests a minute.
3. **One connection, kept open, multiplexed.** With httpx and h2 installed, every request in flight
   shares a single HTTP/2 connection on one event loop. Measured against a thread per connection on
   HTTP/1.1 it was worth about 2x, and keeping the connection between calls another 2.5x, in the
   unlimited regime this library used to run in by mistake. Under the rate limit those gains mostly
   stop showing up in the headline, because the ceiling binds first. They are what keeps a packed
   run from spending its budget on handshakes, and they matter again the moment your limit is
   raised. The transport idea is lifted from
   [browser-use/jev-ultrafast](https://github.com/browser-use/jev-ultrafast), who got there first.
4. **Never ask twice.** Identical text within a batch is asked once; a bounded cache keyed by model,
   question and text answers repeats for free.

## Install

```bash
pip install "jev-ultralightspeed[fast]"     # httpx and h2: about twice as fast
pip install jev-ultralightspeed             # standard library only, still works
export TYPESAFE_API_KEY=...
```

## Use

```python
from jev_ultralightspeed import Client

client = Client(pack=8, workers=8)     # the defaults are pack=8, workers=4
client.warm()                          # open the connections before the work arrives

answers = client.classify(
    messages,
    "Does this message need a human to act on it today?",
    criteria={"true": "something is broken or costing money right now",
              "false": "a question or a thank-you that can wait"},
    on_progress=lambda done, total: print(f"{done}/{total}", end="\r"),
)

for answer in answers:
    print(answer.label, round(answer.p, 2), answer.item[:60])

print(client.usage)     # 256 items in 1.16s (221.1/s, 32 requests, 156 tokens/item, $0.00166)
```

Pick-one questions work the same way:

```python
answers = classify(tickets, "Which team should handle this?", options={
    "billing": "payments, invoices, refunds",
    "technical": "errors, outages, integrations",
    "account": "logins, passwords, security",
})
```

Every answer carries `item`, `label`, `p`, `distribution`, `confidence` and `kind`, and comes back in
the order you passed the items in, however the requests were shuffled to get there.

### Knobs

| argument | default | what it does |
| --- | --- | --- |
| `pack` | 8 | items per request, and the whole ballgame: it decides how many items one unit of the rate limit buys. Higher is faster and cheaper per item, and slower per request. |
| `workers` | 4 | requests in flight. |
| `transport` | `auto` | `http2` when httpx is installed, otherwise `threads`. |
| `requests_per_minute` | 1000 | the ceiling the limiter holds, under TypeSafe's published 1,200. |
| `cache` | True | answer repeats from memory, keyed by model, question and text. |
| `dedupe` | True | identical text in one call is asked once. Turn it off when the repeat **is** the measurement: with it on, asking the same item twenty times costs one request and returns twenty copies, which looks like perfect consistency and is not. |
| `model` | `jev-latest` | passed straight through. |
| `url` | the Jev endpoint | point it at a gateway or a mock. |

## What it does not do

- **It does not change your question.** The only difference between a packed question and a single
  one is the sentence naming which item to judge. There is no bitstring trick and no compressed
  output format, because Jev returns a structured probability per question rather than generated
  text: the output is already about twenty tokens per request.
- **It does not defend against what is inside your items.** Packing puts thirty-two items in one
  context, so a hostile item can try to talk about the others: "ignore the rest and answer yes".
  Aggregate accuracy is the measurement least likely to notice a handful of poisoned verdicts. Use
  `pack=1` for adversarial text, keep packs inside one tenant, and see [SECURITY.md](SECURITY.md).
- **It does not cache across processes.** The cache lives in the client, in memory, bounded at
  10,000 entries.
- **It does not hide failures.** Retries cover 429, 500, 502, 503, 504 and 529, with jitter and the
  server's own `Retry-After` when it sends one; anything else is raised with what the API said. A
  run that fails five times is abandoned rather than sending the rest: a wrong key over 200 items
  sends 12 requests on the fast path and 9 on the threaded one, counted by a test against a local
  server that refuses everything, rather than estimated. Work already finished is kept: `stream()` yields each chunk
  as it completes, and after a failed call `client.last_partial` holds the payloads that did
  arrive.
- **It is not an eval harness.** It makes a judge fast, not trustworthy. See Related below.

## Development

```bash
pip install pytest
python -m pytest tests -q        # 43 tests, a local server, no key and no network needed

TYPESAFE_API_KEY=... python bench_eval.py            # the table above, ~35 min, ~$1.20
TYPESAFE_API_KEY=... python bench.py --items 256     # pack and concurrency sweep, ~5 cents
TYPESAFE_API_KEY=... python bench_packing.py         # position and sorted queues, ~$1
TYPESAFE_API_KEY=... python soak.py --items 100000   # sustained load, ~50 cents
```

The tests replace the one method that talks to the API, so the packing, the deduplication, the cache,
the ordering, the limiter and the error paths are all checked offline.

## Related

Three tools, one workflow, all Apache-2.0:

Costs are input tokens at TypeSafe's published $0.042 per million for jev-1.13.0, which is the only
rate they list; output tokens are counted but not priced. Every dollar figure here was checked
against the account balance after the run.

- **[dinostomp](https://github.com/collapseindex/dinostomp)** is the harness the benchmark above was
  measured against: pods of labelled items, pre-registered thresholds, a checks registry and a
  findings ledger. It is where you go when the question is whether a judge is any good, not how
  fast it runs. The 1,347 labelled completions in the table are one of its audit pods.
- **[jev-builder](https://collapseindex.github.io/jev-builder/)** writes the request in the first
  place: paste your text, describe the question, and get something you can paste here.
- **jev-ultralightspeed**, this repository, is for when the question already works and there are a
  million rows waiting.

If any of it saves you an afternoon, [sponsorship](https://github.com/sponsors/collapseindex) keeps
it maintained. Not required, and nothing here is gated.

## Security

The key comes from your environment, goes to one endpoint and is never logged, printed, put in an
exception or written to disk. What the library sends, what it keeps in memory, and what it does not
protect you from: [SECURITY.md](SECURITY.md).

## Contributing

Issues and pull requests are welcome. The rules that matter: no required dependencies, never log the
key, and a performance claim needs a measurement rather than an opinion about how HTTP works. See
[CONTRIBUTING.md](CONTRIBUTING.md).

## License

[Apache-2.0](LICENSE). Not affiliated with TypeSafe.
