Metadata-Version: 2.4
Name: anyjev
Version: 0.3.0
Summary: AnyJev: Jev-style typed decisions with probabilities from open LLMs: a training-free readout of any model, and the self-distilled Tacit models. Not affiliated with TypeSafe AI.
Author: Jiamu Zhang, Tianze Yang, Yucheng Shi, Liang Wu
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/nokia-applied-research/AnyJev
Project-URL: Repository, https://github.com/nokia-applied-research/AnyJev
Project-URL: Changelog, https://github.com/nokia-applied-research/AnyJev/blob/main/CHANGELOG.md
Keywords: llm,decision,calibration,jev,system-one,self-distillation,vllm
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Provides-Extra: hf
Requires-Dist: torch>=2.3; extra == "hf"
Requires-Dist: transformers>=4.53; extra == "hf"
Provides-Extra: vllm
Requires-Dist: vllm>=0.17.1; extra == "vllm"
Provides-Extra: client
Requires-Dist: transformers>=4.53; extra == "client"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

<div align="center">

<img src="https://raw.githubusercontent.com/nokia-applied-research/AnyJev/main/assets/banner.png" width="100%" alt="AnyJev: turn any LLM into a Jev-style decision model. Typed decisions, real probabilities, training-free or self-distilled. Qwen3-8B on BANKING77-20, raw readout to L0 with no labels: order-flip rate 0.230 to 0.073, accuracy 0.747 to 0.803, auto-decidable at 5% risk 7.7% to 46.3%.">

[![PyPI](https://img.shields.io/pypi/v/anyjev?color=3b82f6)](https://pypi.org/project/anyjev/)
[![Python](https://img.shields.io/pypi/pyversions/anyjev)](https://pypi.org/project/anyjev/)
[![CI](https://github.com/nokia-applied-research/AnyJev/actions/workflows/ci.yml/badge.svg)](https://github.com/nokia-applied-research/AnyJev/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](https://github.com/nokia-applied-research/AnyJev/blob/main/LICENSE)
[![Technical Report](https://img.shields.io/badge/Technical%20Report-PDF-b31b1b?logo=adobeacrobatreader&logoColor=white)](https://arxiv.org/pdf/2610.00831)
[![Models](https://img.shields.io/badge/%F0%9F%A4%97%20Tacit-models-yellow)](https://huggingface.co/collections/morriszjm/tacit-6ac41d0b50af9e5417c5c234)

**English** · [简体中文](https://github.com/nokia-applied-research/AnyJev/blob/main/README.zh-CN.md) · [⚡ Serve it](#-serve-it) · [📊 Results](#-tacit-results) · [🧭 Roadmap](#-roadmap) · [📖 Levels](https://github.com/nokia-applied-research/AnyJev/blob/main/docs/levels.md)

</div>

<p align="center">
  <b>Jiamu Zhang</b><sup>1</sup> &nbsp;&nbsp;&nbsp; <b>Tianze Yang</b><sup>1</sup> &nbsp;&nbsp;&nbsp; <b>Yucheng Shi</b><sup>2</sup> &nbsp;&nbsp;&nbsp; <b>Evan Chen</b><sup>1</sup>
  <br>
  <b>Zixiang Nie</b><sup>1</sup> &nbsp;&nbsp;&nbsp; <b>Kelly Wan</b><sup>1</sup> &nbsp;&nbsp;&nbsp; <b>Liangjie Hong</b><sup>1</sup> &nbsp;&nbsp;&nbsp; <b>Ninghao Liu</b><sup>3</sup> &nbsp;&nbsp;&nbsp; <b>Liang Wu</b><sup>1</sup>
</p>
<p align="center">
  <sub><sup>1</sup>&nbsp;Nokia, Sunnyvale, CA, USA &nbsp;&nbsp;&nbsp;&nbsp; <sup>2</sup>&nbsp;Tencent Hunyuan &nbsp;&nbsp;&nbsp;&nbsp; <sup>3</sup>&nbsp;The Hong Kong Polytechnic University</sub>
</p>

<p align="center">
  <img src="https://raw.githubusercontent.com/nokia-applied-research/AnyJev/main/assets/flip.gif" width="100%" alt="Reverse the option order: the raw logit readout flips its answer, AnyJev L0 gives the same answer both ways">
  <br>
  <sub>Qwen3-8B on a real BANKING77 item. Every number is a model output.</sub>
</p>

> [!TIP]
> **🆕 Tacit: AnyJev's self-distilled models, 1.7B to 9B, are on
> [Hugging Face](https://huggingface.co/collections/morriszjm/tacit-6ac41d0b50af9e5417c5c234).** One
> forward pass per decision. With `adaptive=True`, a capped share of low-confidence decisions goes to
> the model's own reasoning. They run on transformers, in-process vLLM or a stock `vllm serve`.
> `pip install "anyjev[hf]"`, then `Tacit.from_pretrained("morriszjm/Tacit-9B")`.
> [Start here ↓](#-serve-it)

## ⚡ Serve it

**A Tacit model, in Python.**

```python
from anyjev import Tacit

tacit = Tacit.from_pretrained("morriszjm/Tacit-9B")          # transformers, one GPU
d = tacit.decide(state="Customer: my package was due last Monday and it still has not arrived.",
                 question="What does the customer want?",
                 options=["track_order", "cancel_order", "refund", "change_address"])
d["answer"], d["probs"], d["route"]    # an option, {option: probability}, "one_forward" or "cot"
```

`kind` is `"choice"` (default), `"yes_no"` or `"score"` (`options` are ordered levels, lowest
first). `decide_batch([...])` takes a list of such dicts.

**Send the hardest decisions to reasoning, with a cap.**

```python
tacit = Tacit.from_pretrained("morriszjm/Tacit-9B", adaptive=True, tau=0.5, max_cot_share=0.2, cot_window=1000)
```

A decision whose top two options are less than `tau` apart in log-probability is answered again with
thinking on. After the reasoning, the answer is read as a label distribution, so it also has
probabilities and cannot fail to parse. At most `max_cot_share` of the last `cot_window` decisions
escalate, however hard the traffic gets. Every escalation is reported (`route == "cot"`, with the
first pass kept), and `tacit.stats` counts them.

**On vLLM.** A stock server; the client needs only the tokenizer.

```bash
pip install "anyjev[client]"
vllm serve morriszjm/Tacit-9B --host 127.0.0.1 --port 8000
```

```python
tacit = Tacit.from_pretrained("morriszjm/Tacit-9B", engine="server",
                              base_url="http://127.0.0.1:8000", adaptive=True)
```

`engine="vllm"` (`pip install "anyjev[vllm]"`) runs vLLM in the same process instead.

**An HTTP endpoint for many clients sharing one cap.**

```bash
python -m anyjev.serve --model morriszjm/Tacit-9B --upstream http://127.0.0.1:8000 --adaptive --port 8100
curl -s 127.0.0.1:8100/v1/decide -H 'Content-Type: application/json' \
  -d '{"state": "Customer: my package has not arrived.", "question": "What does the customer want?", "options": ["track_order", "refund"]}'
```

`POST /v1/decide` takes one decision or `{"items": [...]}`; `GET /v1/stats` reports the escalation
share. The gateway listens on 127.0.0.1 and has no authentication of its own.

| argument | default | meaning |
|---|---|---|
| `adaptive` | `False` | send low-confidence decisions to the model's own reasoning |
| `tau` | `0.5` | low confidence: the log-probability gap between the top two options is below `tau` |
| `max_cot_share` | `0.2` | at most this share of the last `cot_window` decisions escalates; `None` removes the cap |
| `cot_window` | `1000` | how many recent decisions the cap counts; `None` counts every decision since loading |
| `cot_max_tokens` | `8192` | reasoning budget of one decision |
| `engine` | `"transformers"` | `"vllm"` runs vLLM in this process; `"server"` uses a running `vllm serve` at `base_url` |

**Any other model, training-free.** The `Decider` reads the same typed decisions from any open
causal LM, with no training and no labels.

```python
from anyjev import Decider, Question
from anyjev.backends.hf import HFBackend          # or anyjev.backends.vllm.VLLMBackend(url, model)

d = Decider(HFBackend("Qwen/Qwen3-8B"))           # level="L0" by default
route = Question.choice("Which team should handle this?",
                        ["billing", "technical", "sales", "other"], name="route")
r = d.decide("My card was charged twice for one order.", [route])["route"]
r.distribution, r.level                           # {option: probability}, "L0"
```

L0 reads a K-option choice once per rotation of the options. With the **rotation budget**, it reads
only as many rotations as the decision needs:

```python
d = Decider(backend, adaptive_shifts=True, canonical_order=True)
d.calibrate_adaptive(route, unlabelled_tickets, target=0.01)   # a few hundred states, no labels
```

The stopping threshold is calibrated against our own full-cycle answer, so it needs no labels. At a
certified 1% disagreement it read 7.2 rotations instead of 18. That was 2.2× the decisions per second
on vLLM and 2.3×–2.7× on transformers, with accuracy unchanged
([docs/rotation_budget.md](https://github.com/nokia-applied-research/AnyJev/blob/main/docs/rotation_budget.md)).

## ✨ What it is

Give the model a state and a **typed question** (a choice, a yes/no or a score). You get back a
**decision with a probability over the options**, read from the model's next-token distribution.
Nothing is parsed. There are two ways in:

- **AnyJev Training-Free (`Decider`)** works on any open LLM, with no training. Raw label logits
  change their answer when the options are reordered and carry the model's preference for some
  labels. L0 removes both without a single label.
- **AnyJev Self-Distilled (`Tacit`)** uses the Tacit checkpoints. They are trained by
  self-distillation: the model learns to give, in one forward pass, the answers it reaches when it
  reasons. No human labels and no other model are involved. One prefill per decision; with
  `adaptive=True`, a capped share of decisions goes to reasoning.

<div align="center">

| | ⚪&nbsp;raw&nbsp;logits | 🔵&nbsp;**L0**<br><sub>zero labels</sub> |
|:--|:--:|:--:|
| Labels required | none | **none** |
| Answer flips when options are reversed | 0.230 | **0.073** |
| Accuracy | 0.747 | **0.803** |
| Calibration error (ECE) | 0.240 | **0.184** |
| **Auto-decidable at ≤5% error** | **7.7%** | **46.3%** |

<sub>Qwen3-8B, BANKING77 20-way, 300 test items · `bench/results_v01/2026-09-22/Qwen__Qwen3-8B.json`</sub>

</div>

The last row matters most. Accuracy moves six points, but the share of decisions confident enough to
automate at ≤5% error goes from **7.7% to 46.3%**, with no labels at all.

## 📊 Tacit results

<div align="center">

| model | base | JevBench public (231) | bev-decision test (46,320) |
|:--|:--|:--:|:--:|
| [Tacit-9B](https://huggingface.co/morriszjm/Tacit-9B) | Qwen3.5-9B | **0.823** | **0.725** |
| [Tacit-8B](https://huggingface.co/morriszjm/Tacit-8B) | Qwen3-8B | 0.749 | 0.662 |
| [Tacit-4B](https://huggingface.co/morriszjm/Tacit-4B) | Qwen3-4B | 0.723 | 0.663 |
| [Tacit-2B](https://huggingface.co/morriszjm/Tacit-2B) | Qwen3.5-2B | 0.671 | 0.626 |
| [Tacit-1.7B](https://huggingface.co/morriszjm/Tacit-1.7B) | Qwen3-1.7B | 0.623 | 0.598 |

<sub>Accuracy with one forward pass per decision (`adaptive=False`), the options in the benchmark's
order, on every item of both test sets ([JevBench](https://github.com/fstandhartinger/jevbench) public
set; [bev-decision](https://huggingface.co/datasets/avbiswas/bev-decision) test split) ·
`bench/results_tacit/2026-10-05/one_forward.json`</sub>

</div>

Results with escalation (`adaptive=True`) will be published together with the evaluation harness.

## 🧠 How it works

<p align="center">
  <img src="https://raw.githubusercontent.com/nokia-applied-research/AnyJev/main/assets/how_it_works.png" width="100%" alt="How one decision is read: ask a typed question, read it over every cyclic shift of the options, divide out the label prior estimated without labels, and return a decision that carries its level">
</p>

| | needs | does | does **not** |
|---|---|---|---|
| `raw` | nothing | one prompt, softmax restricted to the option labels | correct order or label bias |
| `L0` | nothing | averages position bias out over the K rotations, divides out the label prior | calibrate against labels |
| Tacit `one_forward` | a Tacit checkpoint | one prompt, read like `raw` from a model trained to answer in one pass | correct order bias |
| Tacit `cot` | `adaptive=True` | reasons first, then reads the label distribution after the thought | run beyond its cap |

Every `Decision` carries its `level`, and `require="L0"` makes downstream code refuse a weaker one.
Every Tacit decision carries its `route`.
[The contract in full →](https://github.com/nokia-applied-research/AnyJev/blob/main/docs/levels.md)

<sub>Every number in this README is read from committed JSON. Not affiliated with TypeSafe AI or Jev.</sub>

## 🧭 Roadmap

- [x] `choice`, `noul` and `score` from one prefill; L0 with zero labels
- [x] The rotation budget: L0 at about 7 of 18 rotations, with a stopping threshold certified without labels
- [x] **Tacit-1.7B, 2B, 4B, 8B and 9B** on Hugging Face; `Tacit` in the library on transformers, vLLM and `vllm serve`, with capped escalation to reasoning and an HTTP gateway
- [ ] 🚧 Results with escalation on JevBench and bev-decision, released with the evaluation harness
- [ ] 🚧 Label-free early exit: read a decision from part of the model's depth, choosing the depth by agreement with the full model
- [ ] **Agent-loop evaluation**: the same decisions inside a real agent
- [ ] More log-prob backends (SGLang, llama.cpp, MLX, Ollama), span readout beyond 26 options

Dated plan and help-wanted files: [ROADMAP.md](https://github.com/nokia-applied-research/AnyJev/blob/main/ROADMAP.md). Known limitations: [docs/limitations.md](https://github.com/nokia-applied-research/AnyJev/blob/main/docs/limitations.md).

## 🤝 Contributing and citation

Backends are one file each, and several are **help wanted** ([ROADMAP.md](https://github.com/nokia-applied-research/AnyJev/blob/main/ROADMAP.md), [CONTRIBUTING.md](https://github.com/nokia-applied-research/AnyJev/blob/main/CONTRIBUTING.md)). Changes: [CHANGELOG.md](https://github.com/nokia-applied-research/AnyJev/blob/main/CHANGELOG.md). Credits: [CREDITS.md](https://github.com/nokia-applied-research/AnyJev/blob/main/CREDITS.md).

Technical Report:
```bibtex
@misc{zhang2026anyjevtechnicalreport,
      title={AnyJev Technical Report}, 
      author={Jiamu Zhang and Tianze Yang and Yucheng Shi and Evan Chen and Zixiang Nie and Kelly Wan and Liangjie Hong and Ninghao Liu and Liang Wu},
      year={2026},
      eprint={2610.00831},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2610.00831}, 
}
```

Software:
```bibtex
@software{anyjev2026,
  title  = {AnyJev: Turn any LLM into a Jev-style decision model},
  author = {Zhang, Jiamu and Yang, Tianze and Shi, Yucheng and Wu, Liang},
  year   = {2026},
  url    = {https://github.com/nokia-applied-research/AnyJev}
}
```

Apache-2.0, see [LICENSE](https://github.com/nokia-applied-research/AnyJev/blob/main/LICENSE). The Tacit models carry their base models' Apache-2.0 license. Datasets keep their own licenses, see [THIRD_PARTY.md](https://github.com/nokia-applied-research/AnyJev/blob/main/THIRD_PARTY.md).
