Metadata-Version: 2.5
Name: pyprocessors-jev
Version: 1.6.8
Summary: Processor based on the Jev System One API
Project-URL: Homepage, https://bitbucket.org/kairntech/pyprocessors_jev
Author-email: Olivier Terrier <olivier.terrier@kairntech.com>
License: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: log-with-context
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pymultirole-plugins<1.7.0,>=1.6.0
Requires-Dist: python-singleton-metaclasses
Requires-Dist: requests
Provides-Extra: dev
Requires-Dist: bump2version; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Provides-Extra: docs
Requires-Dist: lxml-html-clean; extra == 'docs'
Requires-Dist: myst-parser; extra == 'docs'
Requires-Dist: sphinx; extra == 'docs'
Requires-Dist: sphinx-rtd-theme; extra == 'docs'
Requires-Dist: sphinxcontrib-apidoc; extra == 'docs'
Provides-Extra: sbom
Requires-Dist: cyclonedx-bom; extra == 'sbom'
Requires-Dist: pip-audit; extra == 'sbom'
Provides-Extra: test
Requires-Dist: dirty-equals; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest-dotenv; extra == 'test'
Requires-Dist: ruff; extra == 'test'
Description-Content-Type: text/markdown

# pyprocessors-jev

Processor based on the [Jev](https://typesafe.ai) *System One* API: it asks typed questions about a
document and gets back **calibrated probabilities** instead of text, so nothing has to be parsed out
of a completion.

Two providers, one protocol (`POST {base_url}/v1/systemone`):

| entry point | provider | base url | model | api key |
|-------------|----------|----------|-------|---------|
| `jev` | hosted TypeSafe | `JEV_API_BASE`, else `TYPESAFE_BASE_URL`, else `https://api.typesafe.ai` | `JEV_MODEL`, else `jev-latest` | `JEV_API_KEY`, else `TYPESAFE_API_KEY` |
| `openjev` | self-hosted [Open-Jev](https://github.com/Zefan-Cai/Open-Jev) | `OPENJEV_API_BASE`, else `http://127.0.0.1:8791` | `OPENJEV_MODEL`, else `open-jev` | `OPENJEV_API_KEY` (usually none) |

## What it does, in v1

**The v1 does one thing**: `add_categories` — one `choice` question over the project labels, whose
probabilities become the document categories. One document, one question, one HTTP request.

The API allows more, and the processor already carries it, **commented out** in
`src/pyprocessors_jev/jev.py` (and in `tests/test_jev.py`, and in the table below): free-form
`questions` on any target, one `noul` question per label for a multilabel decision, filtering
existing annotations, picking the best alternative text. They come back by uncommenting them —
the v1 keeps one option to explain and one behaviour to support.

Jev *chooses* and *rates*; it does not write. That bounds what any of it will ever produce:

| output | how | not possible |
|--------|-----|--------------|
| `categories` | `choice` over the project labels — the v1 | — |
| `metadata` *(v1+)* | typed values only: `noul` → boolean, `choice` → key of a closed set, `score` → number | free-form extraction (dates, amounts, names) — use `pyprocessors_openai_completion` |
| `annotations` *(v1+)* | filter existing candidate spans | create spans: no offsets come back |
| `altTexts` *(v1+)* | select among the texts already on the document (rerank / judge) | generate a summary or a translation |

Because the API answers **many questions in one call**, a document that will one day be classified
*and* have its metadata filled still costs one HTTP request — there is no `--- METADATA ---`
section to split off.

## Usage

```python
from pymultirole_plugins.v1.schema import Document
from pyprocessors_jev.jev import JevProcessor, JevParameters

processor = JevProcessor()
parameters = JevParameters(
    labels={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    instructions="Which team should handle this?",
)

docs = processor.process([Document(text="Help! My payouts have been failing for 3 days.")], parameters)
for cat in docs[0].categories:
    print(cat.labelName, cat.score, cat.properties)
```

## Options

| Option | Default | Description |
|--------|---------|-------------|
| `base_url` | provider default (see above) | Jev endpoint base url |
| `model` | provider default (see above) | model route |
| `function` | `add_categories` | the question built from `labels`. One value in v1: `add_categories`, a single `choice` over the labels. *(v1+: `add_multilabel_categories` — one `noul` per label, same call; `filter_annotations` — one `noul` per annotation, offsets preserved; `select_altText` — `choice` over the alternative texts.)* |
| `instructions` | `Choose the best category for this text.` | instructions of the question built from `function` |
| `labels` | – | label name → description mapping, injected from the project label set. The description is what Jev reads to decide; the categories written on the document carry **`labelName` only**, never `label` — a description has no place on the document, and the project label set already holds the display name. |
| `state_altText` | – | **which text Jev reads to answer** — what the API calls the *state*. Empty, it is the text of the document. Set to the name of an alternative text, it is that text: how a document is classified on what an upstream processor produced (a cleaning, a translation, a summary, the segment a retrieval step kept, a rendering of the metadata) without copying anything or running a second pipeline. A document that has no alternative text of that name is classified on its own text, with a warning, rather than skipped. `max_chars` truncates whichever is sent. |
| `decision_altText` | – | keep questions, answers and token usage in that alternative text — the audit trail |
| `multi_label_threshold` | `0.5` | the bar a probability has to clear to become a category, and with it how many labels a document comes back with. At `0.5` the decision is single-label — the probabilities of a `choice` are exclusive and sum to 1, so at most one clears the bar. Lower it and the same question becomes multilabel: every label above the bar becomes a category, sorted by decreasing probability |
| `keep_best` | `false` | when no answer of a `choice` reaches `multi_label_threshold`, keep the most probable one anyway, so the document never comes back without a category. A no-op with 3 labels (the winner is mechanically above 1/3), a safety net with a large label set |
| `max_chars` | `0` | truncate the state, `0` sends it whole |
| `timeout` | `60.0` | HTTP timeout, in seconds |
| `max_retries` | `3` | retries of a throttled (429) or overloaded (529) call, exponential backoff, obeying `Retry-After`. A timeout or a dropped connection — the call that never reached a status code — is retried the same way |
| `concurrency` | `1` | how many documents are sent **at the same time**. One document is one call whatever this is set to: it changes the wall clock, never what the model reads, so the decision trace and the blast radius of a failed call stay per document. A call is almost pure network wait, so N at a time divides the duration by about N until the API throttles, which `max_retries` absorbs. Measured on 40 questions of the Cairn corpus: 298 ms per document sequentially, 76 ms at 4, 44 ms at 8, 25 ms at 16 |

A document whose call fails is logged and left untouched — it comes back without a category rather
than taking the rest of the batch down with it, which is what `process` used to do.

Answers land in a predictable shape: a category carries the **probability** as its `score` and the
question `confidence` in its `properties`. *(v1+: a `noul` in metadata writes a boolean under the
question id plus its probability under `<id>_probability`; a `choice` and a `score` write their
value plus `<id>_confidence`.)* An answer naming a choice outside the `criteria` that were sent is dropped with a
warning rather than written as a label the project does not have.

The official `typesafe_sdk` is deliberately not a dependency: it cannot talk to an Open-Jev server,
which would mean two code paths for one protocol.

## Development

The build is driven by [Task](https://taskfile.dev) and [uv](https://docs.astral.sh/uv/),
with the shared stages coming from the `python-archetype` submodule.

### Getting started

The stages live in a Git submodule, so **clone with `--recurse-submodules`**:

```bash
git clone --recurse-submodules git@bitbucket.org:kairntech/pyprocessors_jev.git
cd pyprocessors_jev
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task
```

Already cloned without it? The submodule directory is empty, and `task` fails on:

```
task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"
```

which means exactly that, and nothing worse:

```bash
git submodule update --init
```

**Task is the only manual prerequisite.** An archetype cannot bootstrap itself: uv and the
Python interpreter install themselves on demand (every task that runs uv depends on an
internal `install-python` task), but the thing that runs them does not. Make sure
`~/.local/bin` is on your `PATH` — that is where `task` and `uv` both land.

### Running the pipeline

```bash
task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs
```

`task` with no argument is safe by construction: it runs every stage but the last, and that
bound is computed from the `STAGES` list rather than written down. The last stage is the
only one with an effect outside your machine.

`STAGES`, declared once in `Taskfile.yml`, is the single definition of the pipeline order —
so what you run locally is what Jenkins runs.

### Individual stages

| Task                            | Description                                           |
|---------------------------------|-------------------------------------------------------|
| `task py:sync`                  | Install the project and its dependencies (uv sync)    |
| `task py:lint`                  | `ruff check` and `ruff format --check`                |
| `task py:format`                | Reformat the code with ruff                           |
| `task py:test`                  | Run the test suite                                    |
| `task py:test-marker -- <m>`    | Run the tests carrying one pytest marker              |
| `task py:sbom`                  | Generate a CycloneDX SBOM of the resolved environment |
| `task py:check-vulnerabilities` | Check for known CVEs                                  |
| `task py:check-updates`         | Check for dependency updates                          |
| `task py:build`                 | Build the wheel and sdist (uv build)                  |
| `task py:publish`               | Publish the distributions (uv publish)                |
| `task py:version-file`          | Print the path of the file carrying `__version__`     |
| `task py:set-version VERSION=x` | Write that version into it                            |

`uv.lock` is not versioned here, so `py:sync` always resolves from scratch (`--upgrade`):
a stale lock lying around on a machine would otherwise make you test and audit versions the
CI never sees.

### Tests, and where the api key goes

The unit tests never open a socket: the `recorder` fixture replaces `JevClient.system_one`,
so the whole suite runs without a key and without a server. They check what the processor
*sends* (the questions built, `$labels` substituted, one call for every label) and how it
*reads back* the typed answers — not that Jev answers well. That last part is the job of the
single `integration` test.

Keys for that one live in `tests/.env`, which `.gitignore` keeps out of git
(`pytest-dotenv` loads it, same convention as `pyprocessors_openai_completion`):

```dotenv
# tests/.env
JEV_API_KEY=sk-...
# or, for a self-hosted server:
OPENJEV_API_BASE=http://127.0.0.1:8791
```

A key sitting there does **not** make `task py:test` hit the network: `addopts` carries
`-m 'not integration'`, so the default run stays hermetic and the live test is asked for
explicitly (the `-m` of the command line wins over `addopts`):

```bash
task py:test-marker -- integration
```

Without a key and without `OPENJEV_API_BASE`, that command skips instead of failing.

### Measuring a real label set

`tests/eval/` holds an evaluation harness for the Cairn question classifier: a frozen
dev/holdout split over 574 manually labelled questions, five label-description variants
with what each one scored, paired McNemar comparison, calibration and coverage curves, and
the saved model outputs so the numbers can be rechecked without spending tokens. It is not
part of the test suite — no file there is named `test_*`, so `task py:test` ignores it.
See `tests/eval/README.md`.

`tests/test_cairn_routing.py` pins the delivery configuration of that project — twelve real
questions of the corpus, the probabilities the API actually answered for them, and what the
processor must make of them at `multi_label_threshold=0.25`. It runs offline, like the rest of the suite.
