Metadata-Version: 2.4
Name: mellea-jev-adapter
Version: 0.1.1
Summary: Unofficial TypeSafe Jev Noul verifier and Choice classifier for Mellea
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx2<3,>=2.0.0
Requires-Dist: pydantic<3,>=2.12.0
Requires-Dist: typesafe-sdk<0.8.0,>=0.7.0
Provides-Extra: mellea
Requires-Dist: mellea==0.7.0; extra == "mellea"
Provides-Extra: dev
Requires-Dist: mypy<2,>=1.15; extra == "dev"
Requires-Dist: pytest<10,>=8.3; extra == "dev"
Requires-Dist: pytest-cov<8,>=6; extra == "dev"
Requires-Dist: ruff<0.17,>=0.16; extra == "dev"
Provides-Extra: laya
Requires-Dist: laya-mlx<0.2.0,>=0.1.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "laya"
Dynamic: license-file

# Mellea × Jev

[![CI](https://github.com/SoundBlaster/Jev4Mellea/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/SoundBlaster/Jev4Mellea/actions/workflows/ci.yml)
[![Version](https://img.shields.io/github/v/tag/SoundBlaster/Jev4Mellea?label=version)](https://github.com/SoundBlaster/Jev4Mellea/tags)
![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?logo=python&logoColor=white)
![Mellea 0.7.0](https://img.shields.io/badge/Mellea-0.7.0-6f42c1)
![TypeSafe SDK 0.7.x](https://img.shields.io/badge/TypeSafe%20SDK-0.7.x-6f42c1)
[![Coverage gate: 80%](https://img.shields.io/badge/coverage%20gate-%E2%89%A580%25-success)](https://github.com/SoundBlaster/Jev4Mellea/blob/main/pyproject.toml#L68-L70)

A small Python adapter that brings TypeSafe Jev's semantic checks into
[Mellea](https://github.com/generative-computing/mellea). Use Jev to verify
generated text, classify it into your labels, or rate it on a scale. Mellea
continues to manage generation and repair.

- **Verify requirements** with Jev Noul and configurable accept/reject thresholds.
- **Classify text** with TypeSafe Choice and caller-defined categories.
- **Score text** on an ordered scale with TypeSafe Score.
- **Ask several questions in one request** and inspect returned token usage.
- **Connect checks to Mellea `Requirement`s** so they can participate in sampling.

This is an unofficial, synchronous adapter. Jev evaluates text; it does not
generate or repair it.

## Quick start

Install from a checkout and set an API key from the
[TypeSafe console](https://console.typesafe.ai/). Live requests may incur
charges.

```bash
git clone https://github.com/SoundBlaster/Jev4Mellea.git
cd Jev4Mellea
make install
export TYPESAFE_API_KEY='your-key'
```

The Makefile defaults to Python 3.13. To use another supported interpreter,
run `make install PYTHON=python3.11` (or set `PYTHON` to your installed version).

Ask whether a candidate meets a positive requirement:

```python
from mellea_jev import JevClient, JevVerifier

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The answer gives the museum's opening time from the reference.",
        reference="The museum opens at 10:00 and closes at 18:00.",
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")
    print(verdict.outcome, verdict.p_yes)
```

`outcome` is `pass`, `fail`, or `uncertain`. Use `verifier.as_requirement()` to
attach the same check to a Mellea generation flow. Once the project dependencies
are installed, `make demo` runs without a key or network access using a mocked
Jev response.

## Examples

### Verify a requirement with Noul

Noul returns `p_yes`, the probability that a positively phrased requirement is
satisfied. The adapter uses two configurable thresholds; the space between
them is uncertain and is never silently accepted.

```python
from mellea_jev import JevClient, JevVerifier

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The candidate states the opening time supported by the reference.",
        reference="The museum opens at 10:00.",
        criteria={
            "true": "The candidate gives 10:00 as the opening time.",
            "false": "The candidate omits or contradicts the opening time.",
        },
        accept_at=0.90,
        reject_at=0.10,
        repair_hint="Use the opening time stated in the reference.",
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")

    if verdict.outcome == "uncertain":
        print("Route for another check or human review")
```

The threshold values are application policy, not accuracy guarantees. Noul does
not return a separate confidence field or a textual explanation.

To use the verifier as a Mellea requirement:

```python
requirement = verifier.as_requirement()
```

For generation and repair with a real Mellea session, see
[`examples/mellea_ollama.py`](examples/mellea_ollama.py). That example uses
Ollama for generation and Jev for verification.

### Classify into configured categories with Choice

Choice selects one label from the supplied criteria and returns its confidence
and the probability of every label. Descriptions may be strings, JSON objects,
arrays, or `None`.

```python
from mellea_jev import JevClient, JevClassifier

criteria = {
    "billing": {"what": "Payments, invoices, or refunds", "examples": ["duplicate charge"]},
    "technical": "Product errors or problems using the service",
    "other": None,
}

with JevClient() as jev:
    classifier = JevClassifier(
        jev,
        "Choose the best category for this support message.",
        criteria=criteria,
    )
    result = classifier.classify("I was charged twice for my subscription.")
    print(result.choice, result.confidence, result.probabilities)
```

To require a Mellea candidate to be assigned to a particular category:

```python
requirement = classifier.as_requirement(
    "billing",
    minimum_confidence=0.75,
)
```

Confidence thresholds are caller policy. TypeSafe supports up to 255 Choice
labels.

### Rate text on an ordered scale with Score

Score returns a probability-weighted position on an ordered scale. The result
can fall between two levels.

```python
from mellea_jev import JevClient, JevScorer

levels = ["Cosmetic", "Workaround exists", "Blocking"]

with JevClient() as jev:
    scorer = JevScorer(
        jev,
        "How severe is the reported issue?",
        criteria=levels,
    )
    result = scorer.evaluate("The export button crashes and there is no workaround.")
    print(result.score, result.confidence, result.probabilities)
```

The result can also become a Mellea requirement with inclusive score bounds:

```python
requirement = scorer.as_requirement(
    minimum_score=1.0,
    maximum_score=2.0,
    minimum_confidence=0.6,
)
```

Score supports 2–10 ordered string descriptions.

### Batch questions and read usage metadata

`JevClient.system_one()` sends named Noul, Choice, and Score questions in one
request. Each answer remains a typed result. The returned usage counts are
informational and do not affect validation.

```python
from mellea_jev import ChoiceQuestion, JevClient, NoulQuestion

with JevClient() as jev:
    result = jev.system_one(
        state={"candidate": "I was charged twice and cannot log in."},
        questions={
            "urgent": NoulQuestion("Does the message convey urgency?"),
            "team": ChoiceQuestion(
                "Which team should handle this?",
                {"billing": "Payments and invoices", "technical": "Product errors"},
            ),
        },
    )

    print(result.answers["urgent"].p_yes)
    print(result.answers["team"].choice)
    if result.usage is not None:
        print(result.usage.input_tokens, result.usage.output_tokens)
```

For one-off checks without the Mellea helper classes, call the client methods
directly. Each method sends its own request:

```python
with JevClient() as jev:
    yes_no = jev.noul(
        state={"candidate": "The museum opens at 10:00."},
        question="Does the candidate state the opening time?",
    )
    category = jev.choice(
        state={"candidate": "I was charged twice."},
        question="Choose a category.",
        criteria={"billing": "Payments", "technical": "Product errors"},
    )
    rating = jev.score(
        state={"candidate": "The export is broken."},
        question="Rate the impact.",
        criteria=["minor", "major"],
    )
```

The single-question result objects also expose optional usage metadata.

### Use a check during Mellea generation

For an existing Mellea session `m`, pass the adapter's requirement to
`instruct()`. Keep the Jev client open until sampling finishes because the
requirement calls it during validation:

```python
from mellea.stdlib.sampling import RepairTemplateStrategy
from mellea_jev import JevClient, JevVerifier, accepted_text

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The candidate states the opening time supported by the source.",
        reference="The museum opens at 10:00.",
    )
    sampled = m.instruct(
        "State the museum's opening time using this source: {{source}}",
        user_variables={"source": "The museum opens at 10:00."},
        requirements=[verifier.as_requirement()],
        strategy=RepairTemplateStrategy(loop_budget=3, concurrency_budget=1),
        return_sampling_results=True,
    )
    answer = accepted_text(sampled)
```

`accepted_text()` checks the final validation state before returning text. Do
not return `sampled.result` directly after failed or incomplete sampling.

### Run checks with local Laya-MLX

On Apple Silicon macOS, install the optional backend and load a Laya checkpoint.
The first load may download the model weights; inference then runs locally.

```bash
pip install -e '.[laya]'
```

```python
import laya_mlx
from mellea_jev import JevClassifier
from mellea_jev.providers import LayaProvider

agent = laya_mlx.load("aac6fef/laya-mlx")
classifier = JevClassifier(
    LayaProvider(agent),
    "Which team should handle this request?",
    criteria={"billing": "Payments and refunds", "technical": "Bugs and outages"},
)
print(classifier.classify("I was charged twice.").choice)
```

`LayaProvider` accepts an already loaded agent and does not import Laya or MLX
when the base package is imported. It maps Laya's Noul, Choice, and Score
answers into the same response contracts used by `JevClient`. Review the
selected model checkpoint's license separately; the Laya-MLX runtime is
Apache-2.0 licensed. Laya computes Choice confidence from normalized entropy,
so calibrate `minimum_confidence` for the selected backend rather than copying
a threshold from another provider. See the [Laya-MLX API and platform notes](https://github.com/mizorewww/laya-mlx)
and [confidence implementation](https://github.com/mizorewww/laya-mlx/blob/main/laya_mlx/common.py).

### Evaluate a provider on labeled examples

`examples/evaluate.py` reports false-acceptance, false-rejection, and uncertain
rates for every provider, returned model, and threshold pair. The checked-in
museum example is a small format demonstration, not a quality benchmark. Add
representative, non-sensitive examples for your own task before drawing quality
conclusions. See the [initial live Jev and Laya run](examples/evaluation/live-benchmark-2026-09-21.md)
for a four-example smoke benchmark and its limitations.

The dataset is versioned JSONL: the first line describes the positive Noul
requirement; each following line labels one candidate as `accept` or `reject`.
An optional `reference` is sent with that candidate. For example:

```jsonl
{"type":"dataset","format_version":1,"name":"support-policy","version":"1.0.0","requirement":"The answer follows the refund policy."}
{"type":"example","id":"in-policy","candidate":"...","reference":"...","expected":"accept"}
{"type":"example","id":"out-of-policy","candidate":"...","reference":"...","expected":"reject"}
```

Inference is opt-in. This command sends each example to Jev and saves the raw
predictions, so it may incur charges and transmits dataset text to TypeSafe:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --live --provider typesafe --model jev-latest \
  --threshold 0.10,0.90 --save-predictions /tmp/museum-predictions.jsonl
```

To compare multiple threshold pairs or reproduce a report, load the saved
predictions offline. No provider is constructed and no request is sent:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --predictions /tmp/museum-predictions.jsonl \
  --threshold 0.10,0.90 --threshold 0.20,0.80
```

Use `--live --provider laya --model aac6fef/laya-mlx` to run the same labeled
examples through Laya-MLX on a supported Apple Silicon setup. Loading that
checkpoint may download model weights. Prediction snapshots contain `p_yes`,
provider, and returned model; keep them with the dataset version. The report
also includes the SHA-256 of the exact dataset file, so a changed file cannot be
silently paired with old predictions. A snapshot records the provider, returned
model, and raw probability for each example:

```jsonl
{"type":"prediction_set","format_version":1,"dataset":"support-policy","dataset_version":"1.0.0","dataset_sha256":"..."}
{"type":"prediction","id":"in-policy","provider":"typesafe","model":"jev-1.13.0","p_yes":0.98}
```

The report defines false-acceptance rate as false accepts divided by expected
rejects, false-rejection rate as false rejects divided by expected accepts, and
uncertain rate as uncertain predictions divided by all examples. It also
reports counts and denominators; missing classes are rejected during dataset
loading. Always publish the dataset version, sample count, provider/model,
thresholds, and limitations alongside any observed quality rates. Do not commit
private or sensitive examples or prediction snapshots.

## Requirements and compatibility

- Python **3.11 or newer**.
- Mellea **0.7.0** for the `Requirement` integration.
- TypeSafe API access and `TYPESAFE_API_KEY` for live Jev requests. Mocked tests and `make demo` need no key.
- The TypeSafe provider uses the official TypeSafe Python SDK and its synchronous HTTPX2 transport.
- `laya-mlx` is optional and supported by its upstream project on Apple Silicon macOS.

The CI compatibility matrix runs the package checks with these combinations:

| Python | Mellea integration |
| --- | --- |
| 3.11 | 0.7.0 |
| 3.12 | 0.7.0 |
| 3.13 | 0.7.0 |
| 3.14 | 0.7.0 |

The Mellea extra is pinned to 0.7.0; other Mellea versions are not currently
declared compatible.

The Mellea requirement callback is synchronous, so a Jev request can block the
event loop. This package does not provide an async client. If sampling has
already seen a failed candidate, Mellea may return a failed sampling result
instead of propagating a later Jev error or uncertain verdict; inspect the
final result with `accepted_text()`. See [API notes](API_NOTES.md) for external
contracts and [test report](TEST_REPORT.md) for the evidence behind the current
prototype status.

The Mellea helpers depend on small structural protocols: `NoulProvider`,
`ChoiceProvider`, and `ScoreProvider` (or the combined `PrimitiveProvider`). A
custom backend can implement only the primitive it needs; it does not need to
inherit from a package class. Its response must expose the fields in
`NoulResponse`, `ChoiceResponse`, or `ScoreResponse`. The current criteria
shapes follow the TypeSafe request model, and each provider may impose its own
limits. Batched requests remain a TypeSafe feature. Use `JevClient` as the
existing compatible name, or import `TypeSafeProvider` explicitly from
`mellea_jev.providers`. See [provider contracts](src/mellea_jev/contracts.py).

## Development and further reading

```bash
make help    # list the repository commands
make check   # run lint, formatting, type, test, coverage, and whitespace checks
make demo    # run without API keys or network access
```

GitHub CI runs the same `make check` gate: Ruff linting and formatting,
strict Mypy checks, a maximum cyclomatic complexity of 16, and at least 80%
branch-aware coverage. Live service requests remain opt-in and are not part of CI.

- [API notes and source references](API_NOTES.md)
- [Test report and validation limits](TEST_REPORT.md)
- [Development commands](Makefile)
- [Release process](RELEASING.md)
- [Roadmap](roadmap.md)
- [Offline demo](examples/offline_demo.py)
- [Live Jev example](examples/live_check.py)

Before using the adapter with private data, account for the fact that candidate
text and any supplied reference are sent to TypeSafe. The adapter does not log
request bodies or API keys, and it does not follow redirects or retry requests
automatically. See the [API notes](API_NOTES.md) for details.

Licensed under MIT. This project is unofficial and is not affiliated with Mellea, IBM, or TypeSafe.
