Metadata-Version: 2.4
Name: discern-vl
Version: 0.1.0
Summary: Ask a vision model a question, get the answer from its logits in one forward pass
Author-email: Samuel Otero Agraso <oteroagrasosamuel@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/XxSamaxX/discern
Project-URL: Paper, https://github.com/XxSamaxX/discern/blob/main/paper/paper.pdf
Project-URL: Issues, https://github.com/XxSamaxX/discern/issues
Keywords: vision-language-model,zero-shot-classification,logits,open-vocabulary,llm-inference,qwen
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE.upstream
Requires-Dist: torch>=2.4
Requires-Dist: torchvision>=0.19
Requires-Dist: transformers>=5.0
Requires-Dist: accelerate>=1.0
Requires-Dist: safetensors>=0.4
Requires-Dist: huggingface-hub>=0.26
Requires-Dist: numpy>=1.24
Requires-Dist: pillow>=10.0
Provides-Extra: bench
Requires-Dist: datasets>=3.0; extra == "bench"
Requires-Dist: matplotlib>=3.8; extra == "bench"
Dynamic: license-file

# discern

[![PyPI](https://img.shields.io/pypi/v/discern-vl)](https://pypi.org/project/discern-vl/)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.22899311.svg)](https://doi.org/10.5281/zenodo.22899311)
[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)

![What it does](docs/hero.png)

```bash
pip install discern-vl
```

Ask a vision model a question. Get the answer out of its logits in one forward
pass — no decoding, nothing trained, no fixed label set.

```python
from discern import discern

if discern("photo.jpg", "is there an identifiable person?"):
    blur_faces()

v = discern("photo.jpg", "what is the camera viewpoint?",
            ["from above", "at eye level", "from below"])
print(v)            # at eye level  [16.8 nats]
```

A predicate a detector cannot express — *"is this photo posed?"*, *"is the
white balance wrong?"*, *"is the meal already in progress?"* — costs one
prefill and about 90 ms.

**The confidence you get back is the logit gap in nats, not a probability** —
and the reason is not the one you would guess. The probability these systems
report is a *monotone transform of the gap*: on binary questions it is exactly
`p = sigmoid(gap)`, and both predict correctness equally well (AUC 0.895 vs
0.894 over 4,329 MMBench items). They are one signal.

What separates them is **float32**, the dtype your pipeline almost certainly
uses. Past **17.33 nats** — exactly `-ln(2**-25)` — the probability is no longer
representable in it:

| logit gap | probability | float32 |
|---|---|---|
| 12 nats | 0.99999386 | 0.99999386 |
| 20 nats | 0.999999998 | **exactly 1.0** |
| 25 nats | 0.99999999999 | **exactly 1.0** |

That is **62% of MMBench items** collapsed onto one value. In float16 the
threshold falls to 8.32 nats and it is 81%.
Within that collapsed set the gap still separates right from wrong at AUC
0.736. Same information, one scale that survives being stored.

So `discern` refuses to be used as a condition when the gap is too small:

```python
v = discern("blurry.jpg", "is the person wearing a helmet?")
bool(v)             # raises Uncertain: 1.2 nat gap, below the trust threshold
v.answer            # 'yes'  — still there if you want it anyway
v.trusted           # False
```

Below the threshold it also rotates the options and re-scores before answering,
because rotation changes the winner on 11.7% of items and those have a median
gap of 2.7 nats.

Several questions about one image:

```python
from discern import multi

for v in multi("photo.jpg", [
        "is there an identifiable person?",
        "is the white balance wrong?",
        ("what is the camera viewpoint?", ["above", "eye level", "below"]),
]):
    print(v, v.probabilities)
```

Built on [SemIf](https://github.com/TheoLeeCJ/SemIf) and the
[JEV-CPU](https://huggingface.co/Meanblock/JEV-CPU) port. The scoring engine is
vendored unchanged (MIT) so image results stay comparable to the published text
ones; see [THIRD_PARTY.md](THIRD_PARTY.md).

## Findings

**It works, and it is fast.** On **MMBench en/dev (4,329 items)** Qwen3-VL-4B
scores **87.4%** with one forward pass per item at **91 ms/item**, and **82.2%**
under MMBench's own CircularEval protocol. On our own 22-item probe it answers
22/22, including counting, absence, a false-premise question, an unanswerable
question, and reading an Adidas mark spanning ~20×15 px.

**The probability and the gap are the same signal; only one survives float32.**
*(the bars in the figure above are the gap)* We expected the gap to
out-predict the reported probability. It does not — both reach AUC ~0.894 on
MMBench, and requiring `p >= 0.99` retains 286 of 547 errors while its exact
equivalent, `gap >= 4.60 nats`, retains 288. The real difference is that 62% of
items share the single float32 value 1.0, where the gap still ranks them at AUC
0.736. **Threshold in nats**, not because it discriminates better but because
it is the only one you can still read afterwards. It buys a real operating
point:

| gap threshold | you answer | accuracy |
|---|---|---|
| none | 100% | 87.4% |
| ≥ 12 nats | 75% | 96.8% |
| ≥ 16 nats | 66% | **98.4%** |
| ≥ 20 nats | 55% | 99.1% |

**Position bias is 1–2 nats, and the gap tells you exactly when it matters.**
Rotating the options changes the winner on **11.7% of MMBench items**. Those
items have a **median gap of 2.7 nats**; the stable ones, 22.8. The gap predicts
order-stability with **AUC 0.980**. So: rotate and average only below ~3 nats —
a rule we wrote from a 22-item probe before downloading the dataset, and which
lands on the empirical median of 4,329 items.

**The gap will not protect you from questions the model cannot answer.** On 12
normative questions (consent, retention, licensing — policy no model can
ground) the gap distribution shifts but overlaps badly: AUC 0.788, *p* = 0.006,
and the best possible threshold still admits **7 of 12**. The model claims a
photo needs legal consent at 15.6 nats, about as confidently as it reports
there is no dog in the picture at 26.4. **Use semantic ifs for perception;
keep policy in code.**

**A semantic if is an `if`, not a `switch`.** Our first cascade asked one
four-option policy question whose options were not mutually exclusive, and the
model oscillated between two *correct* answers. Independent binary predicates
composed in Python fixed it.

**The gap signal weakens as options shrink.** On POPE (9,000 binary
yes/no questions about object presence) the same readout scores **89.6%**
(F1 89.0), but the gap only reaches **AUC 0.777** against MMBench's 0.894 (and so does the
probability — they stay equivalent). With two slots
the model is *confidently* wrong: its median gap when wrong is 16.4 nats, against
5.2 on MMBench. **Ask three or more options where you can**, and demand a much
higher threshold on binary questions.

POPE also shows where object hallucination actually hides. This model does not
over-assert presence — its yes-rate is 42.7–45.6%, *below* the 50% base rate, so
the usual headline metric says it is clean. But precision falls 98.2% → 94.0% →
92.0% from the random to the popular to the adversarial split while recall stays
at exactly 83.9% throughout. The co-occurring distractors do pull it into false
positives; the aggregate yes-rate just hides it.

**Where it breaks.** The worst MMBench categories are `spatial_relationship`
(34.5% under CircularEval, near chance) and `image_quality` (57.3%). Our 22-item
probe had flagged exactly these two: its lowest gaps were the spatial-relation
item (8.1 nats) and the white-balance item (7.8). The confidence signal
identified the weak categories from a handful of examples.

**An optimisation that does not work.** Questions about one image share a
prefix — the system turn plus ~300 visual tokens — so caching it and running
only each question's tail looks free. It is numerically exact (max probability
difference 0.0) and **13–14% slower**, flat across 384, 852 and 1,812-token
images, with no crossover: the cached path loses the fast attention kernel and
that costs more than recomputing the prefix. Image preprocessing is only 3% of
the time; the forward is 97%. Kept behind `use_cache=True` and a
`verify_cache()` checker in case flash-attn flips the balance.

**Systems note.** On a hybrid Intel CPU (i9-12900F, 8P+8E), restricting
inference to the 8 P-cores is **1.9× faster** than using 20 threads —
scheduling onto E-cores actively hurts. Overlapping a GPU vision stage with a
CPU text stage in two threads saved 22% wall clock.

Full write-up with method and statistics: [`paper/paper.pdf`](paper/paper.pdf).

## Layout

```
src/discern/__init__.py       the API: discern(image, question) -> Verdict
src/discern/_batch.py         multi(): several questions, one image
src/discern/_readout.py       the vision port; readout identical to upstream
scripts/eval_probe.py         runs the hand-built 22-item probe
src/discern/_semif/           vendored upstream engine (MIT)
scripts/cascade.py            GPU predicates -> CPU policy, threaded pipeline
scripts/bench_cpu4b.py        CPU thread sweep and policy benchmark
scripts/eval_mmbench.py       MMBench harness (resumable JSONL)
scripts/analiza_mmbench.py    accuracy, AUC, abstention curves
scripts/eval_pope.py          POPE harness (9,000 binary questions)
scripts/analiza_pope.py       F1, yes-rate, precision/recall by split
scripts/exp_gap_por_tipo.py   the perceptual-vs-normative experiment
scripts/figuras.py     paper figures
data/                  benchmark decisions + 4 COCO val2017 images
results/               raw output of every run reported
paper/                 LaTeX sources and compiled PDF
```

## Running it

The library is `pip install discern-vl`. To reproduce the measurements you want
the repo and the benchmark extras:

```bash
git clone https://github.com/XxSamaxX/discern && cd discern
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -e ".[bench]"

.venv/bin/python scripts/eval_probe.py --rows data/vl_rows_hard.json --calibrate
.venv/bin/python scripts/exp_gap_por_tipo.py
.venv/bin/python scripts/bench_cpu4b.py      # needs ~21 GiB RAM

# MMBench: ~7 min vanilla, ~25 min CircularEval
.venv/bin/python scripts/eval_mmbench.py
.venv/bin/python scripts/eval_mmbench.py --circular
.venv/bin/python scripts/analiza_mmbench.py results/mmbench_*_4329.jsonl

# POPE: ~34 min for 9,000 binary questions
.venv/bin/python scripts/eval_pope.py
.venv/bin/python scripts/analiza_pope.py
```

First run downloads Qwen3-VL-4B-Instruct (~8 GB).

## Caveats

The MMBench numbers are one model on one split, single run, with one prompt
template; no confidence intervals on latency. MMBench dev is public and may
sit in the model's training data, so 87.4% is not evidence about generalisation
— the claims that matter here are about the *confidence signal*, which is
measured within the same run and does not depend on the absolute score.

Our own 22-item probe is small and author-annotated; one label was wrong (we
called a logo illegible, the model read it correctly) and was corrected. Its
22/22 never located the model's failure boundary, which is why the MMBench run
exists.

## Citing

```bibtex
@software{oteroagraso2026discern,
  author  = {Otero Agraso, Samuel},
  title   = {discern: semantic ifs over images from vision-language logits},
  year    = {2026},
  doi     = {10.5281/zenodo.22899311},
  url     = {https://github.com/XxSamaxX/discern}
}
```

## Licence

MIT, matching upstream. See [LICENSE](LICENSE) and [THIRD_PARTY.md](THIRD_PARTY.md).
