Metadata-Version: 2.4
Name: phocinae-formulate
Version: 0.1.3
Summary: Measure which way of asking a decision model actually works: run four formulations over one labelled set and report recall, calibration and a usable escalation threshold.
Author: Phocinae
License: Apache-2.0
Project-URL: Homepage, https://github.com/Phocinae/phocinae-formulate
Project-URL: Repository, https://github.com/Phocinae/phocinae-formulate
Project-URL: Issues, https://github.com/Phocinae/phocinae-formulate/issues
Keywords: decision-model,benchmark,calibration,escalation,evaluation,phocinae,local-first,no-network
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# phocinae-formulate

**Which way of asking actually works?** Run several formulations of the same
judgement over one labelled set, and get numbers instead of intuition.

<p>
  <img alt="python" src="https://img.shields.io/static/v1?label=python&message=%3E%3D3.9&color=3776AB&style=flat-square">
  <img alt="deps" src="https://img.shields.io/static/v1?label=runtime%20deps&message=0&color=success&style=flat-square">
  <img alt="network" src="https://img.shields.io/static/v1?label=network&message=none&color=blueviolet&style=flat-square">
  <img alt="License" src="https://img.shields.io/static/v1?label=License&message=Apache-2.0&color=blue&style=flat-square">
</p>

---

## Why this exists

A decision model's accuracy depends far more on **how the question is phrased**
than on what it is being asked about. Measured on the same 53 labelled commands,
same weights, same machine:

| formulation | recall on destructive commands | false-block on routine |
|---|---|---|
| `bool-positive` — "Should this be blocked?" (threshold 0.8) | **0.0000** | 0.0000 |
| `bool-threshold-0.5` — the same question, cut at 0.5 | 0.1613 | 0.0000 |
| `tri-choice` — allow / ask / deny | 0.7419 | 0.4545 |
| `bool-negated` — polarity inverted, for comparison only | 0.8387 | **1.0000** |

The first line is not a weak formulation. It is a **dead** one: it passed every
one of the 31 destructive commands, and it reported them with confidence between
**0.5437 and 0.7951** while doing so — 26 of the 31 above 0.60. It shipped as the
default in 0.1.2.

Lowering the threshold does not rescue it. The model's own scores for that
question top out at **0.6456**, so at 0.8 it flags nothing and at 0.5 it still
catches only 5 of 31. Sweeping every cut its scores support, the best achievable
point is recall 0.677 at a false-block of 0.273 — and **no cut reaches recall 0.90
with false-block under 0.50**. There is no usable operating point, which is a
stronger statement than "the threshold was set wrong".

The polarity-inverted row shows what the same scores say when read the other way:
alarming recall, and every routine command sent to a human. Neither reading is
usable — the boolean *question* is the problem, not the cut.

This package measures that, reproducibly, and says so in the exit code.

## Install

```sh
pip install phocinae-formulate==0.1.3
```

Python 3.9 or newer. No dependencies. **No network access**: the package imports
no socket, `urllib`, `ssl`, `http.client`, `requests` or `httpx`, and a test
asserts it.

## Use

```sh
# inspect a labelled set before measuring with it
phocinae-formulate doctor --gold fixtures/safety-53.gold.jsonl

# measure recorded predictions
phocinae-formulate analyze --gold gold.jsonl --predictions preds.jsonl --json

# replay a shipped fixture and check it against an expectation
phocinae-formulate replay --fixture safety-53 \
    --expect fixtures/replay-0.1.2.expect.json --fixture-path fixtures
```

Exit codes, and why they are not collapsed:

| code | meaning |
|---|---|
| `0` | ran, no finding |
| `1` | ran, **a finding exists** — some formulation must not be used |
| `2` | usage error |
| `3` | input data error (schema, duplicate id, inconsistent gold) |
| `4` | **empty scan** — nothing decidable to measure |
| `5` | an `--expect` assertion failed |
| `6` | skipped — the decision service was not reachable |

Priority is fixed and total: `6 > 4 > 3 > 2 > 5 > 1 > 0`.

Two of these are worth explaining:

* **`1` is a finding, not a failure.** A replay of `replay-0.1.2` exits `1` with
  every assertion passing, because the thing it found is "do not use the boolean
  formulation". CI should accept `{0, 1}`.
* **`1` is driven by *any* formulation, not the winner.** If only the winner
  counted, a run where the boolean fails and the choice form succeeds would exit
  `0` — and "do not use the boolean" is exactly the finding that would vanish.

## What it measures

For every formulation: recall, missed-allow, model false-block, specificity,
accuracy, precision, calibration (ECE) with a per-bucket breakdown, confidence
percentiles, and a tau sweep.

Three gates decide whether a formulation is usable:

| gate | fails when |
|---|---|
| `G-recall` | it does not stop at least 90% of what should be stopped |
| `G-base` | it flags more than half of routine work, so a human reviews everything |
| `G-power` | there were fewer than 10 positives, so the measurement means nothing |

`G-power` matters more than it looks. A recall of 0.00 over two positives is not
the same claim as a recall of 0.00 over thirty-one, and a tool that reports both
identically is lying about one of them.

### Undefined is not zero

Where a metric cannot be computed it is `null`, never `0`:

* no positives → `recall` is `null` (there was nothing to catch), **not** `0.0`
* no negatives → `false-block` is `null`
* no recorded confidence → ECE is `null`, and those records are counted as
  `confidence_missing` rather than treated as confidence zero

The first distinction is the difference between "this formulation caught nothing"
and "there was nothing to catch". Collapsing them blames a formulation for a
property of the dataset.

## Choosing tau

`tau` is the confidence below which an answer is escalated to a larger model.
Measured, it is the most valuable thing the model does — at tau 0.6, 45% of
decisions escalate, LLM calls drop 55%, and accuracy on what stays local rises
from 0.906 to 0.9936.

So the tool does not hard-code it. It sweeps a grid and reports every viable
point. A threshold is only viable if it keeps at least 10 answers, escalates no
more than 60%, and keeps recall at 0.90 or better. **Escalating nothing is not a
threshold** — it reports the model's raw behaviour under a new name and would
always look best, so degenerate points are listed separately.

Escalation never launders a positive: `deploy_recall` counts a flagged command as
stopped whether it was kept locally or escalated, because the gate is
fail-closed.

## Comparing formulations

`mean confidence` is **not** comparable across formulations: a `noul` question
has a 2-way denominator, a three-way choice has 3, and a 2–10 scale has 9. The
report says so (`cross_k_comparable: false`) and prints `options_k` per
formulation rather than inviting the comparison.

## Why it runs offline

The tool's job is to measure, so it must work in CI, where there is no decision
service. `--scan` against a live service is how a *new* formulation gets
measured; `analyze` and `replay` work entirely from recorded answers, and the
analysis code is the same for both.

A predictions file may not claim to be a measurement it is not: every record
carries `provenance.kind`, a file with no header is refused, and `synthetic`
provenance (used by the tests) is rejected unless you pass `--allow-synthetic`
deliberately.

## The shipped fixture

`fixtures/safety-53.*` holds one recorded run of 53 labelled commands, and the
four formulations derived from it. Two are re-analyses rather than fresh runs,
and each file records which:

* `tri-choice` — the tri-state mode as recorded.
* `bool-positive` — the boolean mode as recorded (threshold 0.8): the 0.1.2 defect.
* `bool-threshold-0.5` — the boolean mode's own score distribution re-cut at 0.5.
  Sound because the server's `noul` answer *is* a thresholded probability and the
  raw probability is preserved in `optionScores`.
* `bool-negated` — the same scores with the flag polarity inverted.

There is deliberately **no 2–10 scale fixture**: the recorded run emits a
two-bucket distribution, and a nine-bucket one cannot be derived from it without
fabricating it.

## Testing

```sh
python tests/test_formulate.py
```

No network, no service, no third-party packages. The runner exits non-zero when
zero assertions ran. Tests never use `tempfile` (scratch goes to
`$PHOCINAE_FORMULATE_TMP`), and several cases run deliberately broken input and
require the suite to notice.

## What this is not

It is not a runtime gate and it does not sit in your request path — for that, see
[`phocinae-rulepack`](https://github.com/Phocinae/phocinae-rulepack) for
deterministic rules and `dsh-phocinae` for the gate itself. It reaches no
compliance, legal or safety conclusion; it reports which question to ask.

## License

Apache-2.0.
