Metadata-Version: 2.5
Name: groundgate-calibrate
Version: 0.2.0
Summary: Measure a judge's thresholds for groundgate on your own documents.
Project-URL: Homepage, https://github.com/mohanraj00/groundgate/tree/main/calibrate
Project-URL: Source, https://github.com/mohanraj00/groundgate
Project-URL: Issues, https://github.com/mohanraj00/groundgate/issues
Author: Mohan Raj
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: groundgate>=0.5
Description-Content-Type: text/markdown

# groundgate-calibrate

Measure a judge's thresholds for groundgate on your own documents.

A judge is a typed-decision model that answers a question about a flagged or admitted value. A
threshold belongs to one model and one kind of document: the field doubt gave 0.3 on FDA labels
([#106](https://github.com/mohanraj00/groundgate/issues/106)) and 0.2 on NTSB reports
([#110](https://github.com/mohanraj00/groundgate/issues/110)). So groundgate ships no thresholds. This tool measures
yours, as [hybrid design](https://github.com/mohanraj00/groundgate/blob/main/docs/design/hybrid-decisions.md) §9 describes. It is a separate package, so the groundgate core never
contains code that calls a model.

```bash
pip install groundgate-calibrate
```

[Measure a judge's thresholds](https://github.com/mohanraj00/groundgate/blob/main/docs/howto/calibrate.md) runs every step once, on made-up documents with a test
judge. This page is the reference.

## Steps

```bash
groundgate-calibrate sample --work W --docs D --candidates C --schema S --question key \
    --context "The text is from a 10-K filing."
groundgate-calibrate split  --work W        # before any model run
groundgate-calibrate label  --work W        # label in the browser, blind, before ask
groundgate-calibrate ask    --work W --judge NAME   # NAME: a judge (see Judges)
groundgate-calibrate report --work W        # REPORT.md, report.json
```

- `D` holds one `<doc>.txt` for each document.
- `C` holds one `<doc>.json` for each document: a list of your extractor's candidates in the
  candidate format of the installed groundgate's spec (0.6 with groundgate 0.6.1).
- `S` is a schema of that spec. `sample` reads each field's `description` for the field
  question and the label page. Without a description, the field question uses the field name.
- `--descriptions` is an optional JSON file that maps field names to meanings. Each entry
  replaces that field's schema description. Use it with spec 0.5 schemas, which have no
  `description`. The tool needs groundgate 0.5 or later.

- `label` serves the label page at `http://localhost:8780`. `--port` changes the port. Each
  click saves a label in `W/labels.json`: `true`, `false` or `null` (not sure) for the field
  question, and the list of the field's keys (empty for none) for the key question.
- `report --check` writes nothing. It fails when `REPORT.md` or `report.json` differs from what
  `W` gives, so a CI job can check a committed report.

If extraction lost the shape of your tables, put a `links.json` in `W` that maps each document
name to its original, such as the source web page. The label tool then links to it.

`W` keeps everything that a later step needs: the config, a copy of each document with items,
the items, the labels, the split, the answers and the report. A step never asks for an earlier
input again.

## Questions

- **`key`: the key clear.** The items are the candidates that groundgate flags
  `KEY_NOT_AT_VALUE` with no judgment. A judge chooses one of the field's keys, or none. A judge clears a flag
  when it chooses the candidate's key with a confidence of t or more. The clear is right when
  the label holds that key, and an escape when it does not. The ceiling is on the escape rate
  among clears, and the tool takes the lowest threshold that keeps the bound below it.
- **`field`: the field doubt.** The items are the candidates that groundgate admits with no judgment. A judge
  gives the probability that the text states the value as the field. A judge doubts a value
  when that probability is below t. A doubt catches a wrong value, and it sends a right one to
  review. The ceiling is on the share of right values doubted, and the tool takes the highest
  threshold that keeps the bound below it.

The bound is the one-sided 95% upper bound (Clopper-Pearson) on the calibration part. The
report gives the test part at each threshold, and a table of the answers by confidence.

## Before you label

`sample` prints how many right items each ceiling needs with no failure: 59 at 0.05, 29 at 0.1
and 14 at 0.2. The split puts about half of the items in the calibration part. Take enough
documents for the ceiling that you need before you start.

## Rules that the tool keeps

- **Blind labels.** The label tool shows the document, the marked value, the field and the
  options. It never shows the candidate's key, groundgate's reading or a judge's answer.
- **Split first.** `split` is written once, by document, on sha256 of the document name. It
  refuses to run after a judge has answered, and `ask` refuses to run without it.
- **Labels first.** `ask` refuses to run until every item has a label (not sure counts), and it
  records a digest of the labels. The label tool refuses changes once a judge has answered, and
  `report` refuses answers whose labels changed.
- **Fixed prompts.** `split` also records a digest of every prompt. `ask` refuses to run when the
  documents, items or config changed after the split, and `report` refuses answers to other
  prompts. `ask` goes on after a stop only with the same judge, model and prompts.
  The field prompt digest includes its description. A new description needs a new calibration.
- **Fixed scoring.** `ask` also records a digest of the items and the split, and `report`
  refuses answers when either changed.
- **One spec.** `sample` records the groundgate spec version. Every later step refuses to run
  under another version, because another spec can admit and flag other candidates.
- **Your offsets stay right.** `sample` refuses a document that is not NFC, and never changes
  the text, because your candidates' byte offsets point into it.
- **Text leaves the machine.** A hosted judge sends a window of each document out: 400 code
  points before the value and 150 after. Use it only on documents that you may send.

## Judges

Two judges are built in. `groundgate-calibrate ask --help` names them. `jev` is a hosted judge.
It reads its API key from the environment and sends a window of each document out of the machine.

`chat` uses a server that accepts chat requests. Set these environment variables:

- `GROUNDGATE_CHAT_URL`: the base URL, such as `http://127.0.0.1:11434/v1`. It must be `https`,
  or `http` to this machine (`localhost`, `127.0.0.1` or `::1`). A request to this machine never
  goes through a proxy.
- `GROUNDGATE_CHAT_MODEL`: the model name, exactly as the server reports it in each response.
- `GROUNDGATE_CHAT_VERSION`: your version string, such as the local model's file digest.
- `GROUNDGATE_CHAT_KEY`: optional. When set, it goes in the `Authorization: Bearer` header.

Run `groundgate-calibrate ask --work W --judge chat`. It sends the text window to
`<url>/chat/completions`. With a server on your machine, the text stays there. With a hosted
server, use only documents that you may send. The model states its probability in JSON:
`confidence` for a choice question and `p` for a noul question. The calibration measures that
number. The judge uses no log-probabilities.

The digest is `chat-1:<model>@<version>`. `chat-1` names the prompt template and answer parsing.
A change to either needs a new digest and calibration. `ask` and `judge` refuse a response that
reports another model name. An invalid JSON answer, choice or probability gets up to 2 retries.
After that, the judge stops and shows the reply. The request is the same each time, so a retry
helps only when the server does not give the same answer to the same request. The judge refuses
a redirect, so the key goes only to the URL that you set. If the server does not answer, the
judge stops and names the URL.

A judge has an `id`, a `digest` that names one model version, and `ask(state, questions)`.
`state` is the text around the value, with the value in brackets. `ask` answers each question in
the typed-decision format:

- The key question is a choice question. The answer is `{"choice": name, "confidence": p}`, where
  `name` is one of the question's `criteria`.
- The field question is a noul question. The answer is `{"noul": p}`, where `p` is the
  probability that the statement is true.

A package adds a judge with an entry point:

```toml
[project.entry-points."groundgate.judges"]
mine = "my_package:make_judge"   # called with no arguments, it returns the judge
```

Then `--judge mine` uses it. A plug-in cannot take the name of another judge, and the judge's
`id` must equal its name.

## In production: recorded judgments

Since spec 0.4, groundgate reads a judge's answers only as recorded inputs. `judge` writes them for your documents:

```bash
groundgate-calibrate judge --docs D --candidates C --schema S --policy policy.json \
    --calibration W-key W-field --out J
```

- The policy's `judge` block names the judge, the model version and the thresholds, as `report`
  wrote it. The judge must be that version, and each calibration must be of that version.
- Each question that the policy has a threshold for needs its calibration, and is asked as that
  calibration asked it: the same wording, context, descriptions and window.
  `judge` refuses a schema that gives a field another description than the field calibration had.
- A question is asked only where its answer can change the decision: the key question of each
  candidate flagged `KEY_NOT_AT_VALUE`, and the field question of each admitted candidate. With
  both thresholds, the field question also goes to each candidate whose only flag is
  `KEY_NOT_AT_VALUE`, because the key judgment can admit it.
- Give one calibration for each question.
- A judged candidate needs a unique `id`, because a judgment names its candidate by `id`.
- `J/<doc>.json` holds the judgments. Pass them to groundgate:
  `groundgate admit DOC SCHEMA CANDIDATES --policy policy.json --judgments J/<doc>.json`.

## The result

`report` writes, for each ceiling with a threshold, the `judge` block of a policy (hybrid
design §7) with the model and the threshold. Spec 0.4 and later read it, with the judgments that `judge`
writes ([#116](https://github.com/mohanraj00/groundgate/issues/116)). A block for the field
question looks like this:

```json
{"judge": {"id": "mine", "digest": "mine-2026-10", "doubt": {"field_match": 0.3}}}
```

A block for the key question has `"clear": {"KEY_NOT_AT_VALUE": t}` in place of `doubt`. To use
both thresholds, put both in one block.
