Metadata-Version: 2.4
Name: ocr-extraction-understanding-eval
Version: 0.1.0
Summary: Model-agnostic evaluation for user-defined operations over PDF and image data.
Project-URL: Homepage, https://github.com/ajiayi-debug/OCR_extraction_understanding_eval
Project-URL: Repository, https://github.com/ajiayi-debug/OCR_extraction_understanding_eval
Project-URL: Issues, https://github.com/ajiayi-debug/OCR_extraction_understanding_eval/issues
Author: ajiayi-debug
License-Expression: MIT
License-File: LICENSE
Keywords: document-ai,evaluation,llm-judge,ocr,pdf
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: pdf
Requires-Dist: pymupdf4llm>=0.2; extra == 'pdf'
Description-Content-Type: text/markdown

# OCR Extraction & Understanding Evaluation (`visual_evals`)

*An evaluation pipeline for OCR extraction and downstream manipulation of
document-derived data.*

## Abstract

Systems that read documents rarely stop at optical character recognition. A
production pipeline typically **extracts** structured values from a PDF or
image, then **manipulates** them through one or more downstream operations—
recomputing totals, classifying instruments, comparing against a base table,
summarizing, or routing. Errors compound across these stages, and a failure
observed at the end of the pipeline is not necessarily caused by the stage where
it surfaces.

`visual_evals` is a research-oriented, model-agnostic harness for evaluating
such multi-stage systems end to end. It follows the useful part of the Ragas
pattern—test cases, pluggable metrics, and inspectable reports—but makes three
distinctions that matter for document pipelines:

1. **Three levels of reference authority.** Every reference carries one of three
   authority levels, and the harness treats them differently:
   - `evidence` — a raw source (PDF, image, OCR/Markdown) that can support or
     contradict a prediction but is *not* assumed infallible;
   - `generated` — a *provisional, AI-generated ground truth* synthesized from
     the evidence when no human reference exists (see
     [Generating provisional ground truth](#generating-provisional-ground-truth-when-none-exists));
     agreement among models is a reliability signal, not proof of accuracy;
   - `ground_truth` — an authoritative human or trusted-system reference.

   A `generated` reference is never silently treated as authoritative: it is
   promoted to `ground_truth` only through an explicit, signed human approval
   step. Any mixture of the three can appear in a single test case.
2. **Deterministic where possible, judged where necessary.** Exact metrics run
   wherever structured ground truth exists; a visual LLM judge handles images,
   PDFs, layout, semantic equivalence, and rubric-based criteria.
3. **Dependency-aware failure attribution.** Each stage records its upstream
   dependencies, so the harness can localize *where* a pipeline failed rather
   than only reporting *that* it failed.

The library evaluates recorded inputs and outputs; it does not dictate or
execute the system under test. Domain policy—what each operation is and how its
correctness is decided—stays in the calling application.

## Research design goals

| Goal | Mechanism |
| --- | --- |
| Attribute failures across compounding stages | `depends_on` graph + `failure_context` |
| Avoid treating lossy OCR/Markdown as truth | evidence/ground-truth authority levels |
| Reduce single-judge bias | model-balanced `JudgePanelMetric` (PoLL-style) |
| Keep judgments auditable | versioned specs, SHA-256 prompt fingerprints, retained dissent |
| Separate the prompt-under-test from the correctness rule | `OperationSpec` vs. independently reviewed `EvaluationSpec` |

The voting and panel design draw on published findings on self-consistency,
panels of diverse evaluators, and documented LLM-judge biases; see
[Research-backed multi-judge voting](#research-backed-multi-judge-voting) for
citations and the important caveats on what agreement does and does not prove.

## Core contract

```text
                         ┌─ PDF / image ─────────── evidence
prediction JSON ─┬───────┼─ OCR / Markdown ─────── evidence
                 │       ├─ human notes ────────── ground_truth
                 │       └─ JSON / CSV base table ground_truth
                 ▼
       deterministic metrics + visual LLM judge
                 ▼
       JSON report + Markdown summary
```

- **Evidence** can support or contradict a prediction, but is not assumed infallible.
- **Ground truth** is an authoritative human or trusted-system reference.
- References are interchangeable and can be mixed in one test case.
- Exact metrics run where structured ground truth exists.
- The visual judge handles images, PDFs, layouts, semantic equivalence and rubrics.

## The extraction → manipulation pipeline

Evaluation does not stop after OCR or extraction. The unit of study is the whole
chain from source document to final decision:

```text
document ──▶ OCR / extraction ──▶ manipulation A ──▶ manipulation B ──▶ ... ──▶ decision
                                          │                 │                        │
                                  chosen metric      chosen metric           chosen metric
```

An operation can be extraction, calculation, comparison, validation,
summarization, routing, or anything else the caller defines (LLM prompt). `operation` is a
free-form string, not an enum or a library-controlled taxonomy. The library
evaluates recorded inputs and outputs; it does not dictate or execute the
system being tested.

Each `EvaluationCase` represents one stage and contains:

- `action_instruction`: the exact editable prompt or rules given to the action;
- `input_data`: the material received from an earlier stage;
- `prediction`: this stage's output;
- `references`: any mixture of evidence and authoritative ground truth;
- `rubric`: the caller's success rule for this operation;
- `judge_instruction`: optional stage-specific guidance for the evaluator;
- `operation`: a free-form identifier used to select metrics; and
- `depends_on`: upstream stage IDs.

`PipelineEvaluator` selects metrics in this order: exact stage ID, operation,
then default metrics. An operation can use a deterministic built-in metric, an
arbitrary Python function through `FunctionMetric`, an LLM judge, or several of
these together. This keeps domain policy in the application using the library.

The caller defines both sides of the contract:

1. `operation` and `rubric` say what the system was supposed to do.
2. `input_data` and `prediction` record what went in and what came out.
3. A metric says how correctness is decided. Use code when the rule is exact
   (such as arithmetic), and an LLM judge when the rule needs semantic or visual
   judgment.

### Dependency-aware failure diagnosis

Every stage report includes its own `verdict`, the verdicts of all upstream
stages, and a `failure_context`:

- `current_stage`: this stage failed while its upstream stages passed;
- `upstream`: this stage passed on the input it received, but an upstream stage
  failed;
- `upstream_or_combined`: both this stage and an upstream stage failed, so the
  result must not be blamed on this stage alone;
- `indeterminate`: an uncertain or non-applicable result prevents attribution.

This is careful failure localization, not automatic proof of causality. If both
extraction and a later action fail, rerun the action with corrected extraction
input. If it then passes, the extraction error caused the downstream failure; if
it still fails, the action also has an independent problem.

### Stage checkpointing

Long multi-model evaluations can checkpoint after every completed stage:

```python
report = evaluator.evaluate(
    pipeline,
    checkpoint_dir="eval_runs/my-case/checkpoints",
    checkpoint_context={
        "judge_provider": "anthropic",
        "judge_model": "claude-opus-4-8",
    },
)
```

Each report is written atomically. A rerun loads completed stages instead of
calling their metrics or judges again. A checkpoint is reused only when the
serialized stage case and `checkpoint_context` have the same fingerprint, so
include every external setting that can change the result, especially judge
provider, model, temperature, and prompt version.

## Installation

The project is a standard [`src`-layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/)
Python package built with [Hatchling](https://hatch.pypa.io/). Install it into
your environment as an editable dependency:

```bash
pip install -e .
```

The import package is `visual_evals`; the distribution is
`ocr-extraction-understanding-eval`:

```python
import visual_evals
```

The package does not create, load or require a `.env` file. Secret configuration
belongs to the application using the library.

For development and to run the test suite:

```bash
pip install -e ".[dev]"
pytest -q
```

Optional extras:

- `pip install -e ".[pdf]"` — PyMuPDF4LLM Markdown conversion for search/debug.

### Building and distributing

Because packaging metadata lives entirely in `pyproject.toml`, a wheel and
source distribution can be produced with the standard `build` frontend:

```bash
pip install build
python -m build          # writes dist/*.whl and dist/*.tar.gz
```

The resulting wheel is installable anywhere with `pip install dist/*.whl`, and
publishable to a package index with `twine upload dist/*` once you own the name.

## Direct PDF evaluation quick start

Keep the original PDF as primary evidence. The system output is compared
directly with the PDF; it is not evaluated against a lossy Markdown
replacement:

```python
from visual_evals import (
    EvaluationCase,
    Evaluator,
    LLMJudgeMetric,
    ModelJudge,
    OpenAIResponsesAdapter,
    artifact_from_path,
)

source = artifact_from_path("statement.pdf", authority="evidence")
case = EvaluationCase(
    case_id="statement-001",
    operation="statement_extraction",
    prediction=actual_system_output,
    # Optional. Omit this entirely if the producer is unknown.
    producer={
        "provider": "anthropic",
        "model": "claude-sonnet-your-version",
    },
    references=[source],
    rubric=(
        "Every extracted value must match the visible PDF and remain associated "
        "with the correct label, row, column, and page."
    ),
)
adapter = OpenAIResponsesAdapter(model="gpt-4o-mini")
report = Evaluator(
    [LLMJudgeMetric(ModelJudge(adapter))]
).evaluate(case)
print(report.verdict)
print(adapter.total_usage.model_dump())
```

Producer provenance is optional. If omitted, reports show `unknown` and make no
assumption about judge independence. If supplied, judges are labelled
`same_model`, `same_provider`, or `independent`. Producer-related votes remain
visible, but a panel with no independent judge requires human review.

Set `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` in the process that runs this
code. The library never parses `.env`; `.env` is git-ignored only to reduce the
risk of accidentally committing a locally created secret file.

The runnable two-model example evaluates a JSON prediction directly against
the PDF:

```bash
python examples/evaluate_pdf_directly.py statement.pdf prediction.json
```

Markdown conversion remains available as an optional local derivative via
PyMuPDF4LLM. It is useful for search and debugging but never replaces the raw
PDF:

```bash
pip install -e ".[pdf]"
python examples/pdf_to_markdown.py statement.pdf
```

See [the full usage guide](docs/usage.md) for provider selection, voting,
ground truth, single-vs-panel batch reports, token semantics, and Windows
commands.

## Python API

This example evaluates extraction and a subsequent calculation. The custom
metric independently recomputes the total from the stage input:

```python
from visual_evals import (
    EvaluationCase,
    ExactStructuredMetric,
    FunctionMetric,
    PipelineCase,
    PipelineEvaluator,
    artifact_from_path,
    artifact_from_value,
)

extraction = EvaluationCase(
    case_id="document-001",
    stage_id="extract",
    operation="parse_line_items",
    prediction={
        "line_items": [
            {"quantity": 2, "unit_price": 4.25},
            {"quantity": 1, "unit_price": 3.00},
        ]
    },
    references=[
        artifact_from_path("invoice.png", authority="evidence"),
        artifact_from_value(
            {
                "line_items": [
                    {"quantity": 2, "unit_price": 4.25},
                    {"quantity": 1, "unit_price": 3.00},
                ]
            },
            authority="ground_truth",
            artifact_id="human-transcription",
        ),
    ],
)

calculation = EvaluationCase(
    case_id="document-001",
    stage_id="calculate-total",
    operation="sum_quantity_times_unit_price",
    input_data=extraction.prediction,
    prediction={"total": 11.50},
    depends_on=["extract"],
    rubric="Total equals the sum of quantity multiplied by unit price.",
)

def total_matches(stage):
    expected = sum(
        row["quantity"] * row["unit_price"]
        for row in stage.input_data["line_items"]
    )
    return stage.prediction["total"] == expected

pipeline = PipelineCase(
    pipeline_id="document-processing",
    stages=[extraction, calculation],
)
report = PipelineEvaluator(
    operation_metrics={
        "parse_line_items": [ExactStructuredMetric()],
        "sum_quantity_times_unit_price": [
            FunctionMetric(
                "recalculate_total",
                total_matches,
                score_name="calculation_accuracy",
            )
        ],
    }
).evaluate(pipeline)
```

## Custom action and judge instructions

The downstream action prompt is part of the system under test, not a source of
truth. Define it once in an `OperationSpec`, alongside a separate,
independently reviewed `EvaluationSpec`. Every case that names the operation
reuses that configuration:

```python
from visual_evals import (
    EvaluationCase,
    EvaluationSpec,
    OperationRegistry,
    OperationSpec,
    build_judge_request,
)

vehicle_operation = OperationSpec(
    spec_id="investment-vehicle",
    operation="vehicle_classification",
    version="2026-07-23",
    # This is the real production prompt being tested. It may be imperfect.
    action_instruction="Determine the investment vehicle.",
    # These rules are reviewed independently; they define correctness.
    evaluation_spec=EvaluationSpec(
        spec_id="investment-vehicle-correctness",
        version="3",
        status="approved",
        objective="Choose the vehicle using the approved investment taxonomy.",
        criteria=[
            "Use fund for pooled capital managed as one portfolio.",
            "Use equity only for direct ownership in a company.",
            "Use unknown when the evidence is insufficient.",
        ],
        allowed_outputs=["fund", "equity", "bond", "unknown"],
    ),
)

vehicle_stage = EvaluationCase(
    case_id="statement-001",
    stage_id="classify-vehicle",
    operation="vehicle_classification",
    input_data=extraction.prediction,
    prediction={"vehicle": "fund"},
    depends_on=["extract"],
)

# Evaluator and PipelineEvaluator accept operation_specs=[vehicle_operation].
# Resolve manually only when you want to inspect the prompt before evaluation.
resolved_stage = OperationRegistry([vehicle_operation]).resolve(vehicle_stage)

# Inspect exactly what any judge provider will receive before calling a model.
request = build_judge_request(resolved_stage)
print(request.task["action_instruction"])
print(request.task["evaluation_spec"])
print(request.instructions)
```

`action_instruction` records what the action model was asked to perform; the
library never treats it as truth and does not execute the action.
`EvaluationSpec.status="approved"` records that the correctness rules were
reviewed; `draft` specifications must still be calibrated.

`judge_instruction` can add stage-specific evaluation guidance. Its default
mode is `append`, which keeps the protected evidence and response-contract
rules. Replacing that base prompt requires both
`judge_instruction_mode="replace"` and
`allow_unsafe_judge_override=True`; reports display a warning until the
replacement is calibrated against human-labelled cases.

Reports include the complete instructions and evaluation criteria, their
versions, and separate SHA-256 fingerprints. This makes prompt changes visible
without asking users to paste the same configuration into every test case.

## Model-agnostic judges

The scoring layer depends on the `Judge` protocol, not on an OpenAI class. Every
provider receives the same `JudgeRequest`:

- `instructions`: the shared evaluator prompt;
- `task`: operation, input, prediction claims, rubric, and reference inventory;
- `output_schema`: the JSON Schema for `JudgeOutput`; and
- `references`: text, structured data, image, PDF, or file artifacts.

There is one `ModelJudge`. OpenAI, Claude, and local models differ only in the
adapter passed to it. OpenAI and Anthropic have ready-to-use transport
adapters:

```python
from visual_evals import (
    AnthropicMessagesAdapter,
    LLMJudgeMetric,
    ModelJudge,
    OpenAIResponsesAdapter,
)

openai_metric = LLMJudgeMetric(
    ModelJudge(OpenAIResponsesAdapter(api_key=application_secret))
)
anthropic_metric = LLMJudgeMetric(
    ModelJudge(AnthropicMessagesAdapter(api_key=anthropic_secret))
)
```

Any Ollama, vLLM, Transformers, or other model client can be wrapped by the
same judge:

```python
from visual_evals import LLMJudgeMetric, ModelJudge

def call_model(request):
    # This adapter owns the provider-specific SDK or local inference call.
    # Attach request.references when the model supports images or PDFs.
    result = my_model_adapter.generate_json(
        instructions=request.instructions,
        task=request.task,
        schema=request.output_schema,
        references=request.references,
    )
    return result  # JudgeOutput, a matching dict, or matching JSON text

metric = LLMJudgeMetric(
    ModelJudge(call_model),
    name="claude-sonnet-judge",  # or "ollama-judge", "vllm-judge", etc.
)
```

The evaluation cases, evidence semantics, dependency diagnosis, metric
calculation, and reports are identical across providers. Only `call_model`
changes. A stage can also use deterministic metrics and multiple judges
together. Give each judge metric a distinct `name` when comparing models in one
report.

If `api_key` is omitted, the built-in adapters read `OPENAI_API_KEY` or
`ANTHROPIC_API_KEY` from the caller's existing process environment. The
library never reads an environment file. Both adapters expose `last_usage`,
`usage_records`, `total_usage`, and `reset_usage()`.

### Research-backed multi-judge voting

`JudgePanelMetric` supports both different model providers and repeated samples
at different temperatures:

```python
from visual_evals import (
    JudgePanelMetric,
    JudgeVariant,
    ModelJudge,
    OpenAIResponsesAdapter,
)

panel = JudgePanelMetric(
    [
        JudgeVariant(
            name="openai-low-temp",
            judge=ModelJudge(
                OpenAIResponsesAdapter(
                    model=openai_model_id,
                    temperature=0.0,
                    api_key=openai_secret,
                )
            ),
            provider="openai",
            model=openai_model_id,
            temperature=0.0,
        ),
        JudgeVariant(
            name="openai-higher-temp",
            judge=ModelJudge(
                OpenAIResponsesAdapter(
                    model=openai_model_id,
                    temperature=0.8,
                    api_key=openai_secret,
                )
            ),
            provider="openai",
            model=openai_model_id,
            temperature=0.8,
        ),
        JudgeVariant(
            name="claude",
            judge=ModelJudge(call_claude_at_temperature_0),
            provider="anthropic",
            model=claude_model_id,
            temperature=0.0,
        ),
        JudgeVariant(
            name="local",
            judge=ModelJudge(call_local_model),
            provider="local",
            model=local_model_id,
            temperature=0.2,
        ),
    ],
    vote_rule="majority",             # or "unanimous"
    review_on_any_disagreement=True,  # retain dissent for human review
)
```

Voting is model-balanced:

1. Repeated runs or temperature variants of the same provider/model form one
   self-consistency group.
2. Each distinct provider/model group contributes one final vote.
3. A majority or unanimous rule selects the panel verdict.
4. Ties, uncertainty, and—by default—any minority dissent enter the human-review
   queue.

This prevents five temperature samples from one model outvoting one Claude and
one local-model judge. Reports preserve raw-sample agreement, model-group
agreement, every variant's provider/model/temperature, the winning vote, and
dissenting evidence.

The design follows two related but different research findings:

- Wang et al.'s ICLR self-consistency work found that sampling multiple
  reasoning paths and selecting the consistent answer improved several
  reasoning benchmarks. This supports repeated stochastic samples, but is not
  direct proof that temperature voting makes an LLM judge accurate.
  [Google Research paper](https://research.google/pubs/self-consistency-improves-chain-of-thought-reasoning-in-language-models/)
- Verga et al.'s Panel of LLM evaluators (PoLL) found that panels of diverse
  model families outperformed a single large judge across their tested
  evaluation settings and reduced intra-model bias.
  [PoLL paper](https://arxiv.org/abs/2404.18796)
- Zheng et al. documented position, verbosity, self-enhancement, and reasoning
  limitations in LLM judges, supporting calibration and retained human review.
  [MT-Bench/Chatbot Arena paper](https://arxiv.org/abs/2306.05685)

The OpenAI Responses API documents temperature values from 0 to 2 and advises
changing temperature or `top_p`, not both.
[Official Responses API reference](https://developers.openai.com/api/reference/resources/responses/methods/create)
Other providers and model families may expose different sampling controls.

Voting agreement remains a reliability signal, not accuracy. Validate the
panel against human-labelled cases before using it for production gating.

### Generating provisional ground truth when none exists

When the only material you have is an image or a set of pages—and no human has
transcribed them—you do not have ground truth yet. You have *evidence*. This
section describes how to **generate** a provisional, clearly-labelled
`authority="generated"` reference from that evidence, and how to promote it to
authoritative `ground_truth` only through explicit human approval.

**How generation works, end to end:**

1. **Supply the evidence and the extraction contract.** Give
   `GroundTruthPanel` two or more *independent* model providers, an
   `extraction_instruction`, and the `expected_fields` you want read out.
2. **Each provider reads the source blind.** Every provider sees only the
   source images and the extraction instruction—*never* the system prediction
   and *never* another model's output. Each independently emits candidate
   claims (`path`, `expected_value`, cited `reference_id`, visual `evidence`,
   and a `confidence`).
3. **Claims are reconciled field by field.** For each field the panel takes a
   normalized vote across providers. Fields that meet `minimum_agreement`
   become consensus rows; anything below it—including a field one model omitted
   entirely—becomes a `disagreement` and sets `requires_human_review=True`.
4. **A provisional reference is assembled.** The consensus rows are packaged
   into a single `ReferenceArtifact` with `authority="generated"`. It is safe to
   evaluate against immediately, but any score is *agreement with a provisional
   AI reference*, not verified accuracy.
5. **(Optional) reduce the review queue automatically.** Run
   `GroundTruthAdjudicator` for a blind, targeted second read of only the
   disputed paths, or `resolve_ground_truth_union(...)` to accept
   non-conflicting one-sided claims. Both keep `authority="generated"`.
6. **Promote to ground truth via signed human approval.** A generated reference
   becomes authoritative only through `approve_generated_reference(...)` (or the
   file-based `write_ground_truth_review` → `approved_reference_from_review`
   flow), which records `approved_by` and refuses to approve while any
   disagreement is unresolved.

> **Terminology:** what the panel produces is *generated ground truth*
> (`authority="generated"`), not evidence. Evidence is the raw source you feed
> *in*; the generated reference is a hypothesis about what that evidence says.
> Only human approval turns it into `authority="ground_truth"`.

Each provider sees only the source images and extraction instructions—not the
system prediction—and independently generates candidate reference facts:

```python
from visual_evals import (
    EvaluationCase,
    Evaluator,
    ExactStructuredMetric,
    GroundTruthPanel,
    GroundTruthRouter,
    OpenAIResponsesAdapter,
    approve_generated_reference,
    artifact_from_path,
)

source_images = [
    artifact_from_path(
        "page-001.png",
        authority="evidence",
        artifact_id="page-001",
        metadata={"page": 1},
    ),
    artifact_from_path(
        "page-002.png",
        authority="evidence",
        artifact_id="page-002",
        metadata={"page": 2},
    ),
]

panel = GroundTruthPanel(
    {
        "openai": OpenAIResponsesAdapter(api_key=openai_secret),
        "claude": call_claude,
        "local-model": call_local_model,
    },
    extraction_instruction="Extract currency, amount, and account name.",
    expected_fields=["$.currency", "$.amount", "$.account_name"],
    minimum_agreement=1.0,  # Any disagreement creates a review item.
)

router = GroundTruthRouter(image_panel=panel)

prepared = router.prepare(
    case_id="statement-001",
    references=source_images,
)
generation = prepared.image_report

# Agreed claims can be used provisionally, but are not called ground truth.
case = EvaluationCase(
    case_id="statement-001",
    operation="statement_extraction",
    prediction=actual_extraction,
    references=[*source_images, generation.provisional_reference],
)
report = Evaluator([ExactStructuredMetric()]).evaluate(case)

# If providers disagreed, a human supplies the resolved path/value rows.
approved_reference = approve_generated_reference(
    generation.provisional_reference,
    approved_by="reviewer@example.com",
    resolved_content=human_resolved_rows,
)
```

Routing is based on the complete input set:

```text
all PDF/ODF/ODT documents -> preserve original source evidence
all images                -> preserve sources + optional model consensus
explicit markdown mode    -> add a local supplementary derivative
mixed/unsupported inputs  -> explicit error; prepare separately
```

`GroundTruthRouter` uses `document_mode="preserve"` by default. Optional
Markdown derivatives have `authority="generated"` and never replace the raw
document. They are unscored or low-confidence until evaluated and therefore
remain reviewable.

The generation report contains:

- every provider's independently generated candidate claims;
- fields on which the models agreed;
- each disputed field and candidate value;
- expected fields omitted by every provider;
- cited image IDs, page metadata, and visual evidence; and
- `requires_human_review=True` whenever candidates differ or no model produced
  a usable claim.

The consensus artifact has `authority="generated"`. Evaluation against it is
labelled agreement with a provisional AI-generated reference, not accuracy.
It can be promoted to `authority="ground_truth"` only through
`approve_generated_reference`; unresolved disagreements require human-supplied
resolved content. Even unanimous models can share errors, so human calibration
is still recommended.

For a persistent review workflow, use `write_ground_truth_review` before the
tested pipeline runs. A reviewer edits the generated `claims`, resolves each
`review_items` entry with `decision="set"` or `"omit"`, then sets
`status="approved"` and `approved_by`. Load the result with
`approved_reference_from_review`; it rejects unresolved or unsigned files.
Freeze ground truth for the complete batch before generating predictions.

`GroundTruthAdjudicator` can reduce the manual queue with a second blind
source-reading pass. It shows each provider only the disputed paths, never the
first-pass candidate values. The default requires unanimity; callers can set
`minimum_agreement` for a majority-vote panel. Votes below the configured
threshold remain human-review items.

For unattended experiments where omission by one generator should not remove a
field found by another, call `resolve_ground_truth_union(generation)`. It
accepts non-conflicting present values, leaves conflicting present values
unresolved, and keeps the reference authority as `generated`. Its scores remain
provisional agreement signals rather than verified accuracy.

## CLI

The CLI uses the caller's existing `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`
process environment. With `--judge auto`, OpenAI is preferred when both exist,
then Anthropic; without either key, only deterministic metrics run. A model can
be supplied through `--model` or `VISUAL_EVAL_MODEL`, and OpenAI reasoning
effort through `--reasoning-effort` or
`VISUAL_EVAL_REASONING_EFFORT`.

Evaluate directly against the statement:

```bash
python -m visual_evals.cli \
  --prediction examples/output.json \
  --evidence examples/source.pdf \
  --output reports/example-visual-eval.json \
  --markdown reports/example-visual-eval.md
```

Add a human-maintained base table:

```bash
python -m visual_evals.cli \
  --prediction examples/output.json \
  --evidence examples/source.pdf \
  --ground-truth examples/truth.csv \
  --output reports/example-grounded-eval.json
```

Use an image and human notes instead:

```bash
python -m visual_evals.cli \
  --prediction output.json \
  --evidence statement-page.jpg \
  --ground-truth main-points.md \
  --output reports/image-eval.json
```

Run deterministic checks only:

```bash
python -m visual_evals.cli \
  --prediction output.json \
  --ground-truth truth.csv \
  --judge none \
  --output reports/exact.json
```

## Ground truth can use any format

Ground truth means “authoritative reference”; it does not require a special
table shape. You can supply:

- an ordinary CSV/TSV table with any columns;
- free-form `.md` or `.txt` review notes;
- JSON;
- a PDF, image, or document supported by the selected model adapter; or
- several references in different formats.

The LLM judge reads the reference and compares its meaning with the extraction.
This supports cases where a reviewer provides only the important facts:

```csv
topic,main_point
vehicle,Pooled investment fund
risk,Moderate risk
currency,USD
```

That semantic comparison is probabilistic. It should be reported as an LLM
judgment against authoritative material, not deterministic field accuracy.

For deterministic partial field metrics, optionally use explicit paths:

```csv
path,expected
$.heading,Safety inspection
$.status_text,Needs review
$.items[0].code,A-12
$.items[0].quantity,3
```

`field_path` is accepted instead of `path`; `expected_value` or `value` is
accepted instead of `expected`.

Reference comparison modes are:

- `auto`: explicit path/value tables and JSON can use exact metrics; ordinary
  tables and notes use semantic comparison;
- `semantic`: force LLM comparison even for structured JSON; and
- `exact`: explicitly flatten structured data for deterministic comparison.

```python
reference = artifact_from_path(
    "review_notes.csv",
    authority="ground_truth",
    comparison_mode="semantic",
)
```

Exact metrics skip references that are not exact-compatible rather than
producing misleading missing-field scores.

## Interpreting scores

- `field_accuracy`: correct authoritative fields / all annotated fields.
- `claim_precision`: supported predictions / judge-decided predictions.
- `reference_recall`: matched reference claims / judge-decided reference claims.
- `claim_verification_coverage`: fraction of claims the judge could decide.
- `f1`: harmonic mean of claim precision and reference recall.

With source evidence only, recall is an LLM-estimated inventory—not true
accuracy. With arbitrary notes or tables marked as ground truth, the reference
is authoritative but the LLM's interpretation remains probabilistic. Calibrate
the judge on human labels and report judge/human agreement before scaling to a
large test set.
