Metadata-Version: 2.4
Name: estravon-backend-benchmarks
Version: 0.2.1
Summary: Apples-to-apples PDF-to-Markdown engine comparison for estravon-backend
Author-email: tiberavonltd <info@estravon.com>
License: AGPL-3.0
Project-URL: Repository, https://github.com/tiberavonltd/estravon-backend-benchmarks
Project-URL: Documentation, https://tiberavonltd.github.io/estravon-backend-benchmarks/
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx
Requires-Dist: fastdownload
Provides-Extra: dev
Requires-Dist: nbdev; extra == "dev"
Dynamic: license-file

# estravon-backend-benchmarks


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

**Prerequisite:** you need `estravon-backend` installed and reachable with the
engines you want to compare configured – MinerU installed locally
(`pip install "estravon-backend[mineru]"`), and/or `MISTRAL_API_KEY` /
`DATALAB_API_KEY` / `REPLICATE_API_TOKEN` set **as environment variables in
the shell/process that calls `compare()`** – e.g. `export MISTRAL_API_KEY=...`
before running your script, or `os.environ["MISTRAL_API_KEY"] = ...` before
calling `compare()`. This package orchestrates `estravon-backend`; it never
extracts anything on its own, and it never reads or passes engine credentials
itself – see “A note on API keys” below for exactly how they reach the
engine. See `estravon-backend`’s `docs/API.md` for the engine-decision table
and how to get each engine’s key.

**Picking up the `.env` file from `estravon-backend`’s own setup:**
`estravon-backend`’s README has you create a `.env` file (e.g.
`MISTRAL_API_KEY=your_key_here`) in *that* repo’s root, and `estravon` loads
it automatically – but only from its own process’s current working
directory at startup. This package spawns `estravon` as a subprocess from
wherever *your* notebook’s kernel happens to be running, which usually isn’t
that directory – so the reliable way to make the same key visible here is
to load that `.env` explicitly, once, before calling `compare()`:

``` python
from pathlib import Path
from dotenv import load_dotenv

load_dotenv(Path("~/path/to/estravon-backend/.env").expanduser())
```

`python-dotenv` ships as a dependency of `estravon-backend` itself, so if
you’ve already installed that (this package’s own prerequisite, above),
it’s already available – no separate install. `.expanduser()` resolves `~`
portably on Linux/macOS/Windows; for a literal `$HOME`-style variable
instead, use `os.path.expandvars(...)`. Once loaded, the variables are in
your notebook process’s own environment, and `LocalEngineProcess` (Mode A)
inherits them when it spawns `estravon` – it doesn’t matter what that
subprocess’s own working directory ends up being.

## Install

``` python
!pip install estravon-backend-benchmarks
```

Also installable via git, e.g. for the latest unreleased commit on `main`, or for contributing (editing a notebook under `nbs/`):

``` sh
pip install git+https://github.com/tiberavonltd/estravon-backend-benchmarks.git
```

To contribute, clone and install in dev mode instead:

``` sh
git clone https://github.com/tiberavonltd/estravon-backend-benchmarks.git
cd estravon-backend-benchmarks
pip install -e ".[dev]"
nbdev-install-hooks   # keeps notebook diffs clean on commit
```

See `CONTRIBUTING.md` for the edit-notebook -\> `nbdev-export` -\> `nbdev-test` loop
(nbdev 3.x only ships hyphenated console scripts – `nbdev_install_hooks` with an
underscore will not be found).

## Quickstart

`compare()` has two modes (see `estravon-backend`’s `docs/API.md` section
“Engine selection” for why there’s no single-URL multi-engine mode – engine
choice is fixed per running instance, on purpose):

- **Mode A (below, the common case):** pass `engines=[...]` and this package
  spawns one pinned `estravon --backend <engine> --port <N>` subprocess per
  engine for you – nothing to configure beyond having `estravon-backend`
  installed with the relevant keys/local models available.
- **Mode B:** pass `engine_urls={"mistral": "http://host:port", ...}` instead,
  if you already have separately-running single-engine instances.

**A note on API keys – two different things share the name “api_key”:**

- **Engine credentials** (`MISTRAL_API_KEY`, `DATALAB_API_KEY`,
  `REPLICATE_API_TOKEN`) are read by `estravon-backend` itself from the
  environment – this package never touches them. In **Mode A**,
  `LocalEngineProcess` spawns `estravon --backend <engine> ...` as a plain
  subprocess that inherits your calling process’s environment, so setting
  them before you call `compare()` is enough; there’s no parameter for them
  here because there doesn’t need to be one.
- **`compare()`’s own `api_key=` parameter is a different thing entirely**:
  it’s only used in **Mode B**, forwarded as the `X-API-Key` header to
  authenticate to a *hosted* `estravon-backend` instance (the same header
  the hosted service’s own billing/auth uses) – it has nothing to do with
  which engine that instance runs. In Mode A it’s hardcoded to `None`:
  locally spawned instances have no such auth layer.

The cell below is marked non-executing in this rendered copy of the notebook
(no backend/engines are available in the environment that builds these
docs) – otherwise it’s exactly what you’d run locally, unmodified: it uses
`get_artusi()` (a real, public-domain 1891 cookbook PDF, downloaded and
cached on first call – see `estravon_bench.io`) instead of a placeholder
path, so there’s nothing to swap in before trying it.

``` python
from estravon_bench.compare import compare
from estravon_bench.io import get_artusi

result = compare(
    pdf_path=str(get_artusi()),   # public-domain 1891 cookbook -- downloaded/cached on first call
    page_range="1-4",
    engines=["mineru", "mistral"],   # whichever engines you have configured
    mode="balanced",
)
print(result.to_markdown_table())
```

This prints a side-by-side table:

    | engine | time (s) | cost (usd) | local | pages | status |
    |---|---|---|---|---|---|
    | mineru | 34.10 | free (local) | yes | 4 | ok |
    | mistral | 2.30 | $0.0080 | no | 4 | ok |

Then look at each engine’s actual output:

``` python
for r in result:
    print(f"--- {r.engine} ---")
    print(r.markdown[:500] if r.ok else f"ERROR: {r.error}")

print(result.diff("mineru", "mistral"))   # optional -- line diff for eyeballing
```

**⚠️ The two-step fetch is handled for you** – `Client.fetch_markdown()`
already does the `md_url` → actual text round trip described in
`estravon-backend`’s `docs/API.md`. If you’re extending `client.py`
yourself, that’s the detail to know about; `compare()`’s own callers never
see a bare URL.

**Honest-cost labelling:** `local=True` engines (MinerU today) show
`cost_usd=0.0` and the table renders “free (local)” – that means *zero
dollars*, not *zero effort or best value*. A free engine that’s ten times
slower is not automatically the right choice.

**Scope:** this is an evaluation aid for comparing engines on your own PDFs
apples-to-apples – not a production layer, and not a leaderboard. It
compares *your* PDF on *your* configured engines; it makes no claim about
which engine is best in general.

## Images

Engines that extract images (Datalab and Mistral, real-verified; any future engine automatically once it returns `image_urls`) have them fetched for you – `result.images` is a `{filename: bytes}` dict per engine, populated by `compare()` the same way `result.markdown` is. `to_markdown_table()`’s `images` column shows the count per engine at a glance.

No automated image comparison here – engines encode/crop/caption images differently even for “the same” figure, so this package limits itself to presence/count and letting you look:

``` python
from IPython.display import Image

row = result.get("mistral")
for filename, data in row.images.items():
    print(filename, f"{len(data)} bytes")
    display(Image(data=data))
```

## Combining results from separate `compare()` calls

Running one engine at a time (different session, different machine, resource constraints) instead of passing `engines=[...]` all at once? Merge the results back into one `ComparisonResultList` afterwards – `+` and `.merge()` both return a new list without touching the originals; `.extend()` mutates the first list in place if you’d rather accumulate into one object as you go:

``` python
from estravon_bench.compare import compare
from estravon_bench.io import get_artusi
from estravon_bench.report import ComparisonResultList

pdf = str(get_artusi())
mineru_result  = compare(pdf, "1-4", engines=["mineru"])
mistral_result = compare(pdf, "1-4", engines=["mistral"])

combined = mineru_result + mistral_result                      # new list, both inputs untouched
# equivalent, and reads better for more than two lists:
combined = ComparisonResultList.merge(mineru_result, mistral_result)

mineru_result.extend(mistral_result)   # or: mutate mineru_result in place instead

print(combined.to_markdown_table())
```

## Saving and loading a comparison run

`ComparisonResultList.save(dir_path)` writes the whole list to disk as plain files – `manifest.json` plus one subdirectory per engine holding `result.md` and an `images/` folder – not a database or a binary format, so any file browser, `git diff`, or image viewer can inspect a saved run directly. `ComparisonResultList.load(dir_path)` reads it back into real `ComparisonResult` objects, markdown/images/metadata all restored exactly:

``` python
from estravon_bench.report import ComparisonResultList

result.save("runs/mineru_vs_mistral")

# ...later, or in a different session:
reloaded = ComparisonResultList.load("runs/mineru_vs_mistral")
print(reloaded.to_markdown_table())
```

## Worked example: diff analysis between Mistral and Datalab on Artusi’s first 4 pages

Good test case — the Artusi title page is unusually hard (ornamental 1891 display type, blackletter-ish capitals, printer’s devices), so it stresses glyph recognition far more than body text would. Here’s what we see in the diff, grouped by kind of difference.

### Structural / heading policy

Mistral promotes almost anything visually large to `#`: “L’ARTE DI MANGIAR BENE”, “MANUALE PRATICO PER LE FAMIGLIE” and even “PROPRIETÀ LETTERARIA” all become H1. Datalab infers a hierarchy instead — `##` for the subtitle, plain paragraph for “MANUALE PRATICO…”. The stats confirm it: 8 headings vs 4. Mistral is mapping *type size* to heading level; Datalab is mapping *document role*. For Zotero/RAG chunking that matters a lot — a document where every other line is H1 gives you useless section boundaries.

### Line-break fidelity

Datalab emits trailing double-spaces to preserve the couplets:

``` markdown
Un pasto buono ed un mezzano  
Mantengon l'uomo sano.
```

Mistral uses bare newlines, so those verse pairs collapse into one line when rendered. Datalab also captures the italics (`*Prima digestio fit in ore.*`) that Mistral drops, and escapes the decorative asterisks (`Igiene \* Economia`) so they don’t accidentally become emphasis.

### OCR accuracy — the interesting part

they fail in different ways:

| originale            | Mistral     | Datalab      |
|----------------------|-------------|--------------|
| Igiene               | “Agiene” X  | “Igiene” V   |
| Dai due regni        | “Dal due” X | “Dai due” V  |
| …sano e lesto        | “lesto” V   | “lieto” X    |
| Pei tipi di S. Landi | “PEI TIPI”  | “PER TIPI” X |

Mistral’s errors are letterform confusions (I -\> A, i -\> l) — classic optical failures on ornamental capitals. Datalab’s errors are language-model normalisations: “Pei” is archaic Italian printing idiom, so it “corrected” it to the more frequent “PER”; likewise “lesto”→“lieto”.

That distinction has real consequences. Optical errors look wrong and get caught. LM-normalisation errors look right and slip through. Worth flagging in a benchmark.

Note also that both engines are internally inconsistent: the same couplet appears twice in these 4 pages, and Mistral writes “Dal” on p1 but “Dai” on p3, while Datalab writes “lieto” then “lesto”. So a same-document self-consistency check would be a cheap and quite powerful quality metric to add.

### Images — the biggest practical difference

Mistral: 3 images, generic alt (`![img-0.jpeg](...)`). Datalab: 2 images, VLM-generated descriptive alt.

Two problems on the Datalab side:

1.  It dropped an image Mistral caught (the device between “IN FIRENZE” and the printer’s name) — 2 vs 3.
2.  It duplicates the description into the body text, sometimes twice:

``` markdown
![Decorative flourish or ornament.](bench_img_002.jpg)

A decorative flourish or ornament consisting of a horizontal line with...

Decorative flourish or ornament.
```

That’s leaking generated English prose into an Italian document’s text stream. The contamination shows up directly in the stats — Datalab’s top keywords include `lines`, `decorative`, `flourish`, `ornament`, none of which appear in Artusi. It also inflates word_count 171→227 and fk_grade 10.3→12.6.

So the stats block is **not currently comparable across engines**, because one of them is measuring its own captions. If those numbers are meant to drive quality scoring, either strip alt-text before computing them, or compute them on text-only and figure-text separately.

### Layout artifacts

Datalab inserts `---` rules where the original has printed rules and page divisions; Mistral ignores them. Neither is wrong — it depends on whether physical layout or logical content only is wanted.

### Summary judgement:

Datalab gives richer, better-structured markdown (real hierarchy, breaks, italics, figure descriptions) at the cost of speed, one missed figure, text-stream pollution, and a tendency to silently modernise archaic wording. Mistral is faster and more literal, but flattens structure and needs post-processing for headings.
