Metadata-Version: 2.4
Name: june-bench
Version: 0.2.0
Summary: Reproducible benchmark suite for memory/QA systems — June + pluggable competitors.
Author-email: Junemind <access@januraine.ai>
License: MIT
Project-URL: Homepage, https://june.januraine.ai
Project-URL: Repository, https://github.com/Junemind/june-bench
Keywords: benchmark,rag,memory,qa,retrieval,evaluation,llm,june
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Provides-Extra: june-api
Requires-Dist: httpx>=0.24; extra == "june-api"
Provides-Extra: agents
Requires-Dist: june-mcp>=0.3; extra == "agents"
Provides-Extra: tool-surface
Requires-Dist: june-mcp>=0.6.0; extra == "tool-surface"
Provides-Extra: june-local
Requires-Dist: junemind>=0.1.0; extra == "june-local"
Provides-Extra: mcp
Requires-Dist: june-mcp>=0.1.0; extra == "mcp"
Provides-Extra: cognee
Requires-Dist: cognee[evals]>=1.5.4; extra == "cognee"
Requires-Dist: fastembed>=0.2; extra == "cognee"
Provides-Extra: stream
Requires-Dist: ijson>=3.2; extra == "stream"
Provides-Extra: all
Requires-Dist: httpx>=0.24; extra == "all"
Requires-Dist: ijson>=3.2; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ijson>=3.2; extra == "dev"
Dynamic: license-file

# june-bench

A pip-installable, **reproducible** benchmark suite for memory / QA systems — **June + pluggable
competitors** — over LoCoMo, LongMemEval, HotpotQA/2Wiki/MuSiQue, and FinanceBench, with the same
data and the same scorer.

```bash
pip install june-bench
june-bench list
june-bench run --system echo --dataset smoke --split smoke    # offline, no key, no download
```

## Reproduce the June vs Cognee head-to-head

One command runs **both** systems over the same HotpotQA open-pool, the same answer model, and the same
judge, and prints a side-by-side with the metered API cost:

```bash
pip install "june-bench[cognee,june-api]"     # bundles cognee + fastembed
june-bench reproduce-h2h --key <YOUR_ACCESS_KEY> --questions 100
```

* **Access key** — June's endpoint is hardware-limited (not yet funded), so runs are key-gated. Request
  one at **access@januraine.ai**; the reply includes your key and this exact command.
* **Same-embedder by default** — Cognee automatically embeds with `bge-large-en-v1.5`, the commodity open
  model June's dense lane uses, so it's a **same-embedder** matched run out of the box (nothing to export).
  This embedder is a disclosed benchmark parameter, not June's moat; pass `--embedder <id>` to swap it.
* **You bring an OpenRouter key** (prompted) — it pays for *both* systems' gpt-4o answers (~$21 for the
  chain-of-thought tier at n=100); the host never holds or pays for it.
* Cognee runs **locally** (needs RAM + a one-time ~1.3 GB fastembed download); June answers over its
  endpoint. The command batches the pool upload, blocks the $90 Opus-on-Cognee path, and meters real cost.

`june-bench reproduce` runs the June-only HotpotQA number the same way; `reproduce-retrieval` scores
June's recall@k/nDCG/MRR. All three are plain-language and need no `JUNE_BENCH_*` env vars.

A benchmark is `run(system, dataset) → records → score`. Two typed ports are the only extension
points:

* **`System`** — the thing benchmarked. `JuneApiSystem` (default; a thin HTTP client to June's
  `/v1/answer`, so **no June source is shipped**), `JuneLocalSystem` (`[june-local]` extra; a
  source-protected compiled wheel), `CogneeSystem` (`[cognee]` extra), or any future system as one
  adapter.
* **`Dataset`** — what it runs on. The four benchmarks behind a registry.

The scorer is the canonical SQuAD/HotpotQA EM/F1 + selective-accuracy/coverage/cost — Cognee-comparable.
Tiny **smoke fixtures ship in the wheel** (offline wiring proof); full splits are **fetched, sha-verified,
from a pinned release**. No score is ever baked into the package — every result row records
dataset + scorer + system + model + cost, so a published number is reproducible by a stranger, with the
exact command above.

## Tokens saved by a real coding agent — `june-bench tokens-saved`

June's in-product receipts can only measure June's own chat. For **Claude Code, Codex and any
MCP client**, June never sees the bill — so the only honest per-agent number comes from running the
agent itself, twice, and reading its own session accounting:

```bash
pip install "june-bench[agents]"              # june-mcp for the with-June arm
june-bench tokens-saved --harness claude-code --june-url http://localhost:8000 --key <KEY> --runs 3
june-bench tokens-saved --harness both --noise 300 --mcp-readonly --out results/tokens-saved   # Claude Code + Codex, bigger repo
```

What it does: materialises a fixed corpus (a fictional company's internal docs, 60 files) into a
scratch workspace, ingests the same corpus into a **fresh June canvas**, and runs every task in the
set on the real agent **with `june-mcp` attached** and **without it** — same workspace, same prompt,
`N` runs each. Usage is read from the harness's own report (Claude Code's JSON result, Codex's
`turn.completed` event): **prompt tokens fed** = input + cache read + cache creation, every turn.
Two single-call baselines run beside them when you give a model key: `naive-read v1 k=3` (the same
grep-shaped procedure June's in-product A/B uses) and `cited-read` (the task's gold documents in
full — the baseline most flattering to the agent). Money is printed only with its provenance
(OpenRouter credits delta, or the harness's own list-price figure); the report carries a
**control** line (the fixed overhead of each arm, including June's tool manifest), a **correct**
line per arm, June's own receipts as a cross-check, and every unmeasured run by name. Nothing is
estimated or extrapolated; a fake harness (`--harness fake`) exists for wiring only and stamps its
report NOT A MEASUREMENT.

Task classes: `doc-lookup`, `multi-doc`, `multi-turn` (the second turn is where an agent re-reads),
`past-session` (the record is a transcript file), and `unfiled-session` (a decision June captured
that was **never written to the repository** — without June the agent has nothing to read).

What the first live runs showed (Claude Code 2.1, September 2026, smoke-sized): on a small, grep-able
folder of markdown files the agent's own `Grep`+`Read` is already a precise retriever, and June's
tool manifest (~6k tokens read-only, ~10k with writes, per turn) outweighs what June serves — the
with-June arm fed the model *more*. On `unfiled-session` tasks the without-June agent burned ~48k
tokens searching and answered 0/2; with June it answered 2/2 for fewer tokens. Read the correct line
with the ratio, always. Both are in the report; neither is hidden.

## Does an agent pick the right tool? — `tool-surface`

```bash
pip install "june-bench[tool-surface]"
python -m june_bench.tool_surface run --arm oracle --profile compact --out results/ts-oracle      # no model: proves the suite
ENABLE_TOOL_SEARCH=false python -m june_bench.tool_surface run --arm claude-code --model claude-sonnet-5 \
    --profile compact --runs 3 --out results/ts-claude-code
python -m june_bench.tool_surface report --md results/ts-*/results.jsonl
```

124 scenarios against June's MCP connector, judged from the tool-call log and the engine's state
afterwards. The run checks the engine first and stops rather than scoring an outage; `--resume`
picks up where a run stopped. Point it at a June engine with `--engine` / `--key` (or
`JUNE_BASE_URL` / `JUNE_API_KEY`).

## Honest-measurement notes

Both systems get the same documents, questions, gold answers, scorer, and embedder. Databases are
reset before runs so results are never contaminated by prior state. Costs are measured from provider
billing deltas, not estimated.

## Protocol notes (read before comparing numbers)

june-bench runs a **matched-pair protocol**: identical evidence pool, answer model, and judge for
every system, scored with EM/F1 plus a fixed LLM judge. Two deliberate differences from the official
benchmark settings: the default mode pools QA over the corpus (the official LoCoMo/LongMemEval
settings are per-conversation), and default runs use 100-question slices. This makes results
**directly comparable between systems run here** — and NOT comparable to published leaderboard
numbers, which use different protocols. Compare systems, not leaderboards.

Note on difficulty: pooling is the *harder* direction. The official settings give each question its
own haystack (e.g. ~40 sessions in LongMemEval_S); the pool is the union of every conversation in the
run, so each question faces strictly more distractors — including cross-conversation confusables the
official design never tests. Both systems face the same pool.

## Dataset licenses

Full splits are fetched from their official sources, sha-verified (see `june-bench fetch`). The small
bundled reproduce/smoke fixtures are subsets of HotpotQA (CC BY-SA 4.0), LongMemEval (MIT),
LoCoMo (CC BY-NC 4.0), and FinanceBench (CC BY-NC 4.0) — see `DATA_LICENSES.md` for attribution
and modification notes.

## Links

- Junê: https://june.januraine.ai
- Published results: https://june.januraine.ai (benchmarks section)
- Releases (desktop apps): https://github.com/Junemind/June_releases

## Serving platform matters (measured, July 2026)

The answer model's **serving platform is part of the experiment**, and June is the system honest
enough to show it. June answers only what its evidence supports and refuses the rest — it does not
gamble. An aggregator (OpenRouter) routes each request to an unpinned, changing provider mix, and
when that serving drifts conservative, June's honest refusals rise; guess-style systems have no
refusal channel, so the same drift hides inside silently-changed guesses instead.

Measured on an identical engine and identical questions (2026-07-27):

| serving | gpt-4o | claude-opus-4-8 |
|---|---|---|
| via OpenRouter (aggregator) | 45–49 / 100 right-per-asked | 55 / 100 |
| served DIRECT (vendor API)  | **62 / 100** | **71–72 / 100** |

Use OpenRouter for one-key convenience and real-time cost metering. For accuracy-representative
or publishable numbers, choose a **direct platform** in the menu — every result stamps the
platform it ran on (`As-run:`), so numbers from different serving paths are never conflated.
