Metadata-Version: 2.4
Name: infolang-bench
Version: 0.1.0
Summary: Memory-benchmark harness (LoCoMo, LongMemEval, BEAM 1M) for InfoLang, built on the infolang Python SDK.
Project-URL: Homepage, https://infolang.ai
Project-URL: Source, https://github.com/InfoLang-Inc/infolang-bench
Project-URL: Issues, https://github.com/InfoLang-Inc/infolang-bench/issues
Author-email: InfoLang <engineering@infolang.ai>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: beam,benchmark,infolang,locomo,longmemeval,memory,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: infolang>=0.2.0
Provides-Extra: backends
Requires-Dist: mem0ai==0.1.118; extra == 'backends'
Requires-Dist: zep-cloud==2.11.0; extra == 'backends'
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: tracking
Requires-Dist: wandb>=0.19; extra == 'tracking'
Description-Content-Type: text/markdown

# infolang-bench

[![repro-nightly](https://github.com/InfoLang-Inc/infolang-bench/actions/workflows/repro-nightly.yml/badge.svg)](https://github.com/InfoLang-Inc/infolang-bench/actions/workflows/repro-nightly.yml)

Memory-benchmark harness for [InfoLang](https://infolang.ai). Runs **LoCoMo**,
**LongMemEval**, and **BEAM 1M** through the official
[`infolang`](https://pypi.org/project/infolang/) SDK — same ingest, same
questions, same optional LLM judge (see [`docs/JUDGING.md`](docs/JUDGING.md)).
Competitor lanes (Mem0 / Zep) share that harness
([`docs/FAIRNESS.md`](docs/FAIRNESS.md)).
Public claim formulas: [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md).
Claims registry: [`docs/CLAIMS.md`](docs/CLAIMS.md).
Publishing / wandb: [`docs/PUBLISHING.md`](docs/PUBLISHING.md).
Corrections: [`docs/CORRECTIONS.md`](docs/CORRECTIONS.md).

## Reproduce a published number (one command)

Published figures come from **frozen recipes** (`il-bench repro`). Knobs
(`top_k`, `candidate_pool`, `max_per_session`, judge models, strict mode) are
byte-identical to `summary.meta.provenance.repro`.

### Hosted (public path)

The runtime Docker image is **not public** (it embeds core-IP packages and a
private model checkpoint). Anyone can still reproduce **recall quality**,
**judged accuracy**, and **hosted-path latency** against `api.infolang.ai`:

```bash
# Free API key (no credit card) from https://infolang.ai
export INFOLANG_API_KEY=il_live_...

# Option A — Docker (only Docker + git clone required)
docker compose run --rm bench repro locomo

# Option B — local pip (from source; not on PyPI yet)
git clone https://github.com/InfoLang-Inc/infolang-bench.git && cd infolang-bench
pip install -e .
il-bench repro locomo
```

Optional judged phase: also set `OPENAI_API_KEY` (or `ANTHROPIC_API_KEY`).
Without those keys, repro still writes retrieval `*.summary.json` / `*.per_q.json`
and prints why judging was skipped.

Hosted mode **cannot** reproduce self-hosted / on-box runtime latency.

### Local runtime (evaluation / private image holders)

```bash
export IL_RUNTIME_IMAGE=your-registry/infolang-runtime:tag
export INFOLANG_BASE_URL=http://il-runtime:8766
export INFOLANG_DEV_KEY=devsecret:default
docker compose --profile local up -d il-runtime
docker compose --profile local run --rm \
  -e INFOLANG_BASE_URL=http://il-runtime:8766 \
  -e INFOLANG_DEV_KEY=devsecret:default \
  bench repro locomo --local
```

### Smoke

```bash
il-bench repro locomo --limit 1
# or
docker compose run --rm bench repro locomo --limit 1
```

Artifacts land in `results/repro_<benchmark>_<ts>.infolang.{summary,per_q,judged}.json`.

## Expected cost / wall-time / ranges

| Benchmark | Recipe | Wall (full, hosted, approx.) | Judge tokens (if keys set) | Notes |
|-----------|--------|------------------------------|----------------------------|-------|
| LoCoMo | `top_k=50`, `pool=500`, `max_per_session=6`, judge `gpt-4o-mini` | tens of minutes | ~answer+judge calls × questions | Report both `overall_excl_adversarial` and `overall_incl_adversarial` |
| LongMemEval (s) | same pool / session caps, `top_k=50` | tens of minutes–hours | same | Abstention types (`*_abs`) use abstain-is-correct |
| BEAM 1M | `top_k=100`, `pool=500` | hours (all chats) | same | Prefer `--limit` / `--chats` for smoke |

Exact expected metric bands for a given recipe version are those printed in the
matching published `*.summary.json` / `*.judged.json` under `results/` (never
hand-entered). `provenance.repro.recipe_sha256` must match.

## Install

`infolang-bench` isn't on PyPI yet — install from source:

```bash
git clone https://github.com/InfoLang-Inc/infolang-bench.git
cd infolang-bench
pip install -e .                              # pulls infolang>=0.2.0 (on PyPI)
pip install -e ".[tracking]"                  # optional W&B
pip install -e ".[backends]" -c constraints-backends.txt  # Mem0/Zep
```

For development:

```bash
pip install -e ".[dev]"
```

## Connect to a runtime

| Scenario | Configuration |
|----------|---------------|
| Hosted repro (recommended) | `INFOLANG_API_KEY=il_live_...` |
| Local self-hosted | `INFOLANG_DEV_KEY` + runtime on `:8766` (or `INFOLANG_BASE_URL`) |
| Custom host | `INFOLANG_BASE_URL=...` |

```bash
il-bench health
```

## Download datasets

Data is downloaded on demand (`il-bench repro` auto-downloads if missing).
Checksums are verified before every run.

```bash
il-bench download locomo
il-bench download longmemeval --variant s
il-bench download beam --data-dir data/beam-1m
```

## Advanced: `il-bench run`

For ad-hoc sweeps (non-published knobs), backends, or WandB streaming:

```bash
il-bench run locomo --limit 1 --backend infolang
il-bench run locomo --backend mem0 --judge --answer-model … --judge-model …
il-bench judge results/….per_q.json --answer-model … --judge-model …
il-bench compare results/*.judged.json --out compare.md
```

See [`docs/JUDGING.md`](docs/JUDGING.md), [`docs/FAIRNESS.md`](docs/FAIRNESS.md).

### `run` options

| Flag | Meaning |
|------|---------|
| `--backend` | `infolang` \| `mem0` \| `zep` |
| `--limit N` | Limit units / sample size |
| `--top-k K` | Recall depth (default: max K) |
| `--pool N` | Hybrid candidate pool (default 500) |
| `--judge` | Chain the answerer + judge |
| `--no-strict` | Allow hybrid→plain fallback (published runs stay strict) |
| `--wandb` | Stream to W&B (`tracking` extra); `WANDB_MODE=offline` for CI |

## Development

```bash
pip install -e ".[dev]"
ruff check .
mypy
pytest
```

Nightly compose smoke: `.github/workflows/repro-nightly.yml`.


## Where we lose

Public honesty: categories where a competitor or alternative beats InfoLang.
Same table style as the wins. Extend this section from same-harness
artifacts when full judged runs land ([CLAIMS.md](docs/CLAIMS.md)).

| Category | Who wins | Evidence | Why |
|----------|----------|----------|-----|
| LoCoMo **multi-hop** | Mem0 (published LLM-judge; same-harness pending) | Legacy internal signal: retrieval finds evidence turns, but we lack a chain-synthesis step | Closing the gap needs an LLM chain-synthesis pass on top of recall — not a bigger `top_k` |
| STS-B @ **192-byte budget** | PCA-96 | [METHODOLOGY.md#size](docs/METHODOLOGY.md#size): **0.815 PCA vs 0.708 IL** cosine | IL record is a memory primitive (reconstruct + structured query), not an STS-B embedding competitor — publish the trade, don't hide it |

Do **not** cite quarantined workspace `benchmarks/` proxy tables in public copy.

## License

Apache-2.0.
