Metadata-Version: 2.4
Name: Munibench
Version: 0.1.0
Summary: A callback-based benchmark runner for evaluating memory systems
Author: Loreno Amori
Project-URL: Dataset, https://huggingface.co/datasets/lorenoamori/longmemeval
Keywords: benchmark,memory,llm,longmemeval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: huggingface-hub>=0.24
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"

# Munibench

Munibench is a small callback-based benchmark runner for memory systems. It
keeps the benchmark implementation independent from the system being tested:
you provide callbacks for resetting, feeding, and asking your system.

## Install from PyPI

```powershell
py -m pip install Munibench
```

## Install locally

From this directory:

```powershell
py -m pip install --editable .
```

From the thesis repository root:

```powershell
.venv\Scripts\python.exe -m pip install --editable .\package
```

## Download a dataset

```python
from munibench import download_dataset, list_datasets

print(list_datasets())
path = download_dataset("longmemeval-s")
print(path)
```

The first download of `longmemeval-s` is approximately 277 MB. Hugging Face
caches it locally, so subsequent calls reuse the downloaded file.

## Run the benchmark

```python
from munibench import benchmark

bench = benchmark(
    benchmarks="all",
    dataset="longmemeval-s",
    results_path="results/benchmark.jsonl",
)

bench.run(
    system=memory_system,
    reset=reset,
    feed=feed,
    ask=ask,
    new_instance=new_instance,
    feed_format="text",
    limit=5,
    verbose=True,
)
```

You can bypass downloading and use a local file instead:

```python
bench = benchmark(data_path="path/to/longmemeval_s.json")
```

Do not pass both `dataset` and `data_path`.

## Public API

- `Benchmark` and its backwards-compatible alias `benchmark`
- `LongMemEval`
- `download_dataset()`
- `list_datasets()`
- `dataset_info()`
- `normalize_text()` and `is_correct()`

The PyPI distribution is named `Munibench`; Python imports are lowercase:

```python
import munibench
```
