Metadata-Version: 2.4
Name: suiteeval
Version: 0.1.1
Summary: Tools for running IR Evaluation Suites
Author-email: Sean MacAvaney <sean.macavaney@glasgow.ac.uk>, Andrew Parry <0andrewparry@gmail.com>
Project-URL: Repository, https://github.com/mrparryparry/suiteeval
Project-URL: Bug Tracker, https://github.com/mrparryparry/suiteeval/issues
Classifier: Programming Language :: Python
Classifier: Operating System :: OS Independent
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: python-terrier
Dynamic: license-file

# 🍬 SuiteEval

Tools for running IR Evaluation Suites with PyTerrier.

## Installation

```
pip install suiteeval
```

## Example Usage

You should define a function which produces `pyterrier` pipelines, if you do not want to lookup an index, the `DatasetContext` object provides tempoary paths and a corpus iterator for indexing.

The function can either return one or more pipelines or yield pipelines in the case that more complex memory management is required. Here is an example where we only keep one neural re-ranker in memory at a time while evaluating the BEIR suite.

You can choose to either return named systems (useful for larger evaluation) or just return the systems!

```python
from suiteeval import BEIR
from pyterrier_pisa import PisaIndex
from pyterrier_dr import ElectraScorer
from pyterrier_t5 import MonoT5ReRanker

def pipelines(context):
    index = PisaIndex(context.path + "/index.pisa")
    index.index(context.get_corpus_iter())
    bm25 = index.bm25()
    yield bm25 >> context.text_loader() >>  MonoT5ReRanker(), "BM25 >> monoT5"
    yield bm25 >> context.text_loader() >> ElectraScorer(), "BM25 >> monoELECTRA"

results = BEIR(pipelines)
```
