Metadata-Version: 2.4
Name: annotations-orchestration
Version: 0.1.0
Summary: Generic LLM evaluation orchestration: resumable parallel runs, manifest-driven experiments, nervaluate metrics
Keywords: llm,evaluation,orchestration,ner,nlp,benchmark
Author: Nicole Dresselhaus
Author-email: Nicole Dresselhaus <nicole.dresselhaus@hu-berlin.de>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Linguistic
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: rich>=13.9.0
Requires-Python: >=3.11, <4.0
Project-URL: Repository, https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration
Project-URL: Issues, https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/issues
Description-Content-Type: text/markdown

# annotations-orchestration

Generic Python library for orchestrating LLM evaluation runs: a **resumable
parallel scheduler** over experiment manifests, plus **nervaluate-based
metrics** (precision/recall/F1 against gold annotations).

Part of the `annotations-*` family: the annotation pipeline lives in
[`annotations4all`](https://scm.cms.hu-berlin.de/annotations4all/annotations4all),
this library provides the evaluation/experiment infrastructure, and concrete
run projects (e.g. `evaluation_2026`) combine both.

## Features

- **Library-agnostic**: the scheduler speaks a subprocess CLI contract
  (RunSpec → flags), not annotations4all types. Reusable for any LLM
  evaluation workflow.
- **Conservative concurrency**: documents within a run are processed
  linearly; `remote` runs share a pool (`execution.remote_parallelism`,
  default 3); `self_hosted` runs get one sequential lane per host.
- **Resumable**: run state is persisted; completed runs are skipped,
  interrupted/failed runs re-execute on restart.
- **Manifest-driven**: experiment YAMLs + optional controller manifests
  (`defaults`/`execution`/`order`).
- **Metrics**: NER evaluation with precision/recall/F1 per entity type,
  using the classic nervaluate evaluation schemes (strict, entity type,
  partial, exact).

## Installation

```bash
python -m pip install annotations-orchestration
```

Requires Python 3.11+.

## Quickstart

```bash
# Validate an experiments directory (dry run, no execution)
annotations-orchestrate --experiments-dir experiments/ --dry-run

# Run the experiments
annotations-orchestrate --experiments-dir experiments/
```

### Python API

```python
from pathlib import Path

from annotations_orchestration import evaluate_classifier, calculate_metrics

# predictions: list of {"ground_truth": {"prodigy": [...]}, "predictions": [...]}
overall, per_tag = evaluate_classifier(
    data_dir=Path("data"),
    eval_dir=Path("results"),
    predictions=predictions,
)
f1 = calculate_metrics(overall, "ent_type")["f1"]
```

## CLI

`annotations-orchestrate` options: `--manifest`, `--experiments-dir`,
`--state`, `--results-dir`, `--dry-run`, `--active-self-hosted`,
`--graceful-shutdown-seconds`.

## Development

```bash
uv sync
uv run ruff check .
uv run ruff format --check .
uv run pytest
```

## Citation

If you use this software in academic work, please cite it as follows:

> Dresselhaus, Nicole. (2026). *annotations-orchestration* (Version 0.1.0) [Software]. Humboldt-Universität zu Berlin. <https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration>

DOI: `10.5281/zenodo.22015764`

Machine-readable metadata is available in [`CITATION.cff`](https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/blob/main/CITATION.cff).

## License

MIT. The bundled `nervaluate_compat/` package is a vendored, unmodified
snapshot of [MantisAI/nervaluate](https://github.com/MantisAI/nervaluate)
commit `0bf272a` (MIT, © 2020 David S. Batista and Matthew A. Upson).
