Metadata-Version: 2.4
Name: annotations-orchestration
Version: 0.3.0
Summary: Generic LLM evaluation orchestration: resumable parallel runs, manifest-driven experiments, span metrics via annotations-evaluate
Keywords: llm,evaluation,orchestration,ner,nlp,benchmark
Author: Nicole Dresselhaus
Author-email: Nicole Dresselhaus <nicole.dresselhaus@hu-berlin.de>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Linguistic
Requires-Dist: annotations-evaluate>=0.1,<1
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: rich>=13.9.0
Requires-Dist: annotations4all>=0.3.1,<1 ; extra == 'annotate'
Requires-Dist: textual>=8.2.8,<9 ; extra == 'tui'
Requires-Dist: ruamel-yaml>=0.18.16,<0.19 ; extra == 'tui'
Requires-Python: >=3.11, <4.0
Project-URL: Repository, https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration
Project-URL: Issues, https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/issues
Provides-Extra: annotate
Provides-Extra: tui
Description-Content-Type: text/markdown

# annotations-orchestration

Generic Python library for orchestrating LLM evaluation runs: a **resumable
parallel scheduler** over experiment manifests, plus **span metrics** for
precision/recall/F1 against gold annotations — computed by
[`annotations-evaluate`](https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate),
the family's spec-driven metrics library.

Part of the `annotations-*` family: the annotation pipeline lives in
[`annotations4all`](https://scm.cms.hu-berlin.de/annotations4all/annotations4all),
this library provides the evaluation/experiment infrastructure, and concrete run
projects combine both.

## Features

- **Decoupled scheduler**: the existing `legacy` runner speaks a subprocess CLI
  contract (RunSpec → flags), not annotations4all types. The optional `standard`
  runner, provider discovery and hints use `annotations4all`; the base
  installation does not require that library.
- **Conservative concurrency**: documents within a run are processed
  linearly; `remote` runs share a pool (`execution.remote_parallelism`,
  default 3); `self_hosted` runs get one sequential lane per host. The exact HU
  endpoint gets a separate serial lane regardless of mode, with at most 10
  requests per minute; request-level pacing is implemented by the client.
- **Resumable**: run state is persisted; completed runs are skipped,
  interrupted/failed runs re-execute on restart.
- **Manifest-driven and validated**: experiment YAMLs + optional controller
  manifests (`defaults`/`execution`/`order`). Run fields (`model`, `host`,
  `port`, `context`, `shots`, `data_dir`, `net`, `api_url`, `api_key_env`,
  `reasoning`, `reasoning_effort`, `requests_per_minute`, `runner`, `annotation`,
  `tagger`, `nested`) are checked for type and allowed values, and
  **unknown keys are rejected** instead of being ignored silently — with file,
  key and the list of allowed keys (plus a typo suggestion); errors are one line
  (`error: …`, exit code 2) instead of a traceback.
- **Metrics**: span evaluation with precision/recall/F1, provided by
  `annotations-evaluate` — the SemEval-2013 compatible flat schemes (strict,
  entity type, partial, exact) used for the run reports, and that library's
  nested standard report (levels, structure, decode rate) for nested
  annotations.

## Installation

```bash
python -m pip install annotations-orchestration
```

Requires Python 3.11+.

The metrics library of the family,
[`annotations-evaluate`](https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate),
is installed as a regular dependency from PyPI.

## Quickstart

```bash
# Plan an experiments directory (no execution; writes runtime state)
annotations-orchestrate --experiments-dir experiments/ --dry-run

# Run the experiments
annotations-orchestrate --experiments-dir experiments/
```

### Python API

```python
from pathlib import Path

from annotations_orchestration import evaluate_classifier

# One record in the harness format: gold spans per annotation tool, predicted spans.
records = [
    {
        "ground_truth": {"prodigy": [{"start": 0, "end": 3, "label": "PER"}]},
        "predictions": [{"start": 0, "end": 3, "label": "PER"}],
    }
]

# Pass records in memory, or omit "predictions" to read a run directory.
overall, per_tag = evaluate_classifier(
    eval_dir=Path("results"),
    predictions=records,
)
f1 = overall["ent_type"]["f1"]  # 1.0
```

## CLI

`annotations-orchestrate` options: `--manifest`, `--experiments-dir`,
`--state`, `--results-dir`, `--dry-run`, `--active-self-hosted`,
`--graceful-shutdown-seconds`.

## Config editor

Version 0.3.0 includes an optional **Textual TUI** for viewing,
creating and editing single-/multi-run experiments and controller manifests.
A structured form, full YAML editor and effective-run preview share the CLI's
manifest semantics. Saves include a change diff, atomic replacement and
protection against unsaved or externally modified documents.

Install it with the `tui` extra:

```bash
python -m pip install 'annotations-orchestration[tui]'
annotations-tui --experiments-dir experiments/
annotations-tui --manifest experiments/controller.yaml --read-only
```

From a source checkout, `uv sync --extra tui` installs the same entry points, so
`uv run --extra tui annotations-tui …` (or `python -m
annotations_orchestration.tui …` with `--file <path>`) behaves identically.

For provider discovery, capabilities, hints and standard annotation, also install
the `annotate` extra; it requires `annotations4all>=0.3.1` (the 0.2 series has
neither `create_chat_client` nor the capability modules). Local installation and
examples:
[annotation workflow](https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/blob/main/docs/annotation-workflow.md).

Interactive use requires a terminal; `--help` works without a TTY or the extra.
The existing CLI does not require the extra.

- Ctrl+N new, Ctrl+O open, Ctrl+S save, Ctrl+Shift+S save as.
- F2 Form / F3 YAML / F4 Preview / F5 Duplicate; Ctrl+R reload, Ctrl+B file list, Ctrl+Q quit, F1 help.
- Read-only viewing and raw editing of invalid YAML for repair.
- Responsive cards group model/prompting, annotation/input and connection, with
  contextual fields. Groups start expanded; manual collapse choices persist.
- Experiment → Defaults → Runs: consistent inheritance, newly added runs stay
  `{}`; local overrides are red, custom inherited values normal, built-ins dim.
  Legacy headers remain readable and migrate only after explicit confirmation.
- Runs can be added, duplicated, removed and reordered; `false`, `0`, empty
  strings and Unset remain distinct.
- Annotation dialog for labels/descriptions, context and language, plus a
  read-only data-directory inspector showing document/gold/span counts.
- Keyless manual configuration stays available. Current live effort metadata
  limits selection to the exact model levels plus Default/Unset; incompatible
  existing values are diagnosed, never silently rewritten.
- A separate green ✓ means positive live evidence for that exact model,
  reasoning or effort value, not guaranteed quality or successful inference.
- Contextual **Hints…** / F1 and **More → Hints** explain reasoning, costs,
  verification and evaluation from the central static library resource.
- F6 opens provider/model selection with a custom ENV name or manual model ID;
  provider-reported model lists have a per-credential session cache and Refresh.
- F8 manages local `.env` keys with masked display; F9 opens UI options.
  Packaged provider definitions are extended/overridden by `cwd/models.yaml`.
- NerdFont icons are on by default; `--no-nerd-fonts` overrides this for the session.
  The theme is chosen in F9 or with `--theme` (session only), also via `Ctrl+P`;
  explicitly saved preferences live in `cwd/settings.yaml`.
- No experiment execution, run control or state/lock files in the editor. Explicit
  network requests only read model metadata, not test completions; credentials
  never enter experiment YAML/settings.
  `api_key_env` remains an environment variable name, not a credential value.

For full controls, inheritance and storage guarantees, see the
[editor guide (English)](https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/blob/main/docs/tui-editor.en.md).

## Declarative annotation

`runner: standard` with `annotation` configures Configurable/Per-Label jobs
without a custom Python frontend; `runner: legacy` remains the default for
existing consumers. The standard runner preflights all input before inference
and requires fresh output paths, even with `--force-predict`; no item resume.

Top-level Prodigy `spans` and `ground_truth.prodigy/tags` are normalized in memory,
never in the source files. Missing gold is annotation-only; explicitly empty
gold remains evaluable. Per-label makes one model call per label/document;
nesting requires the configurable tagger and user-written hierarchy instructions.

Local-library setup, full manifest, preflight, outputs and the extended consumer
CLI contract:
[annotation workflow](https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/blob/main/docs/annotation-workflow.md).
On 2026-10-09 the maintainer confirmed a successful live consumer run using a
TUI-generated configuration, not blanket support for all providers.

## Development

The full suite requires both extras and the development dependencies. No sibling
checkout is needed; the compatible annotation library is installed from PyPI:

```bash
uv sync --locked --all-extras
uv run --locked --all-extras ruff check .
uv run --locked --all-extras ruff format --check .
uv run --locked --all-extras pytest
```

`uv sync --extra tui` alone does not supply library integration-test dependencies.
For development against local library sources, see the annotation workflow;
`uv run` may replace manually installed editable packages with the locked version.
Build with `uv build`; see the linked annotation workflow for compatibility and
packaging boundaries.

## Citation

If you use this software in academic work, please cite it as follows:

> Dresselhaus, Nicole. (2026). *annotations-orchestration* (Version 0.3.0) [Software]. Humboldt-Universität zu Berlin. <https://doi.org/10.5281/zenodo.23495464>

Concept DOI: `10.5281/zenodo.22015763` (always resolves to the newest version)

Machine-readable metadata is available in [`CITATION.cff`](https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/blob/main/CITATION.cff).

## License

MIT.

The metrics are computed by
[`annotations-evaluate`](https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate)
(MIT, same family), a regular PyPI dependency. This package contains no metric
code of its own: it only adapts a run's records and writes the report.
