Metadata-Version: 2.3
Name: nlp-components-library
Version: 0.1.0
Summary: Reusable inference, analysis and configuration components for NLP and LLM experiment pipelines
Keywords: nlp,llm,transformers,evaluation,psychometrics,logit-lens
Author: Simon Münker
Author-email: Simon Münker <muenker@uni-trier.de>
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Dist: numpy>=1.26
Requires-Dist: polars>=1.20
Requires-Dist: pydantic>=2.7
Requires-Dist: scipy>=1.13
Requires-Dist: openai>=1.40 ; extra == 'api'
Requires-Dist: accelerate>=1.0 ; extra == 'hf'
Requires-Dist: huggingface-hub>=0.34 ; extra == 'hf'
Requires-Dist: torch>=2.6 ; extra == 'hf'
Requires-Dist: transformers>=4.57 ; extra == 'hf'
Requires-Dist: matplotlib>=3.9 ; extra == 'plot'
Requires-Dist: seaborn>=0.13.2 ; extra == 'plot'
Requires-Dist: vllm>=0.10 ; extra == 'vllm'
Requires-Python: >=3.11
Project-URL: Repository, https://github.com/simon-muenker/nlp_components_library
Project-URL: Issues, https://github.com/simon-muenker/nlp_components_library/issues
Provides-Extra: api
Provides-Extra: hf
Provides-Extra: plot
Provides-Extra: vllm
Description-Content-Type: text/markdown

# nlp-components-library

Reusable inference, analysis and configuration components for NLP and LLM
experiment pipelines, extracted from the multi-turn dice roll, UnBLEnD,
decision-model psychometrics and human-disagreement projects so new
experiments can be assembled from tested parts.

| Package | Components |
| --- | --- |
| `config` | model registry with experiment presets, cultural macro regions, cue lexicons, questionnaire schema |
| `inference` | reproducible chat rendering, verified model loading, exact next-token option probabilities, continuation log-likelihoods, persona questionnaires, logit lens, float32 reward scoring, OpenAI-compatible top-logprobs, vLLM batch chat |
| `analysis` | entropies and divergences, history information, item scoring and aggregation, bootstrap and rank statistics, psychometric scales and reliability, rule-based question annotation |
| `data`, `io`, `plotting` | item and persona loading, result files with provenance, figure helpers |

## Installation

```bash
uv add "nlp-components-library[hf]"      # or: pip install "nlp-components-library[hf]"
```

The core (config, analysis, data, io) needs only numpy, polars, pydantic and
scipy. Extras add the heavier parts:

| Extra | Enables |
| --- | --- |
| `hf` | local models: torch, transformers, accelerate, huggingface-hub |
| `api` | `APIScorer` with an OpenAI-compatible client |
| `vllm` | `inference.vllm_chat` (resolve it in its own environment; vLLM pins torch) |
| `plot` | `plotting` with matplotlib and seaborn |

## Usage

### Models

```python
from nlp_components_library.config import models

for spec in models.resolve(["dice-core", "qwen3_14b"]):  # presets, keys or Hub ids
    if spec.fits(gpu_gb=48):
        print(spec.key, spec.hf_id, spec.bf16_gb)
print(models.table(models.resolve(["human-disagreement"])))
```

The same listing is available as `nlp-components-library models dice-core`
and `nlp-components-library presets`.

### Exact option probabilities

```python
from nlp_components_library.inference import NextTokenScorer, build_chat

scorer = NextTokenScorer.from_pretrained("Qwen/Qwen3-8B", dtype="bfloat16")
chat = build_chat(
    "Roll a six-sided die. Return exactly one number from 1 to 6. Answer: "
)
(result,) = scorer.score([chat], options=("1", "2", "3", "4", "5", "6"))
result.probs  # raw probability of every option (all matching tokens summed)
result.option_mass  # share of the full distribution on the options
result.normalized()  # renormalised over the options
scorer.metadata()  # model revision, template hash, token ids, versions, ...
```

Every option sums the probability of all vocabulary tokens that decode to it
(`"4"`, `" 4"`, `"▁4"`, ...), normalised in float64 over the full vocabulary.
Batches never pad (sequences are bucketed by length), so results do not
depend on the batch size. `APIScorer` offers the same interface for
OpenAI-compatible servers, where only the top-20 log-probabilities are known.

### Logit lens and reward models

```python
from nlp_components_library.config import models
from nlp_components_library.inference import (
    ChatTemplate,
    LogitLens,
    RewardScorer,
    build_chat,
    load_spec,
)

tokenizer, model = load_spec(models.BY_KEY["llama3_8b"])
lens = LogitLens(model, ChatTemplate(tokenizer), options=("A", "B", "C", "D"))
output = lens.score([build_chat(prompt) for prompt in prompts])
output.probs  # [item, layer, option]; output.predictions is the last layer
output.hidden_states

tokenizer, rm = load_spec(models.BY_KEY["skywork_qwen3_8b"])
RewardScorer(rm, ChatTemplate(tokenizer)).score_responses(prompt, ["A", "B", "C", "D"])
```

`load_model`/`load_spec` refuse checkpoints that would leave weights
randomly initialised (e.g. a text-only class on a vision-language checkpoint).

### Persona questionnaires

```python
from nlp_components_library.config import Questionnaire
from nlp_components_library.data import sample_personas
from nlp_components_library.inference import (
    ChatTemplate,
    ContinuationScorer,
    QuestionnaireScorer,
)

segment = Questionnaire.from_json("instruments/big5.json").sole_segment
personas = sample_personas(
    "hf://datasets/nvidia/Nemotron-Personas-USA/data/train-*.parquet", 100
)
scorer = QuestionnaireScorer(
    ContinuationScorer(model, tokenizer),
    template=ChatTemplate(tokenizer),
    history_window=5,
)
responses = scorer.score(personas["persona"].to_list(), segment)
```

Each statement is scored by the log-likelihood of every response option; the
most likely answers of the last `history_window` statements are shown with
the next one, and personas beyond `max_context_tokens` are truncated.

### Scoring, aggregation and statistics

```python
from nlp_components_library.analysis import (
    aggregate,
    bootstrap_ci,
    kruskal,
    score_probabilities,
)
from nlp_components_library.config import macro_region_map
from nlp_components_library.data import load_items

items = load_items(
    "blend.json", ("A", "B", "C", "D"), annotations={"macro_region": macro_region_map()}
)
scores = score_probabilities(
    items, probs, ("A", "B", "C", "D"), lens=lens_table, model="llama3_8b"
)
aggregate(
    scores, ["model", "macro_region"]
)  # accuracy, question accuracy, certainty, ...
bootstrap_ci(scores["correct"].to_numpy())
kruskal(scores["p_gold"], scores["macro_region"])
```

### Psychometrics

```python
from nlp_components_library.analysis import psychometrics

choices, probabilities = psychometrics.response_tables(
    [response.probabilities for response in responses], segment
)
scales = psychometrics.scale_scores(choices, segment)
psychometrics.reliability(choices, segment)  # Cronbach's alpha per dimension
psychometrics.run_r_analysis("omega", choices, segment, "reports/omega.csv")  # needs R
```

### Reproducibility defaults

- **Thinking off.** Templates receive `enable_thinking=False` unless
  `ChatTemplate(..., thinking=True)`.
- **Pinned date.** Templates that stamp today's date get
  `config.DEFAULT_TEMPLATE_DATE` (2026-10-26); pass `date=None` for the real
  date. The dice-roll runs used `datetime.date(2024, 7, 26)`.
- **Provenance.** `io.write_results` stores a CSV with a JSON sidecar
  (creation time, library version and your metadata, e.g. `scorer.metadata()`).

## Development

```bash
uv sync --extra hf --extra api --extra plot
uv run pytest                 # tiny random models are built offline
uv run pytest -m "not slow"   # without the model tests
uv run ruff check && uv run ruff format --check && uv run mypy
```

The lock file resolves `vllm` in its own fork. Upgrading everything at once
(`uv lock --upgrade`) can pull the `hf` fork down to vLLM's torch pin; upgrade
packages selectively (`uv lock --upgrade-package transformers`) instead.

## Releasing

Publishing uses PyPI trusted publishing from `.github/workflows/publish.yml`.

1. Once: on PyPI (and TestPyPI) add a trusted publisher for this repository
   with workflow `publish.yml` and environment `pypi` (`testpypi`), and create
   both environments in the GitHub repository settings.
2. Optional dry run: start the *Publish* workflow manually; it runs CI, builds
   and uploads to TestPyPI.
3. Release:

   ```bash
   uv version --bump minor            # or patch / major
   git commit -am "Release $(uv version --short)"
   git tag "v$(uv version --short)"
   git push origin main --tags
   ```

   The tag runs CI, checks that it matches the package version, builds with
   `uv build` and publishes with `uv publish`.
