Metadata-Version: 2.4
Name: mosaic-llm-bench
Version: 0.1.0
Summary: MOSAIC: evaluating the Moral, Social and Individual dimensions of LLMs
Author: Erica Coppolillo, Emilio Ferrara
License-Expression: MIT
Project-URL: Homepage, https://github.com/EricaCoppolillo/MOSAIC
Project-URL: Repository, https://github.com/EricaCoppolillo/MOSAIC
Project-URL: Issues, https://github.com/EricaCoppolillo/MOSAIC/issues
Keywords: llm,benchmark,evaluation,moral-psychology,alignment
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: hf
Requires-Dist: huggingface_hub>=0.30; extra == "hf"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: local
Requires-Dist: transformers>=4.40; extra == "local"
Requires-Dist: torch; extra == "local"
Provides-Extra: all
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: huggingface_hub>=0.30; extra == "all"
Requires-Dist: anthropic>=0.40; extra == "all"
Dynamic: license-file

# mosaic-bench

Run the MOSAIC benchmark — nine validated psychometric questionnaires and four
ethical dilemmas — against any LLM. Test data is pulled directly from the
[MOSAIC repository](https://github.com/EricaCoppolillo/MOSAIC) at a pinned
commit and cached locally, so there is nothing to download by hand.

## Install

```bash
pip install "mosaic-bench[openai]"     # OpenAI, Groq, DeepSeek, vLLM, Ollama, ...
pip install "mosaic-bench[hf]"         # HF Inference Providers
pip install "mosaic-bench[local]"      # offline, via transformers
```

## Run

```bash
export OPENAI_API_KEY=sk-...
mosaic --model gpt-4o --test mfq2 sdo --runs 10
```

Or pass the key inline:

```bash
mosaic --model gpt-4o --api-key sk-... --test mfq2
```

Any OpenAI-compatible endpoint works without code changes:

```bash
mosaic --backend openai --model qwen/qwen3-32b \
       --base-url https://api.groq.com/openai/v1 --api-key gsk_...

mosaic --backend openai --model my-finetune \
       --base-url http://localhost:8000/v1 --api-key dummy   # local vLLM
```

Anything on the Hub:

```bash
export HF_TOKEN=hf_...
mosaic --backend hf --model meta-llama/Llama-3.3-70B-Instruct --test mfq2
```

Fully offline, no keys:

```bash
mosaic --backend local --model Qwen/Qwen3-8B --test mfq2 --runs 3
```

## Output

```
mfq2 — gpt-4o
  authority                      21.00 ± 1.00
  care                           24.33 ± 2.08
  equality                       19.67 ± 5.13
  ! 2.1% of answers imputed — {'ok': 340, 'unparsable': 8}
```

Two files per test under `results/<model>/`:

- `<test>_<variant>.jsonl` — one record per invocation: raw response, run
  index, temperature, timestamp, provider status.
- `<test>_<variant>_scored.jsonl` — the same records plus `parsed_score`,
  `parse_status` and `imputed`.

Runs are resumable: re-running the same command skips (item, run) pairs already
on disk, and raising `--runs` only fills in the missing repetitions.

## Python API

```python
from mosaic_bench import build_backend, run_test

backend = build_backend("gpt-4o", api_key="sk-...")
summary = run_test(backend, "mfq2", runs=10)
print(summary["scores"], summary["quality"])
```

## Notes on scoring

- Answers are parsed leniently but validated against each test's declared
  range; out-of-range values are rejected rather than accepted.
- Unparsable answers are imputed with the test's neutral score, and the
  imputed fraction is reported alongside every result.
- Provider content filters are recorded as `content_filtered` and excluded
  from aggregation — never converted into a score.
- Reverse-keyed items are flipped using `scale_min + scale_max`.

## Publishing

Until this is on PyPI, it installs straight from the repository:

```bash
pip install "git+https://github.com/EricaCoppolillo/MOSAIC#egg=mosaic-bench[openai]"
```

To publish to PyPI, from the package root:

```bash
python -m build
python -m twine upload dist/*
```

Prefer PyPI Trusted Publishing over an API token: it authenticates the GitHub
Actions workflow via OIDC, so no long-lived secret lives in the repo. Configure
the publisher once at pypi.org/manage/account/publishing/, then tag a release.
