Metadata-Version: 2.4
Name: slm_eval
Version: 0.0.1
Summary: An evaluation suite for small language models (SLMs) with objective, reproducible benchmarks
Author: TheLimeDev (@Monster-Code)
Maintainer: TheLimeDev (@Monster-Code)
License: MIT
Project-URL: Homepage, https://github.com/TheLimeDev/SLM-Eval
Project-URL: Repository, https://github.com/TheLimeDev/SLM-Eval
Project-URL: Author, https://github.com/TheLimeDev
Project-URL: Hugging-Face, https://huggingface.co/Monster-Code
Project-URL: Documentation, https://github.com/TheLimeDev/SLM-Eval/blob/main/docs/README.md
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.md
License-File: LICENSE_NOTES.md
License-File: NOTICE
Requires-Dist: datasets>=2.16.0
Requires-Dist: numpy
Requires-Dist: evaluate>=0.4.0
Requires-Dist: jinja2
Requires-Dist: pytablewriter
Requires-Dist: rouge-score>=0.0.4
Requires-Dist: sacrebleu>=1.5.0
Requires-Dist: scikit-learn>=0.24.1
Requires-Dist: sqlitedict
Requires-Dist: dill
Requires-Dist: word2number
Requires-Dist: more_itertools
Requires-Dist: typing_extensions
Requires-Dist: tqdm>=4.59.0
Provides-Extra: api
Requires-Dist: requests; extra == "api"
Requires-Dist: aiohttp; extra == "api"
Requires-Dist: tenacity; extra == "api"
Requires-Dist: tqdm; extra == "api"
Requires-Dist: tiktoken; extra == "api"
Provides-Extra: archiver
Requires-Dist: jsonlines; extra == "archiver"
Requires-Dist: zstandard; extra == "archiver"
Provides-Extra: hf
Requires-Dist: transformers>=4.1; extra == "hf"
Requires-Dist: torch>=1.8; extra == "hf"
Requires-Dist: accelerate>=0.26.0; extra == "hf"
Requires-Dist: peft>=0.2.0; extra == "hf"
Provides-Extra: vllm
Requires-Dist: vllm>=0.18; extra == "vllm"
Provides-Extra: gptq
Requires-Dist: auto-gptq[triton]>=0.6.0; extra == "gptq"
Provides-Extra: gptqmodel
Requires-Dist: gptqmodel>=1.0.9; extra == "gptqmodel"
Provides-Extra: ipex
Requires-Dist: optimum-intel; extra == "ipex"
Provides-Extra: ibm-watsonx-ai
Requires-Dist: ibm_watsonx_ai>=1.1.22; extra == "ibm-watsonx-ai"
Requires-Dist: python-dotenv; extra == "ibm-watsonx-ai"
Provides-Extra: onnxruntime-genai
Requires-Dist: onnxruntime-genai; extra == "onnxruntime-genai"
Requires-Dist: transformers; extra == "onnxruntime-genai"
Requires-Dist: numpy; extra == "onnxruntime-genai"
Provides-Extra: onnxruntime
Requires-Dist: onnxruntime>=1.23; extra == "onnxruntime"
Requires-Dist: transformers; extra == "onnxruntime"
Requires-Dist: numpy; extra == "onnxruntime"
Provides-Extra: litellm
Requires-Dist: litellm<1.85,>=1.60; extra == "litellm"
Requires-Dist: aiohttp; extra == "litellm"
Requires-Dist: requests; extra == "litellm"
Requires-Dist: tenacity; extra == "litellm"
Requires-Dist: tqdm; extra == "litellm"
Provides-Extra: optimum
Requires-Dist: optimum[openvino]; extra == "optimum"
Provides-Extra: habana
Requires-Dist: optimum-habana; extra == "habana"
Provides-Extra: sparsify
Requires-Dist: sparsify; extra == "sparsify"
Provides-Extra: dev
Requires-Dist: pytest>=9.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytest-xdist; extra == "dev"
Requires-Dist: requests; extra == "dev"
Requires-Dist: aiohttp; extra == "dev"
Requires-Dist: tenacity; extra == "dev"
Requires-Dist: tqdm; extra == "dev"
Requires-Dist: tiktoken; extra == "dev"
Requires-Dist: sentencepiece; extra == "dev"
Requires-Dist: pillow; extra == "dev"
Provides-Extra: ifeval
Requires-Dist: langdetect; extra == "ifeval"
Requires-Dist: immutabledict; extra == "ifeval"
Requires-Dist: nltk>=3.9.1; extra == "ifeval"
Provides-Extra: math
Requires-Dist: sympy>=1.12; extra == "math"
Requires-Dist: antlr4-python3-runtime==4.11; extra == "math"
Requires-Dist: math_verify[antlr4_11_0]; extra == "math"
Provides-Extra: benchmarks
Requires-Dist: slm_eval[math]; extra == "benchmarks"
Requires-Dist: slm_eval[ifeval]; extra == "benchmarks"
Requires-Dist: slm_eval[api]; extra == "benchmarks"
Provides-Extra: sentencepiece
Requires-Dist: sentencepiece>=0.1.98; extra == "sentencepiece"
Provides-Extra: trackio
Requires-Dist: trackio>=0.23.0; extra == "trackio"
Provides-Extra: wandb
Requires-Dist: wandb>=0.16.3; extra == "wandb"
Requires-Dist: pandas; extra == "wandb"
Requires-Dist: numpy; extra == "wandb"
Provides-Extra: zeno
Requires-Dist: pandas; extra == "zeno"
Requires-Dist: zeno-client; extra == "zeno"
Dynamic: license-file

# slm_eval

An evaluation suite for **small language models (SLMs)**. It ships a small,
hand-picked set of benchmarks, each chosen because it can be scored objectively,
reproducibly, and on a single CPU machine.

Maintained by [TheLimeDev](https://github.com/TheLimeDev)
(Hugging Face: [@Monster-Code](https://huggingface.co/Monster-Code)).

The evaluation engine, model backends, filters and metrics come from the
MIT-licensed LM Evaluation Harness by EleutherAI. All ~14,000 of its task
definitions were removed. Only the benchmarks below remain, so anything you
already know about running the engine still applies — the task registry is
different.

## The supported benchmarks

| Task name | Benchmark | What it measures | Data source | Licence |
|---|---|---|---|---|
| `hellaswag` | HellaSwag | Commonsense continuation of a described activity | `Rowan/hellaswag` | MIT |
| `arc_easy` | ARC Easy | Grade-school science questions | `allenai/ai2_arc` (`ARC-Easy`) | CC-BY-SA-4.0 |
| `arc_challenge` | ARC Challenge | Harder science questions that retrieval alone does not solve | `allenai/ai2_arc` (`ARC-Challenge`) | CC-BY-SA-4.0 |
| `arithmark_3_0` | ArithMark 3.0 | Arithmetic continuation, resistant to memorisation | `AxiomicLabs/Arithmark-3.0` | Apache-2.0 |
| `arithmark_2` | ArithMark 2.0 | The predecessor of ArithMark 3.0 | `AxiomicLabs/ArithMark-2.0` | Apache-2.0 |
| `math_500` | MATH-500 | Competition mathematics needing multi-step reasoning | `HuggingFaceH4/MATH-500` | MIT |
| `gpqa_diamond` | GPQA Diamond | Graduate-level "Google-proof" science questions | `Idavidrein/gpqa`, mirror `fingertap/GPQA-Diamond` | CC-BY-4.0 |
| `bfcl_v4` | BFCL v4 | Function calling and JSON-schema adherence | `bfcl-eval` package data | Apache-2.0 |
| `int_index` | Int Index | A composite score over HellaSwag, ARC and ArithMark-3 | derived, not a dataset | n/a |

`int_index` is a group, not a dataset: it evaluates its component benchmarks and
then combines them with the published formula. See
`slm_eval/tasks/slm/int_index/README.md`.

### Every licence here is usable commercially

No benchmark in this suite is NonCommercial (`NC`) or NoDerivatives (`ND`), and
every benchmark rests on a licence value that names a specific
commercial-use-safe licence. Two benchmarks carry obligations, and both are met
by this repository:

- **ARC** is CC-BY-SA-4.0. Attribution is required, and redistributing a
  modified copy of the ARC data must stay under CC-BY-SA-4.0. This repository
  never redistributes the data: the task downloads it from Hugging Face at run
  time. The attribution is in `NOTICE` and in the task README.
- **GPQA** is CC-BY-4.0. Attribution is required. The public GPQA repository is
  gated, so a Hugging Face token helps; without one the task falls back to an
  open mirror and says so in the log. Attribution is in `NOTICE`.

Full research, with sources checked against the live dataset metadata, is in
`LICENSE_NOTES.md`.

### Benchmarks that were removed, and why

| Benchmark | Reason |
|---|---|
| GSM-Symbolic | CC-BY-NC-ND-4.0: NonCommercial and NoDerivatives |
| AgentIF | CC-BY-NC-4.0: NonCommercial |
| WildBench | Subjective: needs an LLM judge, so results are biased and costly |
| BrowseComp | Too complex: needs a live browsing agent |
| SWE-bench Lite | Too heavy: needs a full Docker sandbox per sample |
| PIQA | The dataset card declares no licence, so commercial use cannot be verified |
| LiveCodeBench | The card declares only the unversioned `cc` tag, and the problems come from third-party contest sites |
| BananaMind Base Bench 1.1 | The dataset card declares no licence, and the repository is gated |

PIQA is a published component of the Int Index. Because it is not shipped here,
the Int Index is computed from HellaSwag, ARC and ArithMark-3 with a denominator
of 2.65, which is what the published method prescribes for a missing component.

LiveCodeBench is the only removal that costs a whole capability: the suite no
longer ships a coding benchmark. `LICENSE_NOTES.md` records the full reasoning.

## Install

```bash
pip install -e ".[hf,api,benchmarks]"
```

`hf` provides the HuggingFace `transformers` backend. `benchmarks` adds the
scoring dependencies the tasks need (`math_verify`, `langdetect`, and friends).

## Run

```bash
# Everything at once. `slm` is a tag group: every task tagged `slm` runs.
python3 -m slm_eval \
  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
  --tasks slm --device cpu --batch_size 8

# One benchmark.
python3 -m slm_eval \
  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
  --tasks arithmark_3_0 --device cpu

# A quick wiring check on a few examples.
python3 -m slm_eval \
  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
  --tasks hellaswag --limit 16 --device cpu
```

List the available tasks:

```bash
python3 -m slm_eval ls tasks
```

## The Int Index

The Int Index is the composite score published on the Open SLM Leaderboard:

```
N(score, chance) = 100 x (score - chance) / (100 - chance)

Int Index = (HellaSwag + ARC + PIQA + 0.65 x ArithMark-3) / 3.65
```

Run the components in one pass, then compute the index from the saved results:

```bash
python3 -m slm_eval --model hf \
  --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
  --tasks int_index --device cpu --output_path logs/tiny-run

# The tracker writes logs/tiny-run/<model name>/results_<timestamp>.json,
# so let the shell expand the path.
python3 -m slm_eval.int_index --results logs/tiny-run/*/results_*.json
```

The scale is small: chance level scores about 0, and the strongest reported
sub-10B models reach roughly 5. It is not a percentage. Each component is
rescaled so that guessing maps to 0 and a perfect score maps to 100.

## Average percentage (`eval_avg`)

`eval_avg` is the plain number: the average of the per-benchmark percentages in
a result file, with no weighting and no chance correction.

```bash
python3 -m slm_eval.eval_avg --results logs/tiny-run/*/results_*.json
```

It picks exactly one metric per benchmark, in this order of preference:
`acc_norm`, `acc`, `exact_match`, `function_name_match`, `ast_match`,
`valid_json`. If a benchmark reports none of those, it uses its alphabetically
first scored metric. `acc_norm` is preferred because raw `acc` carries a
token-length bias.

Diagnostics such as `unparseable` and `output_emitted` measure the harness, not
the model, so they are never chosen. A benchmark with no usable metric is
reported as skipped and leaves the average, instead of counting as zero. The
printed table names the metric used for every benchmark, so the number stays
auditable.

`eval_avg` is a plain mean, so it is comparable only between runs over the same
benchmark set. It is not the Int Index, which is chance-corrected, weighted and
published.

## Scoring tiers

`bfcl_v4` is built in two tiers:

- **Tier 1** always works. It builds the real prompt, generates a real response,
  and computes a deterministic format or rule-based metric.
- **Tier 2** is the official metric. It is implemented, but gated behind an
  explicit prerequisite. If you request it without the prerequisite, the task
  raises a `RuntimeError` that names exactly what is missing. It never silently
  returns zero.

| Task | Tier 1 | Tier 2 |
|---|---|---|
| `bfcl_v4` | `valid_json`, `function_name_match` | `ast_match`, needs `--metadata enable_tier2_metric=True` |

No shipped benchmark executes generated code: BFCL Tier 2 runs the official AST
checker over the predicted call. The engine still enforces
`--confirm_run_unsafe_code` for any task that marks itself unsafe, so a task added
later cannot execute code without that confirmation. Per-task limits and
prerequisites are documented in each task's `README.md`.

## A note on the inherited documentation

`docs/`, `scripts/` and `examples/` document the engine, the CLI and the model
backends, which this suite does not change. Some examples inside them name
benchmarks that were removed here. For the tasks that do exist,
`slm_eval/tasks/slm/CONVENTIONS.md` and the per-task `README.md` files are
authoritative.

## Layout

```
slm_eval/                     evaluation engine
  int_index.py                the Int Index composite calculator
  eval_avg.py                 the plain average-percentage calculator
  tasks/
    CONVENTIONS.md            the task authoring contract
    slm/                      the supported benchmarks live here
      hellaswag/ arc_easy/ arc_challenge/
      arithmark_2/ arithmark_3_0/
      math_500/ gpqa_diamond/ bfcl_v4/
      int_index/              composite group (not a dataset)
tests/                        tests for this suite
LICENSE_NOTES.md              licence research for every benchmark
NOTICE                        attribution for the CCPL benchmarks and the engine
```

## Add or change a task

Read `slm_eval/tasks/slm/CONVENTIONS.md`. Task discovery globs
`slm_eval/tasks/**/*.yaml`, so a new directory with a YAML file is enough.

## Licence

The engine is MIT, from EleutherAI. `LICENSE.md` keeps that notice. See `NOTICE`
for the attribution required by the CC-BY-4.0 and CC-BY-SA-4.0 datasets, and
`LICENSE_NOTES.md` for the per-benchmark research.
