slm_eval
========

An evaluation suite for small language models, maintained by TheLimeDev
(GitHub: https://github.com/TheLimeDev, Hugging Face: https://huggingface.co/Monster-Code)

This product includes software developed by EleutherAI
(https://www.eleuther.ai/): the evaluation engine is derived from the
MIT-licensed LM Evaluation Harness. The MIT copyright notice is retained in
LICENSE.md. The task definitions for that engine were removed and replaced by
the benchmark tasks shipped here.


Benchmark data attribution
--------------------------

slm_eval ships task definitions and scoring code only. No benchmark dataset is
redistributed with this product: each task downloads its dataset at run time
from the publisher's own distribution point.

The following benchmark datasets require attribution, and are used under the
licence stated:

* ARC (AI2 Reasoning Challenge) -- tasks `arc_easy`, `arc_challenge`
  Dataset: allenai/ai2_arc
  Copyright Allen Institute for AI. Licensed under CC-BY-SA-4.0.
  License text: https://creativecommons.org/licenses/by-sa/4.0/
  Citation: Clark et al., "Think you have Solved Question Answering? Try ARC,
  the AI2 Reasoning Challenge", arXiv:1803.05457.

* GPQA Diamond -- task `gpqa_diamond`
  Dataset: Idavidrein/gpqa, with open mirror fingertap/GPQA-Diamond
  Copyright the GPQA authors. Licensed under CC-BY-4.0.
  License text: https://creativecommons.org/licenses/by/4.0/
  Citation: Rein et al., "GPQA: A Graduate-Level Google-Proof Q&A Benchmark",
  arXiv:2311.12022.

The remaining datasets are used under permissive licences and are credited here
for completeness:

* HellaSwag -- task `hellaswag` (MIT). Dataset: Rowan/hellaswag.
  Citation: Zellers et al., "HellaSwag: Can a Machine Really Finish Your
  Sentence?", arXiv:1905.07830.
* ArithMark 3.0 / ArithMark 2.0 -- tasks `arithmark_3_0`, `arithmark_2`
  (Apache-2.0). Datasets: AxiomicLabs/Arithmark-3.0,
  AxiomicLabs/ArithMark-2.0. Copyright Axiomic Labs.
* MATH-500 -- task `math_500` (MIT). Dataset: HuggingFaceH4/MATH-500.
  A 500-problem subset of the MATH dataset by Hendrycks et al.,
  "Measuring Mathematical Problem Solving With the MATH Dataset",
  arXiv:2103.03874.
* BFCL v4 -- task `bfcl_v4` (Apache-2.0). Data shipped by the `bfcl-eval`
  package, from the ShishirPatil/gorilla repository. Copyright the Gorilla
  project authors.
* Int Index -- task group `int_index`. Not a dataset. The composite formula is
  published by Axiomic Labs on the Open SLM Leaderboard:
  https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard
