<!-- This disables the requirement that the first line is a top-level heading -->
<!-- markdownlint-configure-file { "MD041": false } -->

<div align='center'>
<img
    src="https://raw.githubusercontent.com/EuroEval/EuroEval/main/gfx/euroeval.png"
    height="500"
    width="372"
>
</div>

### The robust European language model benchmark

(formerly known as ScandEval)

______________________________________________________________________
[![Documentation](https://img.shields.io/badge/docs-passing-green)](https://euroeval.com)
[![PyPI Status](https://badge.fury.io/py/euroeval.svg)](https://pypi.org/project/euroeval/)
[![First paper](https://img.shields.io/badge/arXiv-2304.00906-b31b1b.svg)](https://arxiv.org/abs/2304.00906)
[![Second paper](https://img.shields.io/badge/arXiv-2406.13469-b31b1b.svg)](https://arxiv.org/abs/2406.13469)
[![License](https://img.shields.io/github/license/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/blob/main/LICENSE)
[![LastCommit](https://img.shields.io/github/last-commit/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/commits/main)
[![Code Coverage](https://img.shields.io/badge/Coverage-71%25-yellow.svg)](https://github.com/EuroEval/EuroEval/tree/main/tests)
[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](https://github.com/EuroEval/EuroEval/blob/main/CODE_OF_CONDUCT.md)

## Maintainer

- Dan Saattrup Smart ([@saattrupdan](https://github.com/saattrupdan), <dan.smart@alexandra.dk>)

## Installation and usage

See the [documentation](https://euroeval.com/python-package/) for more information.

## Reproducing the evaluation datasets

All datasets used in this project are generated using the scripts located in the
[src/scripts](src/scripts) folder. To reproduce a dataset, run the corresponding script
with the following command

```bash
uv run src/scripts/<name-of-script>.py
```

Replace <name-of-script> with the specific script you wish to execute, e.g.,

```bash
uv run src/scripts/create_allocine.py
```

## Contributors :pray:

A huge thank you to all the contributors who have helped make this project a success!

<a href="https://github.com/peter-sk">
    <img
        src="https://avatars.githubusercontent.com/u/6168908"
        width=50
        alt="Contributor avatar for peter-sk"
    />
</a>
<a href="https://github.com/AJDERS">
    <img
        src="https://avatars.githubusercontent.com/u/38854604"
        width=50
        alt="Contributor avatar for AJDERS"
    />
</a>
<a href="https://github.com/oliverkinch">
    <img
        src="https://avatars.githubusercontent.com/u/71556498"
        width=50
        alt="Contributor avatar for oliverkinch"
    />
</a>
<a href="https://github.com/versae">
    <img
        src="https://avatars.githubusercontent.com/u/173537"
        width=50
        alt="Contributor avatar for versae"
    />
</a>
<a href="https://github.com/KennethEnevoldsen">
    <img
        src="https://avatars.githubusercontent.com/u/23721977"
        width=50
        alt="Contributor avatar for KennethEnevoldsen"
    />
</a>
<a href="https://github.com/viggo-gascou">
    <img
        src="https://avatars.githubusercontent.com/u/94069687"
        width=50
        alt="Contributor avatar for viggo-gascou"
    />
</a>
<a href="https://github.com/mathiasesn">
    <img
        src="https://avatars.githubusercontent.com/u/27091759"
        width=50
        alt="Contributor avatar for mathiasesn"
    />
</a>
<a href="https://github.com/Alkarex">
    <img
        src="https://avatars.githubusercontent.com/u/1008324"
        width=50
        alt="Contributor avatar for Alkarex"
    />
</a>
<a href="https://github.com/marksverdhei">
    <img
        src="https://avatars.githubusercontent.com/u/46672778"
        width=50
        alt="Contributor avatar for marksverdhei"
    />
</a>
<a href="https://github.com/Mikeriess">
    <img
        src="https://avatars.githubusercontent.com/u/19728563"
        width=50
        alt="Contributor avatar for Mikeriess"
    />
</a>
<a href="https://github.com/ThomasKluiters">
    <img
        src="https://avatars.githubusercontent.com/u/8137941"
        width=50
        alt="Contributor avatar for ThomasKluiters"
    />
</a>
<a href="https://github.com/BramVanroy">
    <img
        src="https://avatars.githubusercontent.com/u/2779410"
        width=50
        alt="Contributor avatar for BramVanroy"
    />
</a>
<a href="https://github.com/peregilk">
    <img
        src="https://avatars.githubusercontent.com/u/9079808"
        width=50
        alt="Contributor avatar for peregilk"
    />
</a>
<a href="https://github.com/Rijgersberg">
    <img
        src="https://avatars.githubusercontent.com/u/8604946"
        width=50
        alt="Contributor avatar for Rijgersberg"
    />
</a>
<a href="https://github.com/duarteocarmo">
    <img
        src="https://avatars.githubusercontent.com/u/26342344"
        width=50
        alt="Contributor avatar for duarteocarmo"
    />
</a>
<a href="https://github.com/slowwavesleep">
    <img
        src="https://avatars.githubusercontent.com/u/44175589"
        width=50
        alt="Contributor avatar for slowwavesleep"
    />
</a>
<a href="https://github.com/mrkowalski">
    <img
        src="https://avatars.githubusercontent.com/u/6357044"
        width=50
        alt="Contributor avatar for mrkowalski"
    />
</a>
<a href="https://github.com/simonevanbruggen">
    <img
        src="https://avatars.githubusercontent.com/u/24842609"
        width=50
        alt="Contributor avatar for simonevanbruggen"
    />
</a>
<a href="https://github.com/tvosch">
    <img
        src="https://avatars.githubusercontent.com/u/110661769"
        width=50
        alt="Contributor avatar for tvosch"
    />
</a>
<a href="https://github.com/Touzen">
    <img
        src="https://avatars.githubusercontent.com/u/1416265"
        width=50
        alt="Contributor avatar for Touzen"
    />
</a>
<a href="https://github.com/caldaibis">
    <img
        src="https://avatars.githubusercontent.com/u/16032437"
        width=50
        alt="Contributor avatar for caldaibis"
    />
</a>
<a href="https://github.com/SwekeR-463">
    <img
        src="https://avatars.githubusercontent.com/u/114919896"
        width=50
        alt="Contributor avatar for SwekeR-463"
    />
</a>
<a href="https://github.com/N-essuno">
    <img
        src="https://avatars.githubusercontent.com/u/70454516"
        width=50
        alt="Contributor avatar for N-essuno"
    />
</a>
<a href="https://github.com/harderj">
    <img
        src="https://avatars.githubusercontent.com/u/5573365"
        width=50
        alt="Contributor avatar for harderj"
    />
</a>
<a href="https://github.com/lardinator">
    <img
        src="https://avatars.githubusercontent.com/u/45738512"
        width=50
        alt="Contributor avatar for lardinator"
    />
</a>

### Contribute to EuroEval

We welcome contributions to EuroEval! Whether you're fixing bugs, adding features, or
contributing new datasets, your help makes this project better for everyone.

- **General contributions**: Check out our [contribution guidelines](CONTRIBUTING.md)
  for information on how to get started.
- **Adding datasets**: If you're interested in adding a new dataset to EuroEval, we have
  a [dedicated guide](NEW_DATASET_GUIDE.md) with step-by-step instructions.

### Special thanks

- Thanks to [Google](https://google.com/) for sponsoring Gemini credits as part of their
  [Google Cloud for Researchers Program](https://cloud.google.com/edu/researchers).
- Thanks [@Mikeriess](https://github.com/Mikeriess) for evaluating many of the larger
  models on the leaderboards.
- Thanks to [OpenAI](https://openai.com/) for sponsoring OpenAI credits as part of their
  [Researcher Access Program](https://openai.com/form/researcher-access-program/).
- Thanks to [UWV](https://www.uwv.nl/) and [KU
  Leuven](https://www.arts.kuleuven.be/ling/ccl) for sponsoring the Azure OpenAI
  credits used to evaluate GPT-4-turbo in Dutch.
- Thanks to [Miðeind](https://mideind.is/en) for sponsoring the OpenAI
  credits used to evaluate GPT-4-turbo in Icelandic and Faroese.
- Thanks to [CHC](https://chc.au.dk/) for sponsoring the OpenAI credits used to
  evaluate GPT-4-turbo in German.

## Citing EuroEval

If you want to cite the framework then feel free to use this:

```bibtex
@article{smart2024encoder,
  title={Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks},
  author={Smart, Dan Saattrup and Enevoldsen, Kenneth and Schneider-Kamp, Peter},
  journal={arXiv preprint arXiv:2406.13469},
  year={2024}
}
@inproceedings{smart2023scandeval,
  author = {Smart, Dan Saattrup},
  booktitle = {Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)},
  month = may,
  pages = {185--201},
  title = {{ScandEval: A Benchmark for Scandinavian Natural Language Processing}},
  year = {2023}
}
```
