# Reading Comprehension

## 📚 Overview

Reading comprehension is a task of determining whether a model is able to understand a
given text and answer questions about it. The model receives a text passage and a
question about the text, and it has to provide the answer as it is stated in the text.
This is very related to Retrieval-augmented Generation (RAG) applications, where a
generative model is used to answer a question based on one or more retrieved documents.

When evaluating generative models, we allow the model to generate 32 tokens on this
task.

## 📊 Metrics

The primary metric we use when evaluating the performance of a model on the reading
comprehension task is the [F1-score](https://en.wikipedia.org/wiki/F1_score) on a
character-basis.

We also report the exact match (EM) score, which is the percentage of questions
for which the model provides the exact answer. It is less lenient than the EM score,
as it does not allow for small differences in the answer.

## 🛠️ How to run

In the command line interface of the [EuroEval Python package](/python-package), you
can benchmark your favorite model on the reading comprehension task like so:

```bash
euroeval --model <model-id> --task reading-comprehension
```
