Metadata-Version: 2.4
Name: yoxla
Version: 0.1.0
Summary: Azerbaijani LLM evaluation framework and benchmark
Project-URL: Homepage, https://github.com/LocalDoc-Azerbaijan/yoxla
Project-URL: Repository, https://github.com/LocalDoc-Azerbaijan/yoxla
Project-URL: Dataset, https://huggingface.co/datasets/LocalDoc/YOXLA-Benchmark
Project-URL: Issues, https://github.com/LocalDoc-Azerbaijan/yoxla/issues
Author: LocalDoc
License: Apache-2.0
License-File: LICENSE
Keywords: azerbaijani,benchmark,evaluation,llm,nlp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11
Requires-Dist: datasets>=2.19
Requires-Dist: pyyaml>=6.0
Provides-Extra: api
Requires-Dist: openai>=1.0; extra == 'api'
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: local
Requires-Dist: accelerate; extra == 'local'
Requires-Dist: torch; extra == 'local'
Requires-Dist: transformers; extra == 'local'
Description-Content-Type: text/markdown

# YOXLA

Azerbaijani LLM evaluation framework and benchmark.

YOXLA is split in two:

* the **framework** — this Python package: inference, benchmark
  execution, evaluation, scoring and reporting;
* the **benchmark data** —
  [`LocalDoc/YOXLA-Benchmark`](https://huggingface.co/datasets/LocalDoc/YOXLA-Benchmark)
  on Hugging Face, versioned independently.

Updating the framework never changes the benchmark, and updating the
benchmark never requires a new release of the package.

## Install

```bash
pip install yoxla
```

API providers (OpenAI, OpenRouter, Google AI Studio, vLLM,
llama.cpp):

```bash
pip install "yoxla[api]"
```

Local Hugging Face models:

```bash
pip install "yoxla[local]"
```

## Quick start

```bash
yoxla validate-benchmark --block understanding
```

```bash
yoxla run --provider openrouter --model google/gemini-3.1-flash-lite --block understanding
```

A single task, three examples, for a smoke check:

```bash
yoxla run --provider transformers --model Qwen/Qwen3-0.6B --task nli_v1 --limit 3
```

## Blocks

1443 frozen examples across four blocks.

### Understanding — 600

| Task               | Examples | Answer          | Primary metric   |
| ------------------ | -------- | --------------- | ---------------- |
| `extractive_qa_v1` | 200      | text            | `span_recall`    |
| `qa_abstention_v1` | 100      | text or abstain | `overall_score`  |
| `nli_v1`           | 100      | choice          | `accuracy`       |
| `sts_v1`           | 100      | numeric 0–5     | `spearman`       |
| `sentiment_v1`     | 50       | choice          | `accuracy`       |
| `intent_v1`        | 50       | choice          | `accuracy`       |

### Language — 400

| Task                    | Examples | Answer | Primary metric |
| ----------------------- | -------- | ------ | -------------- |
| `minimal_pairs_v1`      | 200      | choice | `accuracy`     |
| `az_tr_interference_v1` | 200      | choice | `accuracy`     |

Both show two sentences differing in one word and ask which one is
Azerbaijani. What differs is what the one word tests. Position is
balanced inside every breakdown of both, so answering "A" throughout
scores 50 on the task and 50 on each breakdown.

`minimal_pairs_v1` corrupts a named rule — vowel harmony, the
question particle, the definite accusative, case government and four
more — which is what makes the label derivable rather than a matter
of opinion.

`az_tr_interference_v1` replaces an Azerbaijani word with a Turkish
one:

```text
A) Uşaqlar məktəbdə çox gözəl danışmaq öyrənirlər.
B) Uşaqlar məktəbdə çox gözəl konuşmak öyrənirlər.
```

That is not a spelling slip. It is the failure mode of a model
trained mostly on the larger neighbouring language, and it is the one
measurement that separates a model which learned Azerbaijani from one
which learned Turkish and is guessing.

The gold comes from outside the project, because "this word is not
Azerbaijani" is a fact about a lexicon and not something derivable.
Three lookups in two independent sources decide every pair, and each
source is used only in the direction it can support:

| | |
| --- | --- |
| the Azerbaijani word | is in the hunspell dictionary and occurs ≥ 200 times in Azerbaijani Wikipedia |
| the Turkish word | is in the Turkish dictionary and **absent** from the Azerbaijani one |
| the corpus | has the Azerbaijani word outnumbering the Turkish one ≥ 50:1 |

Presence in the dictionary proves a form is Azerbaijani; absence
proves nothing, because its verb paradigms have holes — `soruşmaq` is
missing while `danışmaq` is present, and treating absence as proof
would mark correct Azerbaijani as Turkish. The corpus runs the other
way: a form seen tens of thousands of times is in use whatever the
dictionary says, but a low count is not proof either, since Turkish
words appear in Azerbaijani Wikipedia inside quotations and film
titles — `kitap` 166 times, `kent` 297. What separates a pair is the
ratio between its two forms, which runs from 54:1 to 80162:1 across
the set. Sources and thresholds: `data/lexicon/SOURCES.md` in the
dataset repository.

Whether that machinery was worth it is measured rather than argued.
Twenty pairs were written by hand before the sources were in place
and the dictionary rejected five of them: `araba`, `bilgisayar`,
`durum` and `subay` are all Azerbaijani words, and `sormaq` is a real
stem. A quarter of a hand-written table was wrong in a way no reader
would have noticed, and a model asked to produce the same table is
wrong for the same reason — it is a lexicographic fact, not a
judgement.

Every word of the Azerbaijani sentence is checked too, not only the
target, so the sentence a model is asked to prefer is verified
Azerbaijani throughout rather than merely correct in one position.

`per_interference_type_accuracy` splits the score two ways, and the
split matters more than the total:

| type | the pair | what it tests |
| --- | --- | --- |
| `cognate` | `kitab` / `kitap` | which sound shape Azerbaijani uses |
| `distinct_lexeme` | `danışmaq` / `konuşmak` | which word it uses at all |

Only the second is beyond a model that merely spells Azerbaijani
correctly. The set is 163 distinct pairs to 37 cognates, and on a
subset of 37 a single item is worth 2.7 points — read that column as
an indication, not a measurement.

### Knowledge — 150

| Task                  | Examples | Answer            | Primary metric |
| --------------------- | -------- | ----------------- | -------------- |
| `knowledge_choice_v1` | 150      | choice, 1–20      | `accuracy`     |

Facts about Azerbaijan harvested from Wikidata across seven
categories. Twenty candidates per question; the model returns a
number.

A fact Wikidata answers more than one way is dropped at build time.
Nizami Ganjavi carries three death years, so no question is built
from it: a model that knows one of the other two would be marked
wrong by the gold, not by its own error.

An open-answer version was built first and dropped. Sixty-two of the
hundred and fifty answers are multi-word entities — honorific titles
no two people would word alike, names with up to seven equally
correct forms — and every scoring dispute found in review came from
that group. Scoring them needs a judge or a list of accepted
spellings someone keeps guessing at, and neither belongs in a frozen
benchmark. Closing the answer space gives up the distinction between
recall and recognition; what it buys is that a wrong answer is wrong
because the model chose wrong, never because a check disagreed about
spelling.

A closed space is only worth something if the distractors are hard,
which is a property of the data rather than of the code that reads
it — so it is measured. Three cues would let a model answer without
knowing the fact, and each is closed at build time: distractors come
from the same probe, so the wrong *kind* of thing cannot be
eliminated; the answer is never the most-linked option, so "pick the
famous one" fails; a year distractor sits within a decade of the
answer, so knowing the century does not narrow the field. Cue-only
adversaries then run over the finished set and the build fails if any
beats its floor by more than five points.

`modal_answer_share` is the diagnostic to read beside the score. A
model that does not know the answer tends to return the same number
every time — one run sat at 0.7 — and accuracy alone does not show
it.

### RAG — 293

| Task                  | Examples | Answer          | Primary metric |
| --------------------- | -------- | --------------- | -------------- |
| `rag_verification_v1` | 160      | choice          | `accuracy`     |
| `rag_selection_v1`    | 133      | choice, 1–6 or none | `accuracy` |

The two halves of a retrieval system, measured apart:
`rag_selection_v1` asks whether the right passage was picked up,
`rag_verification_v1` asks what the model does with a passage once it
has one. A system can fail at either, and one number would hide which.

#### `rag_verification_v1`

A passage and one claim about it. The model says whether the passage
supports the claim, contradicts it, or is silent about it —
`TƏSDİQ` / `ZİDD` / `YOXDUR`.

The third label is what makes hallucination measurable. Asked only
"true or false", a model has nowhere to put a claim the passage never
addresses, and inventing support for it looks the same as reading
correctly.

Absence is decidable here in a way it is not for a found document,
because each passage is written from a known list of three facts.
Anything outside that list is provably not stated — a claim about a
death year in a passage that states only a birthplace is unsupported
by construction, not by judgement.

Four claims come from every passage, and `per_claim_type_accuracy` is
the number to read rather than the average:

| claim type | the passage | the claim | label |
| --- | --- | --- | --- |
| `supported` | states a fact | restates it | `TƏSDİQ` |
| `contradicted` | states a fact | alters it | `ZİDD` |
| `counterfactual` | states it **altered** | states the true value | `ZİDD` |
| `absent` | never mentions it | asserts it | `YOXDUR` |

The third row is worth the most. The passage says Nizami died in
1195; the claim says 1202, which is true of the world and false of
the passage. A model answering `TƏSDİQ` has read its own memory
instead of the text — the failure that makes a retrieval system
quietly wrong. A model can hold the first two labels and still fail
that column, and one averaged number hides it.

A fact Wikidata answers more than one way is not used here either,
and for a sharper reason than in Knowledge. The theatre and the
university in the harvest were both renamed, so each carries two
names — and a passage built from one of them while a claim asserted
the other produced a `ZİDD` that is not defensible: two periods are
not a contradiction, and a model answering `YOXDUR` would be reading
correctly. Those subjects are skipped, not relabelled.

The generator model wrote the passages and the claims; it never
decided a label. Every label follows from the fact list, and a
generation that did not match the list was rejected rather than
relabelled. The uploader re-derives all four labels from Wikidata and
the finished passage before publishing, and refuses to publish if the
arrangement it expects is not there.

#### `rag_selection_v1`

A question and six short passages, roughly seventy words in all. The
model answers with the number of the passage containing the answer,
or `HEÇ BİRİ` if none of them does.

The whole measurement is in the negatives, and the first attempt at
this set was abandoned for lack of them. Recombining the extractive QA
contexts would have been free, but those questions were written from
those contexts and inherit their wording: the true context had the
highest word overlap with the question in **187 of 200** items, so
"pick the passage sharing the most words" would have solved the task
without reading anything.

Here the passages are written from single facts about a known
subject, so every passage about that subject carries its name and
overlap stops pointing anywhere:

| negative | example, asked where Nizami was born |
| --- | --- |
| same subject, other aspect | his death year |
| same aspect, other subject | another poet's birthplace |
| same subject, same value kind | his death year against his birth year — one word of relation decides |

A fourth kind is deliberately absent. A passage stating an altered
value — "Nizami was born in Baku" — is relevant to the question and
merely false, so counting it as a wrong pick would conflate "not the
right passage" with "the right passage, wrong fact". The second thing
is what `rag_verification_v1` measures.

One item in seven has its answer in none of the passages, which is
what `per_item_type_accuracy` splits out and the number worth reading.
A retrieval system that always returns its best guess fails there and
nowhere else. The share is one in seven rather than more so that every
one of the seven answers is equally likely: at a quarter, answering
`HEÇ BİRİ` to everything scored 20% against a chance of 14%.

**One adversary here cannot be driven to chance, and the floor is
reported rather than hidden.** The gold is the only passage carrying
both the subject and the relation, and a negative carrying both would
be a passage stating the answer. So the gold is held to a *tie* with
the other passages about its subject — a word counter then faces two
or three equals and earns a third or a half, which is a floor near
0.35 rather than the 0.14 that every other adversary in this benchmark
sits at. It is computed over the finished set, printed at build time,
and re-checked by the uploader.

The other cue was found only after the set looked finished. Passages
about an institution run longer than passages about a person, simply
because the name is longer, so "pick the longest" scored 0.254 against
a permutation null of 0.143. The negatives are now matched to the gold
for length, and items where the gold is still the single longest are
dropped.

The set is 133 items rather than a round 200 because of those two
rules. Every item they removed was one where the gold gave itself
away, and loosening them to reach a round number would have bought the
number with the cues.

**This task does not rank models.** Across nine models it spans 2.3
points, 97.0 to 99.2, and the weakest model scores 96.99 — see
[First results](#first-results). Choosing which of six short passages
holds a fact turns out to be far easier than knowing the fact, and the
negatives being hard *lexically* does not change that. Read it as a
check that a retrieval pipeline is not broken, not as a measurement of
one.

```bash
yoxla tasks
```

Each task score is normalized to `[0, 100]`, and the block score is
the **macro average over tasks** — not over examples, so the 200 QA
items do not outweigh the 50 intent items.

## First results

Nine models, 13 September 2026, every task of all four blocks.
Temperature 0, reasoning disabled, prompts frozen and hashed, and zero
generation errors in every run — so the scores are comparable with one
another.

| Model | Understanding | Language | Knowledge | RAG | YOXLA |
| --- | ---: | ---: | ---: | ---: | ---: |
| `google/gemini-3.1-flash-lite` | 92.48 | 99.50 | 50.67 | 97.06 | **84.93** |
| `openai/gpt-5.4-mini` | 86.68 | 97.00 | 39.33 | 97.75 | **80.19** |
| `google/gemma-4-31b-it` | 91.56 | 95.75 | 26.00 | 97.75 | **77.76** |
| `deepseek/deepseek-v4.1-flash` | 89.98 | 77.75 | 44.67 | 97.37 | **77.44** |
| `qwen/qwen3.6-27b` | 92.91 | 94.50 | 18.67 | 88.06 | **73.53** |
| `qwen/qwen3.8-27b` | 91.24 | 84.50 | 18.00 | 97.25 | **72.75** |
| `google/gemma-4-26b-a4b-it` | 88.01 | 90.25 | 14.67 | 97.00 | **72.48** |
| `openai/gpt-4o-mini` | 84.82 | 67.00 | 21.33 | 98.69 | **67.96** |
| `qwen/qwen3.5-9b` | 87.11 | 79.00 | 10.67 | 94.75 | **67.88** |

### Which tasks rank models, and which do not

The spread across those nine models is the benchmark measuring itself.

| Task | Lowest | Highest | Range |
| --- | ---: | ---: | ---: |
| `az_tr_interference_v1` | 45.5 | 99.5 | **54.0** |
| `knowledge_choice_v1` | 10.7 | 50.7 | **40.0** |
| `rag_verification_v1` | 76.9 | 98.1 | 21.2 |
| `qa_abstention_v1` | 72.1 | 92.7 | 20.6 |
| `nli_v1` | 76.0 | 93.0 | 17.0 |
| `minimal_pairs_v1` | 84.5 | 99.5 | 15.0 |
| `extractive_qa_v1` | 79.3 | 91.6 | 12.2 |
| `intent_v1` | 90.0 | 100.0 | 10.0 |
| `sentiment_v1` | 92.0 | 98.0 | 6.0 |
| `sts_v1` | 87.4 | 91.5 | 4.1 |
| `rag_selection_v1` | 97.0 | 99.2 | 2.3 |

**Four tasks no longer rank anything.** Passage selection spans 2.3
points across all nine models, similarity 4.1, sentiment 6.0, intent
10.0. Their build-time gates proved those sets cannot be solved
*without reading* them; that is not the same as being hard to solve
*while* reading, and only real models show the difference. Keep them as
a check that nothing is broken, not as a measurement.

### A model answered in Turkish without being asked

On `rag_verification_v1` a model must reply `YOXDUR` when the passage
says nothing about the claim. `qwen/qwen3.6-27b` wrote `YOKDUR` — the
Turkish spelling — on 30 of those 40 items.

It had identified the silence correctly and named it in the wrong
language. Strict parsing counted those as unusable output, which is why
its RAG block reads 88.06 instead of about 97; a lenient parser would
have hidden a language failure behind a good score.

That is also a measurement this benchmark is not supposed to be able to
make. `az_tr_interference_v1` tests whether a model *recognises* a
Turkish word, and this model scores 96.00 there while *writing* one
three times out of four. The recognition-production gap
[documented below](#what-the-benchmark-does-not-measure) is real, and
it showed up by accident, because this task's label happens to be a
word the two languages spell differently.

### The widest spread is lexical, not grammatical

`gpt-4o-mini` scores 45.5 on `az_tr_interference_v1` — below chance on a
two-way choice, with no invalid output to explain it. It does not guess;
it prefers the Turkish sentence. The same model scores 88.5 on
grammaticality, so it has the rules and not the vocabulary: it is
reading Azerbaijani through the larger neighbouring language.

Knowledge is the other discriminator, and the only block where the
ordering changes substantially. Models in the nineties on comprehension
fall to between 10.67 and 50.67 on facts about Azerbaijan, against a
chance of 5.

## Span answers

`extractive_qa_v1` and `qa_abstention_v1` put the answer inside a
passage the model is given, so they are scored on character offsets
rather than on strings. The quote is located in the passage and its
range compared with the gold's — no accepted forms, no normalizer,
nothing to maintain when a new way of writing the same span turns up.
The gold answer occurs exactly once in its own context in all 250
examples, which is what makes the position unambiguous; a run refuses
to start if that stops being true.

Two numbers come out of it, and they are reported apart because they
are different abilities:

| | |
| --- | --- |
| `span_recall` | did the quote cover the answer — **the score** |
| `span_precision` | how much else it dragged in |

Measured over five models, recall sits between 0.83 and 0.85 for the
top three while precision spreads from 0.48 to 0.92. The model that
ranked last on token F1 sits fourth on recall: most of its old deficit
was the length of its quotes, not its reading. Folding the two
together reads that as a comprehension failure, which it is not.

Recall is credited only when the quote stays inside the sentences the
answer occupies — otherwise copying the whole passage would score a
perfect recall. Models stay inside that window 79 to 95 per cent of
the time, so the guard bites only on what it is meant to catch.

`unsupported_rate` comes free from the same machinery: a quote that
cannot be found in the passage was not copied from it.

## How answers are judged

Parsing is strict: only the requested answer format counts.

```text
neutral                  valid
`neutral`   "neutral"    valid   (symmetric wrappers are removed)
Neutral                  valid   (case-insensitive, AZ-aware)
Neutral.                 invalid
The answer is neutral    invalid
```

An invalid answer is a result, never a reason to re-prompt: one
benchmark example is one model attempt. Answer normalization never
folds Azerbaijani letters (`ə → e`, `ş → s`, `ı → i`), because that
would reward models that do not write correct Azerbaijani.

Retries exist only for transport failures (timeouts, 429, 5xx).

## What the benchmark does not measure

Every answer space is closed: a label, a number, or a span quoted
from a passage the model was given. That is what lets every score be
checked without a judge and reproduced from stored output — and it
draws a hard boundary around what the numbers can mean.

**Production is not measured.** No task asks the model to write
Azerbaijani and scores what it wrote. An orthography block did
exactly that — a short text from a handful of facts, scored on
whether the spelling rules survived — and it is gone, along with the
open-answer knowledge task. The same model recognises correct
Azerbaijani grammar at 90.5 and, in the same run, scored 57.7 on
writing it, so the two are not interchangeable and the second number
no longer exists anywhere in the benchmark.

**Recall is not measured, only recognition.** Knowledge asks the
model to pick a fact out of twenty candidates, never to produce it,
and the interference task asks it to recognize a Turkish word rather
than to avoid writing one. A model can score well on both while its
own Azerbaijani drifts toward Turkish, and nothing here would show
it.

**Nothing above sentence level is measured.** No task covers
discourse, long context, multi-turn behaviour, or instruction
following beyond answer format.

A benchmark that scores every one of these would need either a judge
model or human annotation. Both were considered and neither is used:
a judge would put one model's Azerbaijani inside the loop that
measures Azerbaijani, and would end reproducibility — `rescore` would
stop reproducing a number and a frozen manifest would guarantee
nothing.

## Reproducibility

Every run writes:

```text
runs/<run_id>/
├── run.json            model, provider, YOXLA version, benchmark
│                       version, dataset revision and fingerprint,
│                       prompt version and prompt hashes
├── predictions.jsonl   one line per example, raw output included
├── task_scores.json    metrics and normalized score per task
└── summary.json        block and benchmark aggregates
```

Pin the data for a comparable run:

```bash
yoxla run --provider openai --model gpt-5 --block understanding --revision a66c827c36b249a6137eda67a0110da469b4b175
```

`yoxla validate-benchmark` compares the data against the frozen
manifest bundled with the package and warns when the Hub content has
changed.

Raw model output is always stored, so metrics can be recomputed
without paying for inference again:

```bash
yoxla rescore runs/openrouter_some-model_20260818T135510Z
```

`rescore` replays the stored answers through the current evaluators
and rewrites `task_scores.json` and `summary.json`. Model answers are
never modified.

### Span-based tasks

Scoring these on one number used to blend two abilities, and it
ranked models almost inversely. Under `token_f1` gpt-4o-mini came
last while locating the answer more often than any other model,
because it returned the sentence around the span rather than the
span.

The offset metrics separate them — see [Span answers](#span-answers).
The same five runs, rescored:

| model | `span_recall` | `span_precision` | over-sentence | old `token_f1` |
| --- | --- | --- | --- | --- |
| gemini-3.1-flash-lite | 0.849 | 0.915 | 2.0% | 0.942 |
| mistral-small-4 | 0.848 | 0.738 | 2.0% | 0.844 |
| llama-3.3-70b | 0.834 | 0.884 | 2.5% | 0.901 |
| gpt-4o-mini | 0.794 | 0.483 | 10.5% | 0.664 |
| qwen2.5-7b | 0.630 | 0.693 | 4.0% | 0.725 |

The top three find the answer equally well and are separated in the
old number mostly by how tightly they quote — which is now its own
column rather than a hidden term in the score. gpt-4o-mini moves from
last to fourth: its `token_f1` of 0.664 was largely a brevity penalty.

The separation is not total. Recall is forfeited when a quote runs
past the sentence holding the answer, so a model that habitually
returns two sentences — gpt-4o-mini does it in one example out of ten
— still loses recall for it. Without that guard recall would be free:
copy the passage, score 1.0.

`token_f1`, `normalized_em` and `strict_em` stay as diagnostics. They
are what earlier runs were scored on, and keeping them visible is what
makes the change in the headline number explainable.

### Running in parallel

A full run is 1443 requests and by default each waits for the last.
`--workers` puts several in flight:

```bash
yoxla run --provider openrouter --model some/model --block all --workers 8
```

Requests are issued from a pool but **results are consumed in example
order**, so `predictions.jsonl` reads the same whatever the schedule
was, `rescore` still replays it, and an interrupted run still resumes
from the last line on disk. On a synthetic 60-example run the speedup
is close to linear — ×3.9 at four workers, ×9.7 at ten — and the score
is unchanged.

**The default stays 1 because the schedule can move the score.** Under
load a provider answers 429 and times out more often; an example whose
retries run out is recorded as a generation error, and that counts
against the model. A model run on twenty threads can therefore score
below the same model run on one, for a reason that has nothing to do
with the model. The worker count is written into `run.json`, and the
summary prints `generation errors` — check it is zero before comparing
two runs.

Local Transformers models gain nothing here: they serialize on the
device.

### Resuming

`predictions.jsonl` is flushed after every example. If a long API run
dies, continue it with the same run id:

```bash
yoxla run --provider openrouter --model some/model --block understanding --run-id my-run --resume
```

Cached examples are replayed into the evaluators, so a resumed run
scores exactly like an uninterrupted one.

## Providers

Providers describe *how to connect*; the model id is given at
runtime, so a newly released model needs no change to YOXLA.

```bash
yoxla providers
```

```text
openrouter   openai_compatible   OPENROUTER_API_KEY
google       openai_compatible   GEMINI_API_KEY
openai       openai_compatible   OPENAI_API_KEY
vllm         openai_compatible   http://127.0.0.1:8000/v1
llamacpp     openai_compatible   http://127.0.0.1:8080/v1
transformers transformers        local / Hugging Face
```

A custom registry can be supplied with `--config my-providers.yaml`.

### Reasoning models

Benchmark v1 runs with thinking disabled. That has to be stated to
the provider, not assumed: OpenRouter turns reasoning on by default
for models that support it, and some of them default to a high effort
that spends the entire output budget before producing an answer
token — the result is an empty response with `finish_reason: length`.

The `openrouter` provider therefore declares:

```yaml
thinking_control: reasoning
```

which sends `reasoning: {"enabled": false}` (or `true` under
`--thinking`). llama.cpp uses `chat_template_kwargs` for the same
purpose. Providers without a declared control are sent no reasoning
parameter at all.

When a model does return reasoning, it is preserved in `raw_text`
inside `<think>` tags while `text` — the scored field — holds only the
final answer.

Connection smoke test:

```bash
yoxla test --provider openrouter --model google/gemini-3.1-flash-lite
```

## Adding a task

A task is a registry entry, not a code change to the runner. It
declares its dataset, field mapping, prompt, answer space, generation
settings, parser mode and evaluator — see
the modules under `src/yoxla/benchmark/tasks/`.

A task id such as `nli_v1` freezes the whole contract: dataset,
prompt, answer space and evaluator. Any substantial change means a
new id (`nli_v2`), never an edit in place, and the superseded task
stays registered and runnable so an older run can be reproduced.

**That freeze starts at first publication, not at first commit.**
All four blocks are now on the Hub, so the nine task ids are frozen
and a substantial change to any of them means a new id. What was
changed in place before publication was changed at `v1`, which is why
there is no `v2` anywhere. No task is retired.

## Status

Implemented: inference (API, local Transformers, bitsandbytes/Unsloth,
GGUF via llama.cpp), the Understanding, Language, Knowledge and RAG
blocks, deterministic evaluators, scoring, run artifacts, resume,
validation and parallel execution. Nine models have been measured —
see [First results](#first-results).

Not implemented yet: RAG extraction, Language Under Load, and the
leaderboard.

AZ/TR purity was on that list and is now half done. Recognizing a
Turkish word in an Azerbaijani sentence is measured
(`az_tr_interference_v1`); whether a model's own output stays
Azerbaijani is not, and cannot be without scoring produced text.

Removed rather than postponed: the generation block and the
open-answer knowledge task, both of which needed a judgement about
what counts as the same answer — see
[What the benchmark does not measure](#what-the-benchmark-does-not-measure).

## License

Apache-2.0
