Metadata-Version: 2.4
Name: option-order-check
Version: 0.1.0
Summary: Measure how much a multiple-choice LLM eval depends on answer-option order: accuracy with intervals, flip rate, accuracy by answer slot.
Author: Robert Morong
License: MIT
Project-URL: Homepage, https://robert-morong-research.netlify.app/free/
Project-URL: Writeup, https://robert-morong-research.netlify.app/writing/arc-option-order/
Project-URL: Source, https://github.com/GrobeStreet/robert-morong-research
Keywords: llm,evaluation,benchmark,multiple-choice,robustness,option-order,position-bias
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# option-order-check

Does your multiple-choice LLM eval depend on where the answers are placed?

`option-order-check` shows every question to your model once per cyclic rotation of its options (A-B-C-D, B-C-D-A, ...) and reports, with uncertainty:

- mean accuracy over all rotations (95% Wilson interval)
- the **flip rate**: the share of questions whose chosen answer changes with option order
- accuracy by the slot the correct answer sits in (position bias shows up here)
- the share of questions answered correctly under every ordering

Pure standard library, no dependencies. Works offline with a built-in mock model, or with any OpenAI-compatible chat endpoint (OpenAI, Ollama, vLLM, LM Studio).

## Install and try it (no API key)

```bash
pip install option-order-check
option-order-check --demo --provider mock
```

The mock model has a deliberate lean toward one answer slot, so you can see what a biased model looks like in the output.

## Run it on your own model

Your file is JSONL, one question per line, with the correct option as a 0-based index:

```json
{"question": "Which gas do plants absorb?", "options": ["Oxygen", "Carbon dioxide", "Helium", "Neon"], "answer": 1}
```

```bash
# OpenAI
export OPENAI_API_KEY=...
option-order-check mytests.jsonl --provider openai --model gpt-4o-mini

# A local model through any OpenAI-compatible server (Ollama shown)
option-order-check mytests.jsonl --provider openai --model llama3 \
    --base-url http://localhost:11434/v1 --api-key-env NONE
```

Add `--json-out report.json` to save the numbers.

## Reading the output

A high flip rate at decent accuracy means your score depends on where answers are placed. A large spread between the best and worst correct-answer slot means the model has a position preference. See the audit this tool came from: [Shuffle the answers and small models change their minds](https://robert-morong-research.netlify.app/writing/arc-option-order/).

## Limits

The Wilson interval treats the rotations of one question as independent, which they are not, so read it as a rough guide. The flip-rate interval resamples questions. This tool measures option-order sensitivity only. It is not a full reliability audit, and it says nothing about safety, bias or domain correctness.

## License

MIT. Part of the free tooling at https://robert-morong-research.netlify.app/free/
