Metadata-Version: 2.4
Name: raidex
Version: 0.1.4
Summary: Self-contained CLI to measure an LLM against the Raidex Responsible-AI index
Author: Vishnu Vettrivel
License: MIT License
        
        Copyright (c) 2026 Vishnu Vettrivel
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Keywords: llm,evaluation,responsible-ai,safety,benchmark
Requires-Python: <3.14,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: litellm[proxy]>=1.40
Requires-Dist: lm-eval[api]>=0.4.5
Requires-Dist: datasets>=2.18
Requires-Dist: huggingface_hub>=0.23
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: pyyaml>=6.0
Requires-Dist: platformdirs>=3.0
Dynamic: license-file

# Raidex

**Measure any LLM's Responsible-AI profile in your own environment: your own fine-tuned or self-hosted model, or a frontier model.** [raidex.ai](https://raidex.ai)

`raidex` scores a model across open Responsible-AI benchmarks (safety, fairness, factuality, security, machine ethics, robustness, privacy, and sycophancy) and reports a composite **RAI Score** plus per-dimension scores. The same measurement core also powers a public [leaderboard of frontier models](#the-public-leaderboard).

## Quickstart

```bash
pip install raidex        # Python 3.10 to 3.13
```

**Measure your own or self-hosted model** against any OpenAI-compatible endpoint (vLLM, Ollama, TGI, and so on):

```bash
raidex eval --model http://localhost:8000/v1 --served-name my-model --tier A+B
```

**Measure a frontier model** with any [litellm](https://docs.litellm.ai/docs/providers) model string. The provider key is read from the matching env var, for example `OPENAI_API_KEY`:

```bash
export OPENAI_API_KEY=sk-...          # plus ANTHROPIC_API_KEY for the judges
raidex eval --model openai/gpt-5.2 --tier A+B
```

No account, no queue, no upload, and no dependency on Raidex servers. `raidex` prints per-dimension and composite RAI scores and writes a self-describing JSON that never leaves your machine.

## What you get

```
=== Raidex RAI Score ===
  RAI Score : 63.5
  Coverage  : 9/9  🟣
  Dimensions:
    safety           71.2
    fairness_bias    35.3
    factuality       52.0
    ...
```

- **Board-comparable.** Identical in scale to the public leaderboard (same core, same benchmarks, same normalization), so you can place your model against the frontier.
- **A self-describing result JSON.** The model spec, per-benchmark pinned dataset versions, judge, sampling settings, and a timestamp, so a score is reproducible and traceable. It is written locally and nothing is uploaded.
- **Honest coverage.** The composite is the mean of the constituents you ran, reported as N/9. Benchmarks you skip, or that need a judge you did not configure, simply lower coverage; they never fake a number.

## CLI reference

```bash
raidex eval --model ...                         # required: a litellm model id OR a local endpoint URL
            --served-name my-model              # the model name served by a local endpoint URL
            --tier A | B | A+B                  # A = 7 core benchmarks, B = +robustness/privacy (default A)
            --benchmarks bbq,strongreject       # explicit subset (overrides --tier)
            --limit 150                         # sample the big benchmarks (small ones always run full)
            --judge anthropic/claude-opus-4-8   # LLM judge for SimpleQA / XSTest / StrongREJECT
            --dry-run                           # print a cost estimate and exit
            --offline                           # use only cached data; never touch the network
            --output results.json               # where to write the result (default: <model>__.json)

raidex fetch-data                               # pre-download and cache all benchmark data (for offline / air-gapped use)
```

**Judges.** SimpleQA, XSTest, and StrongREJECT are graded by an LLM judge. Configure one with `--judge` (or `RAIDEX_JUDGE_MODEL`). If none is available, those three are skipped with a printed reason and honestly reduced coverage, not a failure.

**Offline and air-gapped.** Run `raidex fetch-data` on a networked machine to populate a local cache (pinned dataset versions), copy that cache across, and run with `--offline` for zero network access.

## The benchmarks

`raidex` runs **9 benchmarks across 8 dimensions**. The **RAI Score** is the mean of normalized constituent scores (0 to 100); coverage is reported as N/9.

| Tier | Benchmark | Dimension | Pipeline |
|------|-----------|-----------|----------|
| A | BBQ | Fairness & Bias | lm-eval (generative) |
| A | WMDP | Security | lm-eval (generative) |
| A | SimpleQA | Factuality | litellm + judge |
| A | StrongREJECT | Security (refusal) | litellm + rubric judge |
| A | ETHICS | Machine Ethics | lm-eval (generative) |
| A | XSTest | Safety (over-refusal) | litellm + judge |
| A | Sycophancy | Sycophancy | litellm (judge-free flip-rate) |
| B | AdvGLUE | Robustness | litellm (exact-match) |
| B | ConfAIde | Privacy | litellm (correlation) |

See [`space/METHODOLOGY.md`](space/METHODOLOGY.md) for the index design, generative-task creation, judging, sampling, normalization, and disclosures.

## The public leaderboard

There is also a public board of frontier models, produced by the same core:

- **Live leaderboard:** https://huggingface.co/spaces/cloudronin/raidex-space
- **Results dataset:** https://huggingface.co/datasets/cloudronin/raidex-results
- **Eval queue:** https://huggingface.co/datasets/cloudronin/raidex-requests

Running the board, the Space, or reproducing the published numbers is a maintainer task. See [`docs/leaderboard.md`](docs/leaderboard.md).

## Repository layout

- [`raidex/`](raidex/): the pip-installable **`raidex` CLI** (`raidex/cli.py`) over the pure **`raidex.core`** eval-and-score library. The core is the shared foundation; the CLI and the backend service are two thin frontends over it, which is why a local score matches the board.
- [`space/`](space/): the Hugging Face **Space** (Gradio leaderboard app). See [`docs/leaderboard.md`](docs/leaderboard.md).
- [`backend/`](backend/): the **eval service** that produces the public board. See [`docs/leaderboard.md`](docs/leaderboard.md).

## License

MIT. See [LICENSE](LICENSE).
