Metadata-Version: 2.4
Name: dqneval
Version: 0.1.0rc1
Summary: Deterministic evaluation suite for LLMs
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.27.0

# dqnEval

Deterministic, fully scriptable evaluation for LLMs. No LLM-based judging.

## Features
- 250 question suite across instruction fidelity, programming, reasoning, context awareness, common sense, language & grammar, and tool calling
- OpenAI-compatible Chat Completions API support (local or hosted)
- JSON and Markdown reports
- Reproducible, code-driven scoring
- Suite validation and category descriptions

## Quickstart

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .

dqneval list
dqneval describe
dqneval validate

dqneval run \
  --model gpt-4o-mini \
  --base-url https://api.openai.com/v1 \
  --api-key $OPENAI_API_KEY \
  --out-dir results
```

## CLI

```bash
dqneval list

dqneval run --model MODEL --base-url URL --api-key KEY
  [--categories instruction_fidelity,programming]
  [--max-questions 100]
  [--out-dir results]
  [--format json,md]
  [--shuffle]
  [--seed 123]
  [--verbose]
  [--temperature 0]
  [--max-tokens 512]
  [--max-retries 2]
  [--request-delay 0.5]
  [--question-retries 2]
  [--retry-backoff 5]
  [--retry-max-delay 60]
  [--continue-on-error]
  [--api-key-header Authorization]
  [--api-key-prefix Bearer]

Environment defaults:
- OPENAI_BASE_URL
- OPENAI_API_KEY
```

## Notes
- Prompts enforce JSON-only responses for deterministic parsing.
- Coding tasks run in a time-limited subprocess with tests.
