Metadata-Version: 2.4
Name: ehq
Version: 0.4.0
Summary: Reproducible evaluation framework for the Epistemic Honesty Quotient and EHQ-3000
Author: H. Russell Bernard, Huan Liu
Author-email: Ali Şenol <alisenol@tarsus.edu.tr>
Maintainer-email: Ali Şenol <alisenol@tarsus.edu.tr>
License-Expression: MIT
Project-URL: Homepage, https://github.com/senolali/EHQ
Project-URL: Documentation, https://github.com/senolali/EHQ#readme
Project-URL: Repository, https://github.com/senolali/EHQ.git
Project-URL: Issues, https://github.com/senolali/EHQ/issues
Project-URL: Changelog, https://github.com/senolali/EHQ/blob/main/CHANGELOG.md
Project-URL: Dataset, https://raw.githubusercontent.com/senolali/EHQ/main/data/releases/EHQ-3000.json
Keywords: large-language-models,hallucination,abstention,calibration,benchmark,evaluation,epistemic-honesty
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests<3,>=2.31
Requires-Dist: truststore<1,>=0.10; sys_platform == "win32"
Provides-Extra: analysis
Requires-Dist: numpy<3,>=1.24; extra == "analysis"
Requires-Dist: scipy<2,>=1.10; extra == "analysis"
Provides-Extra: sources
Requires-Dist: lxml<7,>=5; extra == "sources"
Requires-Dist: pdfplumber<1,>=0.11; extra == "sources"
Provides-Extra: reporting
Requires-Dist: openpyxl<4,>=3.1; extra == "reporting"
Requires-Dist: matplotlib<4,>=3.7; extra == "reporting"
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == "dev"
Requires-Dist: pytest<9,>=8; extra == "dev"
Requires-Dist: pytest-cov<7,>=5; extra == "dev"
Requires-Dist: twine<7,>=5; extra == "dev"
Provides-Extra: all
Requires-Dist: numpy<3,>=1.24; extra == "all"
Requires-Dist: scipy<2,>=1.10; extra == "all"
Requires-Dist: lxml<7,>=5; extra == "all"
Requires-Dist: pdfplumber<1,>=0.11; extra == "all"
Requires-Dist: openpyxl<4,>=3.1; extra == "all"
Requires-Dist: matplotlib<4,>=3.7; extra == "all"
Dynamic: license-file

# Epistemic Honesty Quotient (EHQ)

[![PyPI](https://img.shields.io/pypi/v/ehq.svg)](https://pypi.org/project/ehq/)
[![Python](https://img.shields.io/pypi/pyversions/ehq.svg)](https://pypi.org/project/ehq/)
[![CI](https://github.com/senolali/EHQ/actions/workflows/ci.yml/badge.svg)](https://github.com/senolali/EHQ/actions/workflows/ci.yml)
[![Code license: MIT](https://img.shields.io/badge/code%20license-MIT-blue.svg)](LICENSE)
[![Dataset license: CC BY 4.0](https://img.shields.io/badge/data%20license-CC%20BY%204.0-lightgrey.svg)](DATA_LICENSE.md)

EHQ is a provider-neutral, reproducible framework for measuring how large
language models regulate their assertions at knowledge and context boundaries.
It evaluates restraint, avoidance of unqualified false assertions, and
confidence calibration. The repository also releases **EHQ-3000**, a balanced
3,000-item English benchmark covering four distinct boundary conditions.

## Quick links

| Resource | Link |
|---|---|
| EHQ-3000 dataset | [Download JSON](https://raw.githubusercontent.com/senolali/EHQ/main/data/releases/EHQ-3000.json) |
| Dataset card | [Read the data card](docs/EHQ-3000_DATA_CARD.md) |
| Normative protocol | [Read the protocol](docs/protocol.md) |
| Pilot/full guide | [Run an evaluation](RUNNING_PILOT_AND_FULL_STUDIES.md) |
| Citation metadata | [CITATION.cff](CITATION.cff) · [BibTeX](CITATION.bib) |

Canonical dataset identity:

```text
File:       EHQ-3000.json
Items:      3,000
Language:   English
Version:    1.0
SHA-256:    3d8e440a21cca46907a2d70378bc1a9d2fc7690cf63ad20d320d3926bb19ddfa
License:    CC BY 4.0 (subject to the third-party-material notice)
```

Any mirror, including a future Hugging Face or Zenodo copy, should be treated
as the same release only when this SHA-256 value matches.

## What the framework provides

- OpenAI Responses API support;
- Hugging Face Inference Providers support;
- configurable OpenAI-compatible endpoints, including OpenRouter, Together,
  Groq, vLLM, Ollama, and compatible self-hosted servers;
- deterministic, balanced item selection;
- two-turn answer and confidence elicitation;
- auditable response classification and category-aware correctness checks;
- checkpoint/resume and successful-response-only content-addressed caching;
- explicit technical-failure and missing-confidence accounting;
- human-readable progress, throughput, and ETA messages;
- JSON, JSONL, CSV, Excel, figures, and publication-oriented reports;
- SHA-256-bound manifests and artifact verification.

No institution-specific gateway, private endpoint, provider SDK, or hosted
service is required.

## EHQ-3000

| Category | Boundary condition | Items | Subcategories |
|---|---|---:|---:|
| FEQ | Fabricated entities with no factual referent | 750 | 5 |
| PCQ | Real events after the applicable model knowledge boundary | 750 | 5 |
| HNQ | Verified, deliberately low-accessibility facts | 750 | 5 |
| CCQ | Synthetic documents with the answer-bearing span withheld | 750 | 5 |
| **Total** | **20 balanced subcategories** | **3,000** | **20** |

Every subcategory contains 150 items. PCQ and HNQ include source evidence;
PCQ additionally includes claim-level temporal-novelty records. All 1,500
source-backed items received primary human review, and a fixed stratified
20% sample received independent secondary human review. See the
[data card](docs/EHQ-3000_DATA_CARD.md) for provenance, validation scope,
limitations, and appropriate reuse.

EHQ-3000 is an evaluation set. Training on its questions, contexts, labels, or
gold answers contaminates subsequent evaluation and must be disclosed.

## What EHQ measures

Each technically valid answer is assigned one protocol label:

- `ABSTAIN`: declines to provide a purported answer;
- `HEDGE`: offers an answer while explicitly marking uncertainty;
- `CONFIDENT_CORRECT`: gives an unqualified substantive answer scored correct;
- `CONFIDENT_WRONG`: gives an unqualified substantive answer scored incorrect.

The released protocol computes:

```text
EHQ1 = proportion labelled ABSTAIN or HEDGE
EHQ2 = 1 - proportion labelled CONFIDENT_WRONG
EHQ3 = 1 - expected calibration error on substantive answers only
EHQ  = 0.30 * EHQ1 + 0.45 * EHQ2 + 0.25 * EHQ3
```

EHQ1 and EHQ2 are related behavioral summaries, not independent latent traits.
EHQ3 excludes abstentions and hedges so that restraint is not counted again as
calibration. Technical failures are exclusions, never abstentions.

## Installation

EHQ requires Python 3.10 or newer.

```bash
python -m pip install ehq
ehq --version
```

For statistical analyses and publication outputs:

```bash
python -m pip install "ehq[analysis,reporting]"
```

For repository development:

```bash
git clone https://github.com/senolali/EHQ.git
cd EHQ
python -m venv .venv
python -m pip install --upgrade pip
python -m pip install -e ".[all,dev]"
```

On Windows Command Prompt, activate with `.venv\Scripts\activate`. In
PowerShell, use `.\.venv\Scripts\Activate.ps1`.

## Five-minute offline check

A PyPI installation can create a complete, self-contained study directory:

```bash
ehq init my-ehq-study
cd my-ehq-study
ehq dry-run --config config/smoke.json --model openai-example --allow-candidate --run-id smoke-001
ehq verify-run outputs/dry-run_smoke-001
```

`ehq init` includes both the 20-item smoke fixture and the exact EHQ-3000
release. It never creates a populated `.env` and does not overwrite existing
template files unless `--force` is supplied. `dry-run` makes no network calls
and spends no API credit; its scores test the pipeline only.

To locate the dataset bundled with an installed wheel:

```bash
ehq dataset-path
```

## Configure a provider

Copy `.env.example` to `.env` and add only the credentials you use:

```dotenv
OPENAI_API_KEY=
HF_TOKEN=
OPENAI_COMPATIBLE_API_KEY=
```

`.env` is ignored by Git. Never put tokens in `config/models.json`, commands,
screenshots, issue reports, manifests, or committed files.

Model routes live in `config/models.json`. The shipped entries are examples and
are deliberately marked `unverified`. A display name is used by the CLI;
`provider_model` is the exact identifier sent to the provider.

### OpenAI

The `openai` adapter uses the official `POST /v1/responses` endpoint.

```json
{
  "name": "openai-example",
  "provider": "openai",
  "provider_model": "gpt-4.1-mini",
  "api_key_env": "OPENAI_API_KEY",
  "requires_api_key": true,
  "model_identity_policy": "record",
  "pcq_eligible": false,
  "operational_status": "unverified",
  "reasoning_mode": "disabled"
}
```

### Hugging Face Inference Providers

The `huggingface` adapter uses the OpenAI-compatible chat endpoint at
`https://router.huggingface.co/v1`. Provider selection suffixes such as
`:fastest`, `:cheapest`, or a named provider can be appended to the repository
ID when supported by Hugging Face.

```json
{
  "name": "huggingface-example",
  "provider": "huggingface",
  "provider_model": "meta-llama/Llama-3.1-8B-Instruct:fastest",
  "api_key_env": "HF_TOKEN",
  "requires_api_key": true,
  "model_identity_policy": "record",
  "pcq_eligible": false,
  "operational_status": "unverified",
  "reasoning_mode": "disabled"
}
```

### Other hosted or local APIs

Use `openai_compatible` for a standard `/chat/completions` endpoint:

```json
{
  "name": "my-route",
  "provider": "openai_compatible",
  "provider_model": "provider/model-id",
  "base_url": "https://example-provider.com/v1",
  "api_key_env": "MY_PROVIDER_API_KEY",
  "requires_api_key": true,
  "model_identity_policy": "record",
  "pcq_eligible": false,
  "operational_status": "unverified",
  "reasoning_mode": "disabled"
}
```

Common `base_url` values include:

| Service | `base_url` | Typical credential |
|---|---|---|
| OpenRouter | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` |
| Together | `https://api.together.xyz/v1` | `TOGETHER_API_KEY` |
| Groq | `https://api.groq.com/openai/v1` | `GROQ_API_KEY` |
| vLLM | `http://localhost:8000/v1` | none |
| Ollama | `http://localhost:11434/v1` | none |

For an unauthenticated local server, set `"requires_api_key": false` and omit
`api_key_env`. Base URLs and environment-variable names are recorded in the
manifest; credential values are never recorded.

## Run an evaluation

### 1. Validate the dataset

```bash
ehq validate data/releases/EHQ-3000.json \
  --expected-total 3000 \
  --expected-per-category 750 \
  --require-source-evidence \
  --require-pcq-temporal-novelty
```

### 2. Run a small real-provider pilot

```bash
ehq pilot --model openai-example --limit 20 --run-id openai-pilot-001
```

Then inspect and verify the artifacts:

```bash
ehq status pilot_outputs/real_openai-pilot-001
ehq verify-run pilot_outputs/real_openai-pilot-001
```

The terminal reports preflight status, model progress, valid denominators,
technical failures, throughput, ETA, and completion. Use `--json-events` for a
machine-readable progress stream and `--json` for a machine-readable final
result.

### 3. Run EHQ-3000

```bash
ehq full --model openai-example --run-id openai-full-001
```

For several registered routes:

```bash
ehq full \
  --models "openai-example,huggingface-example" \
  --run-id comparative-full-001
```

Full evaluation normally makes two calls per eligible item, one for the answer
and one for confidence. Estimate provider cost, rate limits, and quota before
starting. The command never selects every route implicitly.

Interrupt safely with `Ctrl+C`, then repeat the identical command with
`--resume`. A changed dataset, model definition, configuration, prompt,
selection, or protocol cannot silently reuse an incompatible checkpoint.

## Model identity and PCQ eligibility

During engineering pilots, `model_identity_policy: "record"` records a
provider-returned alias or snapshot without terminating the run. Before a
publication-grade run, use a pinned identifier where possible, confirm the
resolved identity, set `model_identity_policy: "strict"`, record the endpoint
verification fields, and set `operational_status: "verified"`.

PCQ is model-conditional. If `pcq_eligible` is false, PCQ items are excluded for
that model rather than scored under an unsupported assumption. Enable PCQ only
after recording an applicable cutoff and authoritative evidence, for example:

```json
{
  "pcq_cutoff": "2024-06-30",
  "cutoff_precision": "month",
  "cutoff_definition": "training_data_cutoff",
  "cutoff_source_url": "https://provider.example/official-model-card",
  "cutoff_source_type": "official_model_card",
  "cutoff_evidence_status": "verified_primary",
  "cutoff_verified_at": "2026-09-07",
  "pcq_eligible": true
}
```

The exact date and evidence must come from the applicable provider or model
record; do not copy the illustrative value above.

## Outputs and reproducibility

Each run contains, as applicable:

- `manifest.json`, binding inputs, model routes, software, prompts, and hashes;
- `records.jsonl`, containing item-level normalized outcomes;
- model, category, subcategory, and overall summaries;
- `summary.csv` and `summary.json`;
- optional Excel, figures, statistical analyses, and LaTeX fragments;
- `artifact_catalog.json`, containing SHA-256 and byte counts.

Run `ehq verify-run <run-directory>` before sharing or archiving results.
Generated responses, `.env`, caches, checkpoints, results, build products, and
credentials are excluded by `.gitignore` and distribution checks.

For a reproducible study, retain the git tag or wheel version, protocol version,
dataset hash, model-registry hash, resolved model identifiers, configuration,
selection hash, manifest, raw normalized records, failure/missingness counts,
and artifact catalog. See [Reproducibility](docs/REPRODUCIBILITY.md).

## Python package design

The package uses a standard `src/` layout and exposes the `ehq` console command.
Runtime dependencies are intentionally small; providers are called through
documented HTTP interfaces, so separate OpenAI or Hugging Face SDKs are not
required. Optional analysis and reporting dependencies are isolated in extras.

```text
src/ehq/
  clients/       provider adapters, pacing, retries, identity checks
  datasets/      loading, release validation, and quality audits
  evaluation/    classification, correctness, confidence, and scoring
  analysis/      correlations, comparisons, intervals, and tests
  resources/     version-pinned init templates and EHQ-3000
  artifacts.py   atomic outputs and SHA-256 catalog verification
  cache.py       successful-response-only content-addressed cache
  checkpoint.py  append-only resumable checkpoints
  config.py      experiment and model-registry validation
  provenance.py  run manifests and fingerprints
  reporting.py   machine-readable and publication outputs
  cli.py         command-line orchestration
```

See [Framework architecture](docs/framework.md) to add a provider or alter the
protocol safely.

## Citation

If you use EHQ-3000, the framework, or the EHQ protocol, please cite the
accompanying paper and the versioned software/data release. Ready-to-copy
records are provided in [CITATION.bib](CITATION.bib), and GitHub citation
rendering uses [CITATION.cff](CITATION.cff).

```bibtex
@dataset{senol2026ehq3000,
  author  = {Şenol, Ali and Bernard, H. Russell and Liu, Huan},
  title   = {EHQ-3000: A Benchmark for Epistemic Honesty at Knowledge and Context Boundaries},
  year    = {2026},
  version = {1.0},
  url     = {https://raw.githubusercontent.com/senolali/EHQ/main/data/releases/EHQ-3000.json}
}
```

When a DOI is minted through Zenodo, add it to `CITATION.cff`,
`CITATION.bib`, the data card, and the GitHub release. Do not replace the
version and checksum with an unversioned moving target.

## Development and release

```bash
python -m unittest discover -s tests -v
python -m build
python -m twine check dist/*.whl dist/*.tar.gz
python tools/check_distribution.py
```

GitHub Actions tests Python 3.10, 3.12, and 3.13. PyPI publishing is configured
for Trusted Publishing on a GitHub Release; no long-lived PyPI token is stored
in the repository. See [PyPI release instructions](docs/PYPI_RELEASE.md).

## Scope and limitations

EHQ measures observable response behavior under a specified protocol. It does
not establish consciousness, subjective knowledge, honesty as moral intent, or
general intelligence. Results depend on dataset composition, provider routing,
model version, prompting, cutoff evidence, and confidence elicitation.

## Licenses

Framework code is released under the [MIT License](LICENSE). EHQ-3000 is
released under [CC BY 4.0](DATA_LICENSE.md), subject to the stated
third-party-material notice.
