Metadata-Version: 2.4
Name: inferencebench-quality
Version: 0.1.0
Summary: LLM quality plugin for InferenceBench Suite (deterministic fixture scoring; LLM-as-judge deferred)
Project-URL: Homepage, https://github.com/yobitelcomm/bench
Author-email: Yobitel Communications <bench@yobitel.com>
License: Apache-2.0
Keywords: accuracy,ai,benchmark,llm,ml,quality
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: inferencebench-envelope
Requires-Dist: inferencebench-harness
Requires-Dist: pydantic~=2.9
Requires-Dist: pyyaml~=6.0
Description-Content-Type: text/markdown

# inferencebench-quality

Quality (accuracy) plugin for the InferenceBench Suite.

Scores model answers against bundled fixture ground-truth using deterministic
exact-match, substring-match, or token-F1 strategies. LLM-as-judge is deferred
to a later revision — this is the contract-validation skeleton that proves the
plugin abstraction is not vLLM/perf-specific.

Suite ID: `llm.quality`

Bundled benchmarks:

- `llm.quality.factual-mini` — 10 short factual questions, substring-match scoring.
- `llm.quality.reasoning-mini` — 10 GSM8K-style single-number answers, exact-match scoring.
