Metadata-Version: 2.4
Name: enquiry-bench
Version: 0.1.0
Summary: Offline benchmark for fictional service-enquiry routing, cited drafts and abstention
Author: Tyler Smith
License-Expression: MIT
Project-URL: Homepage, https://github.com/failstocode66/enquiry-bench
Project-URL: Source, https://github.com/failstocode66/enquiry-bench
Project-URL: Changelog, https://github.com/failstocode66/enquiry-bench/blob/main/CHANGELOG.md
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# enquiry-bench

An offline benchmark for routing, cited draft replies and abstention on a small fictional
service-enquiry task. It scores prediction files you supply. It is not a customer-service agent, makes
no model or network calls, and needs no accounts or third-party packages.

**Status: 0.1.0 — released 2026-11-05 · github.com/failstocode66/enquiry-bench**
Exercised on Windows with CPython 3.12.12, 3.13.12 and 3.14.5. Other platforms are unverified.

- Tests: on 2026-11-05 the 36-test suite was run from the release source tree against each of those
  three interpreters and passed in full each time: 36 passed, with no failures, errors or skips. A
  run without skips means every link fixture was created, so the native checks ran on real links: a
  symlinked file, parent directory and output, both aliases of a hard link, and a junction given as
  input, which was rejected with `E_LINK` while its target survived. The tests of refusal
  classification and of product-failure propagation also passed.
- The 2026-11-03 validation run (CPython 3.14.5, 36 passed, no skips) used a non-elevated session
  (medium integrity, outside an AppContainer, UAC-filtered token). Nothing ran as administrator, and
  no OS, ACL, security or Developer Mode setting was changed. It shows the link checks working where
  links can be created. It does not show that the session was unrestricted, and it does not explain
  the earlier refusal below.
- Earlier revisions: a first 30-test suite passed in full, link tests included. On the 34-test
  suite that followed, 32 passed, the junction test errored and the hard-link test skipped, both at
  fixture setup because creating the link was refused (access denied), so that was not a full pass.
  A standalone check without this package was refused the same way; the cause is not established.
- Packaging: the 0.1.0 archives were built from the release source tree on 2026-11-05 with `build`
  1.6.1 and setuptools 84.2.1. The source distribution carries all 20 project files plus generated
  metadata; the wheel carries exactly the 10 runtime files plus metadata and license, and declares no
  third-party dependencies. The wheel's 10 runtime files are byte-identical to those of the wheel
  validated on 2026-11-03. Installed with `--no-index --no-deps` into a fresh environment, the exact
  new wheel ran `enquiry-bench demo` outside the source tree and produced the figures below, with
  `manifest.txt` written. The same two archives are the assets of the `v0.1.0` GitHub release, with
  their SHA-256 sums, and the files published on PyPI.

## Install

From PyPI:

```
pip install enquiry-bench
```

Or the tagged release straight from the repository:

```
pip install "git+https://github.com/failstocode66/enquiry-bench@v0.1.0"
```

Requires Python 3.12 or later; older interpreters are refused with `E_RUNTIME` rather than run
without Windows junction checks.

## Quickstart

From a clone of the repository, or after either install above, skip the first line:

```
python -m pip install --no-deps .
enquiry-bench demo --output demo-out
```

The demo scores two bundled prediction files, both synthetic: an authored reference that matches
the gold, and an always-abstain baseline generated from the case inputs. Neither is output measured
from any model. The frozen fixture is designed to give the figures below, so they show the scorer
working, not AI-model performance; confirm them in `demo-out/summary.md`.

| metric | reference | always-abstain |
|---|---|---|
| joint task pass | 48/48 | 12/48 |
| route accuracy | 48/48 | 12/48 |
| abstention precision | 24/24 | 24/48 |
| abstention recall | 24/24 | 24/24 |
| answered coverage | 24/48 | 0/48 |

## Commands

Bundled inputs live in `src/enquiry_bench/data/` and install with the package. The source
distribution also carries `tests/`, `docs/` and `CHANGELOG.md`; the wheel does not.

```
enquiry-bench validate --cases CASES --gold GOLD --knowledge KNOWLEDGE
enquiry-bench run --baseline always-abstain --cases CASES --knowledge KNOWLEDGE --output DIR
enquiry-bench score --cases CASES --gold GOLD --knowledge KNOWLEDGE --predictions PRED --output DIR
    [--repeat-predictions PRED2 ...] [--rates RATES]
enquiry-bench demo --output DIR
```

`run` never reads gold; `score` never executes file content. `--output` must be a new directory.
`score` writes `score.json` (byte-reproducible), `cases.csv`, `report.md`, `confusion.svg` and
`manifest.txt` (file hashes, written last). Errors are fixed codes that never echo input text or
paths. Python equivalents: `validate_files`, `run_baseline`, `score_files`, `demo`.

## Tests

```
python -m unittest discover -s tests
```

Needs the package installed or `src` on `PYTHONPATH`. The symlink, hard-link and junction tests
create real links. A recognized refusal to create one (link type unsupported, access denied,
privilege not held) skips that test with the reason, so its product check did not run and the skip
is not a pass. Any other error while creating a link is reported as an error, and once a link exists
its product checks run unchanged. No administrator rights, Developer Mode or permission changes are
required or made.

## What the numbers mean

The headline, **joint task pass**, needs the right route, the right abstention flag, a consistent
route/flag pair, an exact acceptable draft and the exact expected citations. Draft match is exact
comparison against a short authored list: a closed-format rubric, not a semantic judge. A reworded
correct answer fails, and a citation that exists is not proof the draft is true. Results on this
synthetic task say nothing about real-world model performance.

See [metrics and schemas](docs/metrics-and-schema.md), [synthetic provenance](docs/synthetic-provenance.md)
and the [publication post](docs/publication-post.md).

## Known gaps

- Only Windows has been exercised (CPython 3.12.12, 3.13.12 and 3.14.5); other platforms are
  unverified.
- Junction and hard-link rejection has been exercised only on Windows, in the sessions above.
  Wherever link creation is refused those tests skip, and a skip is not a pass. Why creation was
  refused where it was seen is not established.
- The tests give qualified coverage of path handling, not a proven containment or security
  guarantee, and path checks have time-of-check races.

## License

MIT. Copyright (c) 2026 Tyler Smith. The code and the invented fixtures share this license.
