Metadata-Version: 2.4
Name: inferencebench-code
Version: 0.0.2
Summary: Code-generation plugin for InferenceBench Suite (HumanEval-style execution-based scoring).
Project-URL: Homepage, https://github.com/yobitelcomm/bench
Author-email: Yobitel Communications <bench@yobitel.com>
License: Apache-2.0
Keywords: ai,benchmark,code-generation,humaneval,llm,ml
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: inferencebench-envelope
Requires-Dist: inferencebench-harness
Requires-Dist: pydantic~=2.9
Requires-Dist: pyyaml~=6.0
Description-Content-Type: text/markdown

# inferencebench-code

Code-generation plugin for the InferenceBench Suite.

HumanEval-style execution-based benchmarks: the plugin sends a function-signature
prompt to the model, extracts the Python code from its response, executes it
against bundled unit tests in a subprocess, and reports `pass_at_1`.

Suite ID: `code.generation`

Bundled benchmarks:

- `code.generation.humaneval-mini` — 5 stdlib-only Python tasks, `pass_at_1`
  scoring with a 5-second per-task wall-clock timeout.

## SAFETY WARNING — read before running

**This plugin executes model-generated code.** Every run prints a yellow banner
reminding you of that. The execution layer is *best-effort* defence-in-depth,
not a real sandbox:

- Each task's solution + tests are written to a temp file and invoked with
  `python -I` (isolated mode) under a `subprocess.run(timeout=...)` wall clock.
- A cheap substring pre-scan refuses any solution that imports `subprocess`,
  `os.system`, `socket`, `urllib`, `multiprocessing`, or `ctypes`.
- The bundled fixtures are stdlib-only, no I/O, no network.

This is **deliberately not airtight**. Phase 2 adds real isolation (firejail /
nsjail / container-per-task). Until then: only run code-generation benchmarks
against models you trust, on machines you can afford to throw away, and never
against the bundled fixtures replaced with untrusted input.

## Metrics

The envelope's `metrics` block includes:

| Metric             | Direction       | Meaning                                   |
| ------------------ | --------------- | ----------------------------------------- |
| `pass_at_1`        | higher is better | mean of per-task passed booleans          |
| `pass_at_1_p05/50/95` | higher is better | bootstrap quantiles of per-sample scores |
| `timeout_rate`     | lower is better  | fraction of tasks that hit the wall clock |
| `ttft_p50_ms`      | -               | model time-to-first-token, median         |
| `total_p50_ms`     | -               | model total request time, median          |
| `tokens_out_total` | -               | total generated tokens across the run     |
| `ok_rate`          | -               | fraction of model calls that succeeded    |
| `n_samples`        | -               | fixture row count                         |
