Metadata-Version: 2.4
Name: private-lab-bench
Version: 0.1.0
Summary: Local-first evaluation for scientific machine learning on private data.
Author: Dipesh Tharu Mahato
License-Expression: MIT
Project-URL: Homepage, https://github.com/dipeshbabu/private-lab-bench
Project-URL: Repository, https://github.com/dipeshbabu/private-lab-bench
Project-URL: Issues, https://github.com/dipeshbabu/private-lab-bench/issues
Project-URL: Changelog, https://github.com/dipeshbabu/private-lab-bench/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/dipeshbabu/private-lab-bench/tree/main/docs
Keywords: scientific machine learning,evaluation,private data,reproducibility,privacy auditing,benchmarking
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.21
Requires-Dist: pandas>=1.3
Requires-Dist: scikit-learn>=1.0
Requires-Dist: PyYAML>=6.0
Provides-Extra: proteingym
Requires-Dist: biopython<2,>=1.81; extra == "proteingym"
Provides-Extra: matbench
Requires-Dist: matminer<0.11,>=0.9.3; extra == "matbench"
Provides-Extra: signing
Requires-Dist: cryptography>=41.0; extra == "signing"
Provides-Extra: rdkit
Requires-Dist: rdkit>=2023.9.1; extra == "rdkit"
Provides-Extra: tdc
Requires-Dist: PyTDC==0.4.1; extra == "tdc"
Requires-Dist: numpy<2,>=1.26.4; extra == "tdc"
Requires-Dist: pandas<3,>=2.1.4; extra == "tdc"
Requires-Dist: huggingface-hub<0.26,>=0.20.3; extra == "tdc"
Requires-Dist: setuptools<81,>=77; extra == "tdc"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: jsonschema>=4.0; extra == "dev"
Requires-Dist: cryptography>=41.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
Requires-Dist: mkdocs-material<10,>=9.7; extra == "docs"
Dynamic: license-file

<h1 align="center">PrivateLabBench</h1>

<p align="center">
  Evaluate scientific models on private data, from your terminal.
</p>

<p align="center">
  <a href="https://github.com/dipeshbabu/private-lab-bench/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/dipeshbabu/private-lab-bench/ci.yml?branch=main&label=build&style=flat-square"></a>
  <a href="https://pypi.org/project/private-lab-bench/"><img alt="PyPI" src="https://img.shields.io/pypi/v/private-lab-bench?style=flat-square"></a>
  <a href="https://github.com/dipeshbabu/private-lab-bench/blob/main/pyproject.toml"><img alt="Python 3.10, 3.11, and 3.12" src="https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-blue?style=flat-square"></a>
  <a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-green?style=flat-square"></a>
</p>

<p align="center">
  <a href="docs/quickstart.md">Quickstart</a> ·
  <a href="docs/prediction_tables.md">Prediction tables</a> ·
  <a href="docs/evaluation_receipts.md">Receipts</a> ·
  <a href="CONTRIBUTING.md">Contributing</a>
</p>

---

PrivateLabBench is a local evaluation framework for scientific machine learning. Bring a prediction table, measure performance across datasets and subgroups, estimate uncertainty, and compare models on matched samples. Keep the underlying data in your environment and exchange aggregate evaluation receipts with collaborators.

Evaluation runs locally without an account, hosted service, or data upload.

## Installation

Use **Python 3.10 or later**. Create an environment for **v0.1.0**:

```sh
python -m venv .venv
```

Activate the environment and install:

```sh
# macOS / Linux
source .venv/bin/activate

# Windows PowerShell
.venv\Scripts\Activate.ps1

python -m pip install private-lab-bench==0.1.0
plb --help
```

The `privatelabbench` command is also available. Optional molecular featurization uses `python -m pip install 'private-lab-bench[rdkit]==0.1.0'`. See the [installation guide](docs/install.md) for source/development installs, or run the [quickstart notebook in Colab](https://colab.research.google.com/github/dipeshbabu/private-lab-bench/blob/v0.1.0/examples/notebooks/quickstart.ipynb).

## Run Your First Evaluation

Start with a bundled synthetic benchmark pack:

```sh
plb list-packs
plb run-pack community-tabular-regression@1.0.0
plb verify reports/benchmark_packs/tabular_regression_receipt.shareable.json
```

Packs include example predictions and versioned evaluation settings. They work from installed wheels and require no model downloads or training. Other packs cover molecular regression and protein binary classification.

## Evaluate Your Predictions

Use `prediction-table/v1` to evaluate outputs from your own model:

```csv
sample_id,target,prediction,site,batch
s01,0.12,0.14,lab-a,b1
s02,0.20,0.23,lab-a,b1
s03,0.33,0.31,lab-b,b2
s04,0.41,0.45,lab-b,b2
```

Save the table as `predictions.csv`, then run:

```sh
plb evaluate predictions.csv --slice site --slice batch
plb verify reports/evaluation/evaluation_receipt.json
```

Regression, binary classification, and multiclass probability tables are supported. Ambiguous columns or task types require explicit flags; binary probabilities with named labels require ordered negative/positive classes. Results stay local until a reviewed release policy permits export. This four-row example is too small for the default bootstrap interval or slice minimum. See the [5-minute quickstart](docs/quickstart.md) for defaults and the [prediction-table reference](docs/prediction_tables.md) for classification schemas.

## Make Runs Reproducible

Store task, input, slice, uncertainty, and reporting settings in YAML:

```sh
plb init predictions.csv --slice site --out evaluation.yaml
plb run evaluation.yaml
plb validate-config configs/tabular_uncertainty_eval.yaml
plb run configs/tabular_uncertainty_eval.yaml
```

Config runs write local JSON and Markdown reports, an artifact manifest, and evaluation receipts. Review the uncertainty settings and choose a sampling method appropriate to your data. See [statistical uncertainty](docs/uncertainty.md) for assumptions and limits.

Use `plb list-protocols` and `plb run local.yaml --protocol plb://generic/regression@1.0.0` to apply fixed, versioned evaluation rules across private datasets. Receipts record the exact protocol digest. See [evaluation protocols](docs/protocols.md) for local definitions, package providers, and release controls.

## Compare Models

Compare two prediction tables containing the same sample IDs and targets:

```sh
plb compare model-a.csv model-b.csv --target target --task-type regression --metric rmse --resamples 1000 --practical-threshold 0.01
```

Paired comparisons report metric differences, bootstrap intervals, and a decision relative to your practical threshold. Optional slice comparisons help locate differences across groups. See the [paired-comparison guide](docs/paired_comparison.md).

Combine compatible shared receipts with `plb aggregate lab-a.shareable.json lab-b.shareable.json --model model@revision`. The [aggregation guide](docs/aggregation.md) covers site identities, signer verification, and the metrics that support pooled point estimates.

Inspect cohort changes with `plb evaluate current.csv --reference reference.csv --slice site`. [Reference/current diagnostics](docs/degradation.md) report performance degradation, slice prevalence, and selected distribution changes, with explicit uncertainty and release controls.

Track the same model over repeated evaluations with [`plb trend`](docs/trends.md). It compares verified shareable receipts at declared collection times, retains uncertainty and quality decisions, and reports changing slices without needing prior raw datasets.

For repeated or nested observations, declare a [sampling design](docs/sampling_design.md) to resample independent clusters and keep paired models aligned. Optional slice-test families record dependence assumptions and multiplicity adjustments.

Check declared split integrity before scoring with `plb audit-design` or `plb evaluate --evaluation-design design.yaml`. [Evaluation-design audits](docs/evaluation_design.md) detect supported overlap/group/time violations and keep unavailable training evidence explicit.

Evaluate real Caco2 and PGP benchmark predictions with the optional [TDC ADMET integration](docs/tdc_admet.md). It preserves the pinned official test sets and metrics, checks upstream score agreement, and supports the same evaluation protocols on private ADMET tables.

The [Matbench integration](docs/matbench.md) supports composition and structure tasks with pinned official folds, units, corrected probability ROC-AUC, and verified upstream fold aggregation.

The [ProteinGym integration](docs/proteingym.md) evaluates DMS substitution scores within each assay, with tied-rank Spearman and explicit protein/function aggregation.

Describe evaluations of released models with the [external-validation profile](docs/external_validation.md): immutable model identity, declared evaluator relationship and dataset provenance, plus explicitly approved aggregate receipts that can be compared across labs.

When predictions need to be generated inside the lab, use the [trusted local scorer](docs/trusted_scoring.md) with a reviewed numeric model, pinned digest, and explicit feature units. Its optional evaluation pipeline reuses the normal protocol and release controls.

Operate a [hidden benchmark](docs/hidden_benchmarks.md) with owner-held labels, strict prediction submissions, approved result receipts, and a verified results table. Ranking requires an explicit rule in the pinned scientific protocol.

## Inspect Evaluation Receipts

Follow the [Caco2 model-selection case](docs/case_study.md) from a fresh environment to a paired decision, critical-group diagnosis, release check, and signed receipt. It distinguishes real public benchmark evidence from its synthetic measurement-shift proxy; independent researcher trials remain pending.

`evaluation-receipt/v1` separates local context from aggregate information intended for exchange:

| Artifact | Contents |
| --- | --- |
| `<run>_receipt.json` | Local receipt, including paths, configuration, and exact results |
| `<run>_receipt.shareable.json` | Approved aggregate receipt, written only when the export policy allows it |
| `<run>_receipt.md` | Human-readable receipt summary |

```sh
plb verify reports/tabular_uncertainty_demo_receipt.shareable.json
```

Shareable receipts require an explicit `release-policy/v1` policy. A blocked export leaves local results and a release audit, and removes any previous shareable file. The bundled synthetic packs include reviewed example policies; private-data runs need their own review. Use `plb release-check <local-receipt.json> --policy release.yaml` for a dry run.

Verification checks artifact integrity and optional signatures. A matching self-contained hash alone does not establish authorship, correct execution, or scientific validity. See [evaluation receipts](docs/evaluation_receipts.md) and [report integrity](docs/report_integrity.md).

For offline signer authentication, use optional Ed25519 signatures and pinned public keys:

```sh
plb verify receipt.shareable.json --trust-store trusted-labs.yaml --require-trusted-signer --json
```

See the [receipt trust guide](docs/receipt_trust.md) for signing, key rotation, protocol binding, and the limits of these checks.

## Privacy and Sharing

Keeping data local limits data movement; aggregate results can still disclose sensitive information. PrivateLabBench provides separate tools for exact reporting, empirical privacy audits, release-policy checks, and heuristic `metric_perturbation`.

Metric perturbation, including the deprecated `dp` alias, **does not provide a formal differential privacy guarantee**. Bounded-query DP reference primitives have their own sensitivity and accounting requirements. Read the [privacy guide](docs/privacy.md) before selecting a release method.

## Reference

Enforce explicit model thresholds in CI with `plb gate <receipt> --policy quality.yaml`.
See [quality gates](docs/quality_gates.md) for confidence, baseline, slice, and privacy requirements.

Use `--help` to explore commands and options:

```sh
plb eval-predictions --help
plb compare --help
plb list-tasks
plb list-privacy-attacks
```

- [Five-minute quickstart](docs/quickstart.md)
- [Core concepts](docs/concepts.md)
- [Tasks and plugins](docs/tasks.md)
- [Community benchmark packs](docs/benchmark_packs.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Project scope](docs/project_scope.md) and [roadmap](docs/roadmap.md)

## Developing

Install the development and documentation tools:

```sh
python -m pip install -e '.[dev,docs]' build twine
python -m pytest -q
python -m build
python -m twine check dist/*
mkdocs build
```

CI tests Python 3.10, 3.11, and 3.12, validates packages and documentation, and exercises the Docker runner. It also installs the wheel in a clean environment and runs a bundled pack outside the checkout. Release publishing uses the [tag-based release workflow](docs/releases.md).

Useful source entry points:

| Path | Purpose |
| --- | --- |
| `privatelabbench/cli.py` | Command-line interface |
| `privatelabbench/core/` | Task registry and extension interfaces |
| `privatelabbench/eval/` | Metrics, slices, uncertainty, and paired comparisons |
| `privatelabbench/privacy/` | Privacy audits and release policies |
| `privatelabbench/reports/` | Reports, receipts, and artifact integrity |
| `privatelabbench/data/` | Bundled benchmark packs and schemas |

## Contributing

Focused fixes, scientific-domain plugins, benchmark packs, and documentation improvements are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) for setup, extension boundaries, and validation requirements. Discuss new tasks, privacy methods, or schema changes in an issue before implementation.

The project follows its [Code of Conduct](CODE_OF_CONDUCT.md) and [governance guide](GOVERNANCE.md). Report sensitive vulnerabilities through the process in [SECURITY.md](SECURITY.md).

## License

Released under the [MIT license](LICENSE). Use [CITATION.cff](CITATION.cff) to cite PrivateLabBench in research.
