Metadata-Version: 2.4
Name: pinnmark
Version: 0.1.0
Summary: PINNMark inverse PDE benchmarking toolkit and CLI facade
Home-page: https://pypi.org/project/pinnmark/
Author: PINNMark Authors
Author-email: opensource@pinnmark.org
License: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy==1.26.4
Requires-Dist: pandas==2.3.3
Requires-Dist: matplotlib==3.9.4
Requires-Dist: tqdm==4.67.3
Requires-Dist: PyYAML==6.0.3
Requires-Dist: torch==2.8.0
Requires-Dist: scipy==1.13.1
Requires-Dist: tensorboard==2.20.0
Requires-Dist: psutil==7.2.2
Requires-Dist: pypdf==6.8.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# PINNMark

[![PyPI version](https://img.shields.io/pypi/v/pinnmark.svg)](https://pypi.org/project/pinnmark/)
[![Python 3.9+](https://img.shields.io/pypi/pyversions/pinnmark.svg)](https://pypi.org/project/pinnmark/)

`pinn_bench` is a governed, reviewable benchmark system for PINN and operator-ready AI4S evaluation, anchored on a frozen formal baseline, fair-budget nominal benchmarking, structured OOD benchmarking, comparison-rights methodology, and bounded benchmark recoverability.

PyPI package:

```bash
pip install pinnmark
```

Facade usage:

```python
from pinnmark import run_full_inverse_benchmark

run_full_inverse_benchmark.main(["--help"])
```

Project layers:
- `pinn_bench` core is the governed benchmark OS
- `pinn_bench.easy` is the sklearn-like researcher front door
- `PINNAtlas` is the evidence/publication product built on top of that OS

## Start Here
- I need benchmark truth / release semantics.
  Layer 1.
  Goal: understand the benchmark OS, the current canonical roots, and the governed paper-facing benchmark identity.
  First destination: `docs/product/getting_started.md`
  Second destination: `docs/product/core_story.md`, `docs/product/project_positioning.md`
- I need Python-first research workflow.
  Layer 2.
  Goal: run, inspect, compare, study, or sweep from the researcher front door without entering governance surfaces first.
  First destination: `docs/easy/index.md`
  Second destination: `docs/easy/quickstart.md`, `docs/easy/choose_guide.md`
- I need review / paper-facing benchmark package.
  Layer 1.
  Goal: inspect the current NeurIPS-facing benchmark-system package, evidence path, and reviewer route.
  First destination: `docs/product/reviewer_summary.md`
  Second destination: `paper/neurips2026/paper_reviewer_safe_summary.md`, `docs/product/paper_submission_package.md`
- I need PINNAtlas corpus / method atlas / supplement / TPAMI package.
  Layer 3.
  Goal: enter the companion evidence/publication product without confusing it with the runtime system.
  First destination: `docs/product/pinnatlas_program.md`
  Second destination: `paper/pinnatlas_supplement/README.md`, `paper/pinnatlas_tpami/README.md`

## Easy Path (Python-first researcher core)

The Python-first Easy Path provides one unified researcher core: the six easy objects ([`ModelSpec`, `TaskSpec`, `BenchmarkEstimator`, `BenchmarkSuite`, `BenchmarkReport`, `BenchmarkComparison`]) behave like a sklearn estimator graph with shared config discovery, fitting, and inspection semantics. Every `BenchmarkEstimator` still compiles down to `ExperimentConfig`, runs through the standard runner, and writes the governance artifacts (`config_snapshot.yaml`, `metrics.json`, `train_log.csv`, `model.pt`, `loss_curve.png`). The easy layer is additive: it exposes a single researcher-front door while leaving CLI commands, sectioned configs, and governance artifacts untouched.

```python
from pinn_bench.easy import BenchmarkEstimator

est = BenchmarkEstimator(
    task="heat1d",
    model="mlp",
    epochs=5,
    lr=1e-3,
    metrics=("mse", "relative_l2"),
)
est.fit(output_dir="outputs_playground/easy_nominal")
print("Score:", est.score())
print("Report:", est.report().summary())
```

Need to tweak robustness, comparison, or metrics? The Easy Path helpers clone the estimator to adjust parameters, so you keep a chainable API that still resolves to a valid config:

```python
est.with_robustness(protocol="noise", level=0.05).fit()
est.with_comparison(
    baseline_family="pinn",
    data_budget_class="sparse",
    compute_budget_class="medium",
    tuning_budget_class="fixed_recipe",
    comparison_scope="matched_all",
).score()
```

For small multi-run research workflows, stay in the same object graph with `BenchmarkSuite` and `BenchmarkComparison` instead of dropping into a separate toolchain immediately, and use the starter catalog (`all_starters()` / `describe_starter(...)`) when you need ready-made task+model+profile combos.

See `docs/easy/quickstart.md`, `docs/easy/python_api.md`, `docs/easy/which_path_should_i_use.md`, `docs/easy/compatibility_policy.md`, `docs/easy/error_guide.md`, and `docs/easy/sklearn_gap_scorecard.md` for the easy-path workflow guidance, and `examples/easy` for runnable recipes (single run, starter run, comparison, study, sweep, inspection). Use `describe_model(...)` and `describe_task(...)` when you want the docs card, starter configs, source path, and parameter signature for a component before running it, and use `describe_metric(...)` when you only need the metric source path and callable signature.

## External Model Integration

`PINNMark` now keeps a unified native + external model universe with:
- a checked-in external family registry
- adapter-backed first-wave PINN family integration
- source benchmark / state / capability / provenance metadata

Canonical-first discovery examples:

```bash
pinn-bench list families --json
pinn-bench list families --name pinnmark.pinn.vanilla --json
pinn-bench list models --external-only --source-benchmark PINNacle --json
pinn-bench list models --name PINNacle::fnn --json
```

Python equivalents:

```python
from pinn_bench.easy import all_external_models, all_families, describe_family, describe_model

print(all_families())
print(describe_family("pinnmark.pinn.vanilla"))
print(all_external_models(source_benchmark="PINNacle"))
print(describe_model("PINNacle::fnn"))  # lineage / variant lookup
```

Current integration boundary:
- some external families are runnable through thin adapters or native proxies
- some are registry-only or adapter-ready but intentionally not runnable yet
- the registry is broader than the runnable subset by design

## Canonical Family Layer

`PINNMark` now distinguishes three layers:

1. **canonical family**
   - benchmark-facing identity
   - example: `pinnmark.pinn.vanilla`
2. **implementation variant**
   - runnable or reference implementation backend
   - example: `native.mlp`, `external.pdebench.pinn_proxy`
3. **provenance / lineage**
   - source benchmark or source family reference
   - example: `PDEBench::pinn`, `PDENNEval::PINN`, `PINNacle::fnn`

This is not just aliasing. It means the platform can now say:
- these original source families collapse into one canonical `PINNMark` family
- but their provenance and fidelity differences are still preserved below that identity

See:
- `docs/models/native_family_internalization.md`
- `artifacts/benchmark_model_inventory/canonical_family_collapse_map.csv`
- `artifacts/benchmark_model_inventory/native_family_internalization_status.csv`
- `paper/neurips2026/canonical_family_wording_migration_report.md`
- `paper/neurips2026/canonical_family_authoring_style_guide.md`

Canonical-family discovery examples:

```bash
pinn-bench list families --json
pinn-bench list families --name pinnmark.pinn.vanilla --json
```

```python
from pinn_bench.easy import all_families, describe_family

print(all_families())
print(describe_family("pinnmark.operator.fno"))
```

Canonical-family wording rule:
- README, docs, paper summaries, figure titles, and table method columns should lead with canonical family ids such as `pinnmark.pinn.vanilla` and `pinnmark.operator.fno`.
- External names such as `PDEBench::pinn`, `PDENNEval::PINN`, and `PINNacle::fnn` are still kept, but as lineage / provenance notes rather than the primary benchmark-facing method names.
- If a paper or appendix surface needs both layers, use the canonical family in the main label and move the external name into a lineage note, provenance clause, or dedicated mapping column.

## Primary Method Layer

`PINNMark` now also tracks a benchmark-primary method universe above raw runtime selectors.

- top-level families remain the stable long-horizon taxonomy
- primary methods are the author-facing benchmark method objects
- runtime selectors remain the concrete constructible defaults when they exist

Examples:
- `pinnmark.pinn.cpinn`
- `pinnmark.hybrid.fbpinn`
- `pinnmark.pinn.vpinn`
- `pinnmark.pinn.hpinn`
- `pinnmark.operator.fno`
- `pinnmark.operator.deeponet`
- `pinnmark.operator.pino`
- `pinnmark.multifidelity.mf_pidnn`

Primary-method discovery examples:

```bash
pinn-bench list methods
pinn-bench list methods --name cPINN --json
```

```python
from pinn_bench.easy import all_primary_methods, describe_primary_method

print(all_primary_methods())
print(describe_primary_method("cPINN"))
```

Primary-method implementation report:
- `paper/neurips2026/twenty_method_internalization_report.md`

## Suite Path (NeurIPS 2026 first wave)

Need one suite-level front door instead of piecing together matrix / transfer / reference runners by hand?

```bash
pinn-bench suite list
pinn-bench suite run --suite inverse_main_benchmark --family-id inverse_main_benchmark__allencahn1d__parameter_recovery__epsilon_family_v1__mlp__physics_on --seeds 0
pinn-bench suite inspect --suite inverse_main_benchmark
```

Or stay in Python with the unified suite object:

```python
from pinn_bench import Suite

suite = Suite.load("small_sample_inverse_suite")
result = suite.run(dry_run=True)
summary = suite.summary()
print(summary.execution_state, summary.figure_state)
print([figure.figure_id for figure in suite.figures()])
print([claim.claim_id for claim in suite.claims()])
```

Use `paper/neurips2026/zero_to_suite_quickstart.md` for the shortest “run one suite, find the results, find the figure, find the claim” path.

## External Variant Lookup

`PINNMark` still supports external-variant lookup when authors or reviewers need to trace runnable proxies and source-benchmark lineage.

Useful discovery paths:

```bash
pinn-bench list models --json
pinn-bench list models --external-only --json
pinn-bench list models --external-only --source-benchmark PINNacle --json
pinn-bench list models --external-only --integration-state external_runnable --json
pinn-bench list models --external-only --runnable-only --json
```

Python discovery:

```python
from pinn_bench.easy import all_external_models, describe_model

print(all_external_models(source_benchmark="PINNacle"))
print(describe_model("PINNacle::fnn"))
```

Implementation anchors:
- `pinn_bench/models/external_registry.py`
- `pinn_bench/models/external_adapters/`
- `artifacts/benchmark_model_inventory/external_model_integration_status.csv`

Current boundary:
- not every external family is runnable
- runnable families are adapter-backed, not full native rewrites of the upstream benchmarks
- blocked or registry-only families are still discoverable and provenance-traceable

Current implementation report:
- `paper/neurips2026/external_model_integration_report.md`

Current machine-readable status export:
- `artifacts/benchmark_model_inventory/external_model_integration_status.csv`
- `artifacts/benchmark_model_inventory/external_model_integration_status.json`

## PINNAtlas Companion
`PINNAtlas` is not the runtime front door. It is the companion evidence/publication product layered on top of `pinn_bench`. Use it when you need field-scale corpus, audited claim, method-atlas, supplement, or TPAMI package surfaces.

A reproducible PINN benchmark toolkit with frozen formal baselines, a current six-suite execution truth layer, reviewer-facing bundle controls, and explicit formal-freeze workflows.

## Core Claim Surfaces
1. governed nominal benchmarking
2. structured OOD benchmarking
3. comparison-rights methodology
4. benchmark recoverability

## Supporting Mechanisms
- stable truth and governance
- DX and role-specific entry surfaces
- operator-ready abstraction without silent relabeling
- synthesis and packaging for external inspection

## Not This Project
- not a Main Track algorithm paper
- not a hosted leaderboard or hosted service
- not a complete geometry benchmark
- not a complete inverse-trust benchmark
- not a complete operator benchmark
- not a participant-validated recoverability paper
- not an all-AI4S breadth paper

## Fast Start
1. Inspect repo truth:
```bash
pinn-bench version --json
```
2. List the current suite front door:
```bash
pinn-bench suite list
```
3. Inspect the flagship current suite:
```bash
pinn-bench suite inspect --suite inverse_main_benchmark
```
4. Read the current execution truth:
   `paper/neurips2026/current_execution_result_audit.md`
5. Read the reviewer-safe package route:
   `paper/neurips2026/paper_reviewer_safe_summary.md`, `paper/neurips2026/submission_bundle_manifest.md`, `docs/product/paper_submission_package.md`

## Principal 50-Seed Package

For a seed-expanded principal inverse package that keeps the current 20-seed submission route intact, materialize a separate package directory:

```bash
PYTHONPATH=. python3 scripts/run_full_inverse_benchmark.py \
  --seeds 50 \
  --stress-seeds 50 \
  --flagship-seeds 50 \
  --outdir artifacts/submission_50seed \
  --figdir artifacts/submission_50seed/figures
```

This package writes:
- principal per-seed metrics, main-board summaries, sidecar/plausibility metrics, and audit summaries under `artifacts/submission_50seed/`
- bounded OOD and flagship summaries under the same package root
- LaTeX snippets under `artifacts/submission_50seed/latex/`
- a package README at `artifacts/submission_50seed/README.md`

Reviewer/user navigation assets for the current branch live at:
- `paper/neurips2026/user_playbook_quick_reference.md`
- `paper/neurips2026/adoption_checklist.md`
- `paper/neurips2026/method_family_lane_guide.md`
- `paper/neurips2026/scenario_matrix_coverage.md`

## Reader Paths
Supporting navigation:
- first-time reader: `docs/product/core_story.md`, `docs/product/why_pinn_bench.md`, `docs/product/getting_started.md`
- benchmark user: `paper/neurips2026/zero_to_suite_quickstart.md`, `paper/neurips2026/current_execution_result_audit.md`
- benchmark contributor: `docs/product/contribution_path.md`, `paper/neurips2026/execution/real_result_inventory.md`, `paper/neurips2026/execution/suite_status_board.csv`
- governance / review reader: `docs/product/reviewer_summary.md`, `paper/neurips2026/paper_reviewer_safe_summary.md`, `docs/product/paper_submission_package.md`, `paper/neurips2026/submission_bundle_manifest.md`

## Highlights
- Frozen formal benchmark lines for `v0.1`, `v0.2`, `v0.3`, `v0.4`, `v0.5`, `v0.6`, and `v1.0`
- Unified `pinn-bench` CLI for runs, matrix execution, reporting, presets, demos, scaffold flows, doctor checks, summaries, audits, formalization, and community bundle workflows
- Current 2026 paper route is anchored on one frozen fair-board layer, one current six-suite executed extension line, and one reviewer-safe truth chain
- Comparison-rights support remains bounded and reviewer-legible through current paper/package surfaces rather than preview-era launch routing
- Recoverability remains bounded and not participant-validated; no completed B1 participant outcome ledger is checked into current truth
- Reviewer-facing bundle controls, metadata routing, and claim-evidence crosswalks aligned to the NeurIPS Datasets & Benchmarks paper package
- Additive `v3.0-preview` super-benchmark registry scaffolding for strict official/staging/evidence cohort governance across absorbed SciML sources

## Install

```bash
pip install pinnmark

# local development install
pip install -e .
```

Minimum Python version: `3.9`

Core runtime dependencies:
- `numpy`
- `pandas`
- `matplotlib`
- `tqdm`
- `PyYAML`
- `torch`
- `scipy`

The PyPI-facing CLI is `pinnmark-run`. The repository-native CLI `pinn-bench` remains supported, including `python -m pinn_bench.run --config ...`.

Reproducibility note:
- benchmark outputs depend on the selected config, explicit seed, and available hardware/runtime backend
- for reproducible runs, prefer passing explicit seeds and retaining `config_snapshot.yaml`, `metrics.json`, and `train_log.csv`

## Version State

- `v0.1`: frozen historical baseline for `mlp` + `siren` on `outputs/`
- `v0.2`: frozen historical baseline for `mlp` + `siren` + `kan` on `outputs_v02/`
- `v0.3`: frozen historical baseline for the 33-family benchmark core and first product-layer formalization, on `outputs_v03/`
- `v0.4`: frozen historical baseline for the 47-family union baseline and first community-layer formalization, on `outputs_v04/`
- `v0.5`: frozen historical baseline for the thin-delta submission/community/platform release, on `outputs_v05/`
- `v0.6`: frozen historical operator-ready formal baseline, on `outputs_v06/`
- `v1.0`: current frozen formal baseline, on `outputs_v10/`
- current preview line: `none`
- latest completed preview line: `v1.9-preview`
- archived preview provenance docs: `docs/product/v1_9_preview.md`, `docs/product/v1_9_scope.md`, `docs/product/v1_9_governance.md`, `docs/product/v1_9_closeout.md`
- previous completed preview docs: `docs/product/v1_8_preview.md`, `docs/product/v1_8_scope.md`, `docs/product/v1_8_governance.md`, `docs/product/v1_8_closeout.md`
- freeze completion record: `docs/product/v1_0_freeze_complete.md`

Formal `v1.0` source-of-truth assets:
- manifest: `configs/matrix/baseline_v1_0_formal.yaml`
- canonical formal root: `outputs_v10/`

## Researcher Path

### Formal `v1.0` workflow

Use the current frozen formal manifest and formal outputs root:

```bash
pinn-bench formalize --from-preview v0.9-preview --source-root outputs_preview_v09 --target-root outputs_v10 --target-version v1.0 --dry-run
pinn-bench formalize --from-preview v0.9-preview --source-root outputs_preview_v09 --target-root outputs_v10 --target-version v1.0
pinn-bench doctor --formal --outputs-root outputs_v10 --matrix-manifest baseline_v1_0_formal.yaml

# read-only doctor verification
pinn-bench doctor --formal --outputs-root outputs_v10 --json

# explicit doctor artifact refresh
pinn-bench doctor --formal --outputs-root outputs_v10 --json --write
pinn-bench doctor --formal --outputs-root outputs_v10 --write

# read-only audit verification
pinn-bench audit --formal v1.0 --outputs-root outputs_v10 --json

# explicit audit artifact refresh
pinn-bench audit --formal v1.0 --outputs-root outputs_v10 --json --write
pinn-bench audit --formal v1.0 --outputs-root outputs_v10 --write
```

### Archived Preview Provenance

The older preview closeout package is still preserved for historical traceability.

Use these archived surfaces only when you need preview-era provenance rather than the current 2026 paper route:
- `docs/product/v1_9_preview.md`
- `docs/product/v1_9_scope.md`
- `docs/product/v1_9_governance.md`
- `docs/product/v1_9_closeout.md`
- `docs/product/public_quickstart.md`

## Community Path

Package, validate, compare, and import one official run bundle against the current formal line:

```bash
pinn-bench package-run outputs_v06/heat1d_mlp_seed0 --output-dir outputs_playground/example_bundle
pinn-bench validate-submission outputs_playground/example_bundle --against-manifest baseline_v1_0_formal.yaml
pinn-bench compare-bundle outputs_playground/example_bundle --against outputs_v10/_leaderboard/leaderboard_grouped.json
pinn-bench import-bundle outputs_playground/example_bundle --against outputs_v10/_leaderboard/leaderboard_grouped.json --output-dir outputs_playground/imported_bundle_views/example_bundle
```

Build the additive `v3.0-preview` super-benchmark registry from absorbed source packs:

```bash
pinn-bench superbench-registry --source-root examples/superbench_v3_preview/sources --output-dir /tmp/pinn_superbench_registry
```

Run one absorbed source entry through the native rerun + bundle + registry-sync path:

```bash
pinn-bench superbench-rerun-entry   --source-record examples/superbench_v3_preview/sources/PINNacle/source_record.json   --entry-id PINNacle::poisson2d::mlp::official_rerun   --workspace-root examples/superbench_v3_preview/sources/PINNacle   --registry-root /tmp/pinn_superbench_registry
```

Run one ordered SciML tranche through the same orchestration path:

```bash
pinn-bench superbench-run-tranche   --tranche-manifest examples/superbench_v3_preview/tranches/sciml_platform_tranche_01.json   --registry-root /tmp/pinn_superbench_registry
```

## Docs Hubs

- Product docs hub: `docs/product/index.md`
- Experiment docs hub: `docs/experiments/index.md`
- Community docs hub: `docs/community/index.md`
- Paper packaging hub: `docs/product/paper_submission_package.md`

## Historical Formal Releases

- `docs/experiments/v0_6_release.md`
- `docs/experiments/v0_6_runbook.md`
- `docs/experiments/v0_5_release.md`
- `docs/experiments/v0_5_runbook.md`
- `docs/experiments/v0_4_release.md`
- `docs/experiments/v0_4_runbook.md`
- `docs/experiments/v0_3_release.md`
- `docs/experiments/v0_3_runbook.md`
