Metadata-Version: 2.5
Name: diag3d
Version: 0.2.0
Summary: DIAG3D: general machine-learning diagnostics — held-out failure regions, lifecycle findings, frozen-prediction analysis and 3D evidence
Project-URL: Homepage, https://github.com/falhezaimi/DIAG3d
Project-URL: Repository, https://github.com/falhezaimi/DIAG3d
Project-URL: Issues, https://github.com/falhezaimi/DIAG3d/issues
Project-URL: Documentation, https://github.com/falhezaimi/DIAG3d/tree/main/docs
Project-URL: Changelog, https://github.com/falhezaimi/DIAG3d/blob/main/CHANGELOG.md
Author: Diag3D contributors
License-Expression: MIT
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Keywords: diagnostics,error-analysis,leakage,model-validation,slice-discovery,visualization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Python: >=3.11
Requires-Dist: numpy<3,>=1.26
Requires-Dist: plotly<8,>=6
Requires-Dist: polars<2,>=1.30
Requires-Dist: rich<16,>=13.9
Requires-Dist: scikit-learn<2,>=1.5
Requires-Dist: scipy<2,>=1.11
Requires-Dist: threadpoolctl<4,>=3.1
Requires-Dist: typer<1,>=0.15
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == 'dev'
Requires-Dist: lightgbm<5,>=4; extra == 'dev'
Requires-Dist: packaging>=24; extra == 'dev'
Requires-Dist: pygltflib<2,>=1.16; extra == 'dev'
Requires-Dist: pytest<10,>=8; extra == 'dev'
Requires-Dist: pyyaml<7,>=6; extra == 'dev'
Requires-Dist: ruff<1,>=0.11; extra == 'dev'
Requires-Dist: twine<8,>=6; extra == 'dev'
Provides-Extra: export
Requires-Dist: kaleido<2,>=1; extra == 'export'
Provides-Extra: gallery
Requires-Dist: matplotlib<4,>=3.8; extra == 'gallery'
Requires-Dist: pillow<13,>=10; extra == 'gallery'
Provides-Extra: glb
Requires-Dist: pillow<13,>=10.1; extra == 'glb'
Provides-Extra: lightgbm
Requires-Dist: lightgbm<5,>=4; extra == 'lightgbm'
Provides-Extra: test
Requires-Dist: lightgbm<5,>=4; extra == 'test'
Requires-Dist: packaging>=24; extra == 'test'
Requires-Dist: pygltflib<2,>=1.16; extra == 'test'
Requires-Dist: pytest<10,>=8; extra == 'test'
Requires-Dist: pyyaml<7,>=6; extra == 'test'
Provides-Extra: xgboost
Requires-Dist: xgboost<4,>=2; extra == 'xgboost'
Description-Content-Type: text/markdown

# Diag3D — machine-learning behavior, made spatial

Diag3D transforms machine-learning datasets and model outputs into interactive 3D environments for understanding data before training and interrogating prediction, error, failure, and repair after training.

**DATA → MODEL → PREDICTION → ERROR → FAILURE → REPAIR**

One viewport retains camera orientation, selected observations or regions, and filters as you move through the model. The existing v0.2 research engine supplies advanced failure discovery, null calibration, full-refit stability and nested repair validation. The evidence report remains available separately.

**Version 0.2.0 is the first public release: `pip install diag3d`.** This is research software; visual patterns and diagnostic explanations do not establish causality or universal statistical error control.

![Exact failure geometry and its scientific inspector](docs/model-space/05-failure.png)

## Enter a dataset or results table

```bash
# Install (Python 3.11+).
pip install diag3d

# Pre-run: schema in the terminal, then the interactive environment.
diag3d inspect data.csv
diag3d report data.csv --target y --open

# Post-run: first-class CSV workflow, no serialized model required.
diag3d report results.csv --target y --prediction y_pred --open

# Classification: specify the task and optional probability mapping explicitly.
diag3d report classes.csv --target class --prediction predicted_class \
  --task classification --probabilities '{"0":"p0","1":"p1"}' --open

# Existing manual visualization and static report remain compatible.
diag3d visualize data.csv --x feature_1 --y feature_2 --z feature_3
diag3d report data.csv --target y --legacy --open
```

`report` now opens the model-space explorer by default. `--model` still invokes the existing held-out baseline report. `Dataset.report()` retains its established API; `Dataset.explore()` and `diag3d.explore()` produce the new environment. `analyze` now writes `explorer.html` alongside the original research `report.html`, JSON, and OOF CSV.

The new `report` default loads **all rows**. `--max-points 12000` caps only rendering; exact descriptive metrics and field aggregation use every loaded row. `--sample N` explicitly caps analyzed rows and is disclosed. Large HTML payloads still require memory proportional to the loaded dataset.

```python
from diag3d import explore, ModelSpace, SklearnAdapter

space = explore("results.csv", target="y", prediction="y_pred")
space.save("explorer.html")

# model is already fitted in the caller's Python process.
adapter = SklearnAdapter(model, feature_names=["feature_1", "feature_2"])
ModelSpace.from_adapter(dataframe, adapter, target="y").save("model.html")

# Existing research result plus matching original inputs.
ModelSpace.from_research(result, X, y).save("research-model-space.html")
```

CSV-only fields are **piecewise constant means of observed predictions**, with other features pooled. They are not invented queries to an unknown model. Direct adapters can generate support-masked, selected-feature model slices with other inputs fixed at medians. Neither representation reconstructs an arbitrary high-dimensional model.

## Portable 3D files, without HTML

Export existing predictions as a self-contained `.glb` scene that researchers can
open, rotate, and zoom in a compatible 3D viewer. No model is fitted during export.

```bash
pip install "diag3d[glb]"
diag3d export-glb results.csv --x NDMI --y LST --z cover \
  --target richness --prediction y_pred --color error --output error.glb
```

In Python, use `space.export_glb("error.glb", color="error")` or
`space.save("observations.glb")`. Scenes contain colored observations, labeled axes,
a legend, and embedded original coordinates and row references. They are snapshots;
filters and model switching remain in the explorer. Axes are independently normalized,
and the default display cap is 12,000 observations. See the [GLB guide](docs/glb_export.md)
for model selection, observation IDs, sampling, and interpretation.

## Demonstrations

```bash
PYTHONPATH=src python examples/model_space_demos.py
# Open gallery/model-space/regression.html and press “28s walkthrough”.
```

The three offline demonstrations cover the existing interaction regression benchmark, held-out nonlinear class geometry, and real wine chemistry before training. See [sources and licenses](examples/MODEL_SPACE_DATASETS.md). Generation reruns the primary 199-null-search / 50-refit / nested-repair experiment; `--quick` explicitly uses smaller development budgets.

Read the [implementation status](IMPLEMENTATION_STATUS.md), [model-space architecture](MODEL_SPACE_ARCHITECTURE.md), [UX specification](UX_SPEC.md), [validation](VALIDATION_RESULTS.md), [audit](SOURCE_AUDIT.md), and [before/after comparison](BEFORE_AFTER.md).

## Null-validation milestone

The original interaction result is preserved: **220/221 planted observations**, IoU **0.995**, error lift **4.49×**. It exceeds all **199** full-search residual-permutation null maxima (empirical p **0.005**, with plug-in-null assumptions). Full-refit bootstrap detects the geometry in **50/50** runs. A repair chosen entirely on training-side validation reduces untouched outer-test MAE from **0.856 to 0.099**.

The negative results matter: the initial 99-replicate null banks accepted **25/300 negative-control datasets**, and **8/50** in the 50-feature stress test. We retain these results and the 499-replicate sensitivity extension. The larger frozen-bank 50-feature experiment accepted **30/500 (6.0%)**, with a conditional-bank 95% interval of **4.2%–8.4%**. This does not establish universal 5% error control. Read [null calibration](NULL_CALIBRATION_RESULTS.md), [independent refits](INDEPENDENT_BOOTSTRAP_RESULTS.md), [nested repairs](NESTED_REPAIR_RESULTS.md), and [visual evidence](VISUAL_EVIDENCE_REPORT.md).

![Diag3D method](docs/figures/concept.svg)

## Lifecycle diagnostics: before, during and after training

Enter at any stage; each works on its own (details: [docs/lifecycle.md](docs/lifecycle.md)).

* **Preflight** — data quality and split design, no model training:
  `diag3d preflight data.csv --id sample_id --target y --role split --group site --objective held_out_group`
* **Training integration** — explicit calls from your own loop (`TrainingMonitor.log_metrics`,
  `record_predictions`, a native LightGBM callback); the monitor never owns the loop, folds or seeds.
* **Post-hoc only** — frozen predictions (CSV/Parquet/NPZ; pickle only when explicitly trusted),
  no fitting: `diag3d posthoc preds.csv --data data.csv --id sample_id --target y --map y_pred=pred --default model=rf,role=oof --split-design kfold`
  (checks that need to know which samples trained each fold run only when the split design is declared or an explicit split manifest is given).

A `Run` directory (versioned manifest, append-only event log, Parquet predictions, findings)
connects the stages and can be reopened to reproduce the diagnostics. Every check yields a finding
with evidence, counts, thresholds and a `pass`/`warn`/`fail`/`not_evaluable` status; missing
metadata is never reported as a pass. Runnable examples: `examples/lifecycle_1_preflight.py`,
`examples/lifecycle_2_training.py`, `examples/lifecycle_3_posthoc.py`.

## Installation

Python 3.11 or newer (CI covers 3.11–3.13 on Linux and 3.12 on Windows).

```bash
pip install diag3d
diag3d --version
```

Optional extras, installed only when you need them:

```bash
pip install "diag3d[xgboost]"    # XGBoost baseline
pip install "diag3d[lightgbm]"   # LightGBM baseline and training callback
pip install "diag3d[gallery]"    # Matplotlib scientific figures
pip install "diag3d[glb]"        # text labels in .glb scenes (Pillow)
pip install "diag3d[export]"     # static Plotly export (Kaleido)
```

macOS users can install the command-line tool through the project's Homebrew tap:

```bash
brew tap falhezaimi/tap
brew install diag3d
```

`import diag3d` loads only the core dependencies (NumPy, Polars, SciPy, scikit-learn,
Typer, Rich); Plotly, Matplotlib, XGBoost and LightGBM are imported by the functions
that use them. The supported API is listed in [docs/public_api.md](docs/public_api.md).

### Contributors

```bash
git clone https://github.com/falhezaimi/DIAG3d.git
cd DIAG3d
python -m venv .venv
source .venv/bin/activate  # PowerShell: .venv\Scripts\Activate.ps1
pip install -e ".[dev,gallery]"
diag3d --help
```

Use `python -m diag3d` if the console command is not on PATH. See the
[dependency policy](docs/dependencies.md) and [releasing.md](docs/releasing.md).

```python
import diag3d
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge

print(diag3d.__version__)
X, y = make_regression(n_samples=100, n_features=2, random_state=42)
analyzer = diag3d.FailureAnalyzer(
    models={"linear": Ridge()}, cv=3,
    config=diag3d.AnalysisConfig(seed=42, bootstrap_replicates=0),
)
result = analyzer.fit_analyze(X, y)
result.save("results")
```

Research outputs record Diag3D/Python/dependency versions, seeds, model parameters,
configuration and validation metadata. Save `python -m pip freeze` with experiments
for exact environment reconstruction. Maintainers: follow [docs/releasing.md](docs/releasing.md).

## Research CLI

```bash
diag3d analyze data.csv --target richness --models rf hgb --cv 5 \
  --discover-failures --bootstrap 30 --null-replicates 199 \
  --refit-bootstrap 50 --alpha 0.05 --output results/

# Group identity is excluded from predictors, and groups never cross train/test.
diag3d analyze data.csv --target richness --models rf,hgb \
  --group site --cv 5 --seed 42 --max-feature-pairs 15 \
  --min-region-support 25 --repair --output results/
```

Outputs: `analysis.json`, observation/fold/model-aligned `oof_predictions.csv`, and an offline `report.html` with model selection, support/disagreement/persistence overlays, exact cell boundaries and region inspection. A report can embed observation indices and diagnostic values; review it before sharing.

Research analysis uses numeric predictors and a finite continuous target. Nonnumeric predictors are listed as omitted. Supply `--ignore record_id,post_outcome_measurement` for identifiers or inappropriate predictors; no algorithm can determine whether an available feature was actually known at prediction time. Missing predictor values are imputed using training folds only. The legacy commands retain mixed-type descriptive CSV analysis.

## Python API

```python
from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor
from diag3d import FailureAnalyzer, AnalysisConfig

models = {
    'rf': RandomForestRegressor(n_estimators=100, min_samples_leaf=3),
    'hgb': HistGradientBoostingRegressor(max_iter=100),
}
analyzer = FailureAnalyzer(models=models, cv=5, random_state=42)
result = analyzer.fit_analyze(X, y, groups=site_ids)  # omit groups for random CV
for region in result.regions:
    print(region.region_id, region.metrics.error_lift, region.characterization)
figure = result.visualize()       # Plotly Figure; call .show() in an appropriate environment
result.report('report.html')
result.save('results')
```

Pass `RepeatedKFold(...)`, `GroupKFold(...)`, or audited train/test index pairs as `cv`. Identical folds are reused across all models. When groups are supplied, any custom split with group overlap is rejected. Explicit partial holdout requires `AnalysisConfig(allow_partial_validation=True)`; untested observations never become residual evidence.

Advanced controls belong in `AnalysisConfig`: uniform or quantile grids, support floors, threshold sweep, bootstrap count, feature/pair caps, score weights and characterization rules. A supplied config controls its own seed and job count. Pipelines and arbitrary compatible regression estimators can be passed as named models. Models are cloned, never fitted in place.

## Calibrate, refit, and validate repairs

```python
from diag3d import CalibrationConfig, NestedRepairConfig, RepairGuardrail

# Start from an existing result from fit_analyze(X, y).
analyzer.calibrate(X, y, result=result,
    config=CalibrationConfig(replicates=199, strategy="residual_permutation"))
analyzer.refit_bootstrap(X, y, result=result, replicates=50)
result.nested_validation = analyzer.validate_repairs(X, y,
    config=NestedRepairConfig(guardrail=RepairGuardrail(
        minimum_regional_improvement=0.05,
        outside_tolerance=0.02, global_tolerance=0.02)))
result.report("research-evidence.html")
```

Default residual permutation assumes an adequate mean and exchangeable homogeneous residuals; its estimated reference makes it a plug-in diagnostic. Grouped data requires a domain-appropriate null generator through `run_null_bank`, rather than row permutation. Full-refit bootstrap currently supports integer CV. The original conditional bootstrap remains labeled and available.

## What is implemented

- Typed prediction, field, region, persistence, stability, support, characterization, ranking and repair results.
- KFold, RepeatedKFold and GroupKFold OOF prediction; train-only imputation/scaling and neighbor-support analysis.
- Supported 1D/2D empirical MAE grids, face-connected components, threshold lineages and interpretable cell-union geometry.
- Conditional row/group bootstrap plus identity-isolated full model refits; fold rediscovery; cross-model error and prediction disagreement.
- Complete-search max-statistic null calibration, six negative controls, multiplicity stress tests and detection-rate experiments.
- Nested training-side repair selection with explicit guardrails and untouched outer-test evaluation.
- Twelve requested scientific figures, three publication composites, and additional calibration/bootstrap comparisons in PNG, PDF and SVG.
- Seven transparent diagnostic categories, exposed evidence-score components and nested held-out repair experiments.
- Six deterministic planted-mechanism benchmarks, grid/seed sensitivity and a null-data audit.
- Offline evidence explorer and research report, plus preserved v0.1 diagnostics/visualization commands.

`inspect`, `report`, `visualize`, `fit`, `diagnose`, `compare`, `Dataset`, `Config`, `inspect_csv`, `report_csv`, and `visualize3d` remain available. The legacy `fit` command uses a single holdout; `analyze` is the research OOF path. See [legacy usage](docs/legacy/README-v0.1.md).

## Reproduce the evidence

```bash
python -m pytest -q
python experiments/run_benchmarks.py --output benchmark_outputs --bootstrap 20 --repairs
python experiments/sensitivity.py --output sensitivity_results.json
python experiments/null_validation.py --output results
python experiments/calibration_sensitivity.py --results results --replicates 499 --workers 4
python experiments/search_precision.py --results results --datasets 500 --workers 4
python experiments/plugin_null_pilot.py --results results
python experiments/make_figures.py --results results
python experiments/make_evidence_report.py --results results
python -m build
python -m twine check dist/*
```

See [actual benchmark results](BENCHMARK_RESULTS.md), [method](docs/research_method.md), [architecture](docs/architecture.md), [limitations](docs/limitations.md), and [prior-art questions](docs/prior_art_and_novelty_questions.md).

## Scientific limits

OOF prediction avoids training residuals. It does **not** remove bias from searching and evaluating slices on the same OOF errors. Conditional bootstrap recurrence is not model-refit uncertainty or statistical significance. Full-refit recurrence is also not a confidence region. The max-statistic calibration depends on the specified null; observed false acceptance and Monte Carlo sensitivity remain documented. Shared error does not distinguish irreducible noise from misspecification. Candidate categories and scores prioritize investigation; independent confirmation is required.
