Metadata-Version: 2.5
Name: caddforge-ml
Version: 0.4.0
Summary: Machine learning workflows for the CADDForge toolbox.
Author-email: Manuel Sellner <manuel.sellner@idorsia.com>, Christoph Sager <christoph.sager@idorsia.com>
License: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: caddforge-core>=0.21.0
Requires-Dist: numpy>=1.25
Requires-Dist: pandas>=1.5
Requires-Dist: rdkit>=2022.9.5
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.10
Requires-Dist: xgboost>=1.7
Provides-Extra: ui
Requires-Dist: plotly>=5.15; extra == 'ui'
Requires-Dist: streamlit>=1.29; extra == 'ui'
Description-Content-Type: text/markdown

# CADDForge ML Tool

Install this plugin with `python -m pip install caddforge-ml`. It depends directly
on `caddforge-core>=0.21.0`; Python imports and CLI commands still use `caddforge`.
For the complete public toolbox, including user-facing extras, install
`python -m pip install caddforge`. When migrating an existing environment, uninstall
the former `caddforge` core distribution before installing renamed core and updated plugins.

Machine learning workflows for the CADDForge toolbox. The initial tool provides a **simple Gaussian Process regressor** (Tanimoto or RBF kernel) with one-file training + inference for rapid prototyping. Additional tools include PCA and a scikit-learn model zoo.

## Repo Layout
- `src/cadd_ml/simple_gp/`: core GP primitives and workflow.
- `src/cadd_ml/pca/`: PCA workflow.
- `src/cadd_ml/model_zoo/`: scikit-learn model zoo workflow.
- `tests/`: unit/integration tests.
- `src/cadd_ml/model_factory/`: orchestrates multiple model-zoo runs in one go.
- `src/cadd_ml/explore_classifier/`: interactive threshold exploration for classifiers.

## Tools at a Glance
- **simple-gp** (regression): lightweight GP with optional PCA preprocessing, log-targets, standardization, and model serialization for predict-only runs.
- **pca** (preprocessing): standalone PCA with standardized feature handling and explained-variance reporting.
- **model-zoo** (regression/classification): unified runner for scikit-learn models (ridge, lasso, elasticnet, bayesian_ridge, logistic_regression, random_forest, extra_trees, svm, knn, gaussian_process, xgboost) with log/standardize targets, time split, and predict-only mode.
- **model-factory** (regression/classification): runs multiple model-zoo models, aggregates metrics/uncertainty correlations/confusion matrices, and pools artifacts under the main run.
- **explore-classifier** (classification UI): Streamlit UI to tweak decision thresholds, view live metrics/curves, plot metrics vs threshold, and save thresholds into model artifacts for downstream prediction runs.

## Simple Gaussian Process Tool (regression)

With `caddforge-core` available (or `PYTHONPATH` pointing to `src/`), the simple-gp tool is exposed via the `caddforge.tools` entry point:

```bash
# List tools
caddforge list-tools

# Show template config
caddforge show-config ml simple-gp

# Train + predict
cat > gp.yaml <<'YAML'
train_path: train.csv      # CSV with columns f1,f2,...,y
target_column: y
feature_columns: [f1, f2, f3]   # optional; defaults to all non-target columns
smiles_column: smiles           # optional; auto-generates fingerprints
ignore_columns: [id, comment]   # optional; excluded from auto feature detection
fingerprint_types: [morgan, atompair, featmorgan]  # default set
kernel: tanimoto            # or rbf
noise: 0.05
# use_pca: true             # optional: reduce feature dimensions, requires kernel: rbf
# pca_n_components: 128     # optional: selects PCA output dims (default=min(n_samples, n_features))
test_path: test.csv         # optional; defaults to training data
save_report: true           # write JSON with train/test metrics
log_targets: false          # fit in log-space (requires positive targets)
generate_scatter_plot: true # render scatter if csv-scatter is available
scatter_filename: gp_scatter.html
drop_non_positive_targets: true  # when log_targets is true, drop non-positive rows instead of erroring
mode: train                # train (fit + predict) or predict (load serialized model)
model_path: simple_gp_model.joblib  # where to save/load the model bundle
id_column: id              # optional; carries IDs into predictions
YAML

caddforge run ml simple-gp --config gp.yaml
```

Outputs land under `runs/<date>/<job-id>/` with `results/predictions.csv`, `results/results.json` (if enabled), and `results/result_record.json`.
When `log_targets` is enabled, `gp_variance` is reported in model (log) space for interpretability; a linear-space variance is additionally stored as `gp_variance_linear`. Predictions are standardized to include `y_pred`, `y_std` (aliases for gp columns), and optional `y_true`/`id`/`split`.

If `caddforge-datascience` (csv-scatter) is installed and the target column is present in the predictions, the workflow also renders an optional plot (`gp_scatter.html`) showing `target_column` vs `gp_mean` (color-coded by observation std in linear space when `log_targets` is true, otherwise observation std in model space), with SMILES depictions if a `smiles_column` is provided).

### Notes on log targets and variances
- `log_targets: true` fits the GP on `log(target)`; non-positive targets are dropped by default (`drop_non_positive_targets`).
- `gp_variance` is always the model-space variance (log-space when `log_targets` is true). A linear-space variance derived from log-normal moments is emitted as `gp_variance_linear`, along with standard deviations for both latent and observation scales.
- Scatter plots switch both axes to log scale when `log_targets` is enabled; color uses observation std.
- When `use_pca: true` the feature matrix is centered/scaled (configurable) and reduced via PCA before fitting; RBF kernel is required and enforced.
- `mode: predict` loads a serialized model bundle (including PCA/standardization) from `model_path` and skips refitting; `mode: train` fits and saves the bundle.
- When a `smiles_column` is provided, SMILES are carried into `predictions.csv`.

### Python API
```python
from cadd_ml.simple_gp.workflow import SimpleGPConfig, SimpleGPWorkflow

cfg = SimpleGPConfig(train_path="train.csv", target_column="y", test_path="test.csv")
wf = SimpleGPWorkflow(cfg)
job = wf.prepare_job()
result = wf.run(job)
print(result.metrics)
```

## PCA Tool

A lightweight PCA workflow (powered by scikit-learn) is available for exploratory analysis or as a preprocessing step for other tools.

```bash
cat > pca.yaml <<'YAML'
data_path: features.csv          # CSV with feature columns
# feature_columns: [f1, f2, f3]  # optional; defaults to all except ignore_columns
ignore_columns: [id]             # optional
n_components: 5                  # optional; defaults to min(n_samples, n_features)
standardize: true                # center + scale to unit variance
drop_missing_rows: true          # drop rows with missing feature values
scores_filename: pca_scores.csv
components_filename: pca_components.csv
results_filename: results.json
YAML

caddforge run ml pca --config pca.yaml
```

Outputs (under `results/`):
- `pca_scores.csv`: per-row principal component scores (PC1, PC2, ...).
- `pca_components.csv`: component loadings.
- `results.json`: metadata + explained variance ratios.

### Python API
```python
from cadd_ml.pca.workflow import PCAConfig, PCAWorkflow

cfg = PCAConfig(data_path="features.csv", n_components=5)
wf = PCAWorkflow(cfg)
job = wf.prepare_job()
result = wf.run(job)
print(result.metrics)
```

## Model Zoo Tool (beta)

A general-purpose scikit-learn model zoo supporting regression/classification with a unified output format.

```bash
cat > model_zoo.yaml <<'YAML'
train_path: train.csv
target_column: y
feature_columns: [f1, f2]
model_name: random_forest   # ridge, lasso, elasticnet, bayesian_ridge, logistic_regression, random_forest, extra_trees, svm, knn, gaussian_process, xgboost
task: regression            # or classification
test_size: 0.2              # used if no test_path provided
random_state: 123
classification_thresholds: null  # optional: bin float targets into classes for classification
time_column: null                # optional: chronological split when no test_path
standardize_targets: false       # regression only: standardize y then invert outputs
target_transform: none           # regression only: none, log, log1p (applied before standardize_targets)
log_targets: false               # deprecated; use target_transform: log
drop_non_positive_targets: true  # drop invalid targets for log/log1p when enabled
positive_class: 1                # optional for classification; defaults to 1 or max class
predict_chunk_rows: null         # optional chunk size for predict mode to reduce memory
predictions_filename: predictions.csv
results_filename: results.json
metrics_filename: metrics.json
mode: train                # train or predict (predict uses model_path only)
model_path: artifacts/model.joblib
YAML

caddforge run ml model-zoo --config model_zoo.yaml
```

Outputs (under `results/`):
- `predictions.csv`: includes `id`, `split`, `y_pred`, optional `y_true`, `smiles` when present, and `proba_*` or `y_std` when available. For thresholded classification, `y_true_class` is added.
- `metrics.json`: task-specific metrics.
- `results.json`: summary metadata.
- Model artifact stored under `artifacts/model.joblib` (includes `positive_class` and any saved decision threshold from the explorer).
- Large predict-only runs can stream in chunks via `predict_chunk_rows` to reduce memory; progress is logged per chunk.

Supported models and tasks:
- Regression only: `ridge`, `lasso`, `elasticnet`, `bayesian_ridge`, `gaussian_process`
- Classification only: `logistic_regression`
- Both regression and classification: `random_forest`, `extra_trees`, `svm`, `knn`, `xgboost`

### Python API
```python
from cadd_ml.model_zoo.workflow import ModelZooConfig, ModelZooWorkflow

cfg = ModelZooConfig(train_path="train.csv", target_column="y", model_name="random_forest")
wf = ModelZooWorkflow(cfg)
job = wf.prepare_job()
result = wf.run(job)
print(result.metrics)
```

## Model Factory Tool (beta)

Runs multiple model-zoo models against the same training/evaluation data and aggregates their
metrics, uncertainty diagnostics, confusion summaries, and serialized model artifacts. The
factory is a training workflow; predict-only operation remains a `model-zoo` feature.

```bash
cat > model_factory.yaml <<'YAML'
train_path: train.csv
target_column: y
task: regression
feature_columns: [f1, f2]
ignore_columns: [compound_id, measured_at]
test_path: null
smiles_column: smiles
fingerprint_types: [morgan, atompair, featmorgan]
fp_bits: 2048
fp_radius: 2
fp_use_chirality: true
model_names: [ridge, random_forest, xgboost]
model_params:
  random_forest:
    n_estimators: 500
    random_state: 123
  xgboost:
    max_depth: 6
calibrate: false
calibrate_method: sigmoid
test_size: 0.2
random_state: 123
predictions_filename: predictions.csv
results_filename: results.json
metrics_filename: metrics.json
id_column: compound_id
classification_thresholds: null
time_column: measured_at
standardize_targets: false
positive_class: null
target_transform: none
drop_non_positive_targets: true
# log_targets: false  # deprecated alias for target_transform: log
YAML

caddforge run ml model-factory --config model_factory.yaml
```

Set `task` explicitly when possible. With an explicit task and no `model_names`, the factory
selects all registry models compatible with that task. If both are omitted, it attempts every
registered model and lets each subrun infer the task, which can include incompatible models.

### Configuration options

The following are all model-factory-specific options. Common CADDForge settings such as
`backend`, `run_directory`, `registry_path`, `job_name`, result export, and Slurm settings are
inherited from `BaseConfig`.

| Option | Type and default | Description |
| --- | --- | --- |
| `train_path` | path, required | Training CSV. It must contain `target_column` and all configured feature, SMILES, ID, and time columns used for training. |
| `target_column` | string, required | Column to predict. Numeric values are required for regression and for threshold-binned classification. |
| `task` | `regression`, `classification`, or `null`; default `null` | Task passed to every model-zoo subrun. When omitted, each subrun infers classification for thresholded or non-numeric targets and regression otherwise. Explicitly setting it also lets the factory choose only compatible default models. |
| `feature_columns` | list of strings or `null`; default `null` | Numeric tabular features. When omitted, all training columns except the target, SMILES column, and `ignore_columns` are used. These features are concatenated with fingerprints when `smiles_column` is also set. |
| `ignore_columns` | list of strings; default `[]` | Columns excluded only from automatic feature discovery. It has no effect when `feature_columns` is explicit. |
| `test_path` | path or `null`; default `null` | Optional external evaluation CSV. When set, all training rows are fitted and this file is predicted. Otherwise a holdout is taken from `train_path`. |
| `smiles_column` | string or `null`; default `null` | SMILES column used to generate RDKit fingerprints. Fingerprints may be used alone or appended to numeric `feature_columns`. |
| `fingerprint_types` | list of strings; default `[morgan, atompair, featmorgan]` | Fingerprints to concatenate for each SMILES. Supported values are `morgan`, `atompair`, and `featmorgan`. Used only with `smiles_column`. |
| `fp_bits` | integer; default `2048` | Bit count generated for each fingerprint type. The total fingerprint width is this value times the number of fingerprint types. |
| `fp_radius` | integer; default `2` | Radius for Morgan and feature-Morgan fingerprints. |
| `fp_use_chirality` | boolean; default `true` | Include chirality in generated fingerprints. |
| `model_names` | list of strings or `null`; default `null` | Models to run. Accepted names are `ridge`, `lasso`, `elasticnet`, `bayesian_ridge`, `logistic_regression`, `random_forest`, `extra_trees`, `svm`, `knn`, `gaussian_process`, and `xgboost`. With an explicit `task`, `null` selects all compatible models. |
| `model_params` | mapping; default `{}` | Per-model estimator keyword arguments keyed by model name, for example `random_forest: {n_estimators: 500}`. Values are passed directly to the underlying scikit-learn or XGBoost constructor. |
| `calibrate` | boolean; default `false` | For classification, wrap each estimator in three-fold `CalibratedClassifierCV` before prediction. Ignored for regression. |
| `calibrate_method` | `sigmoid` or `isotonic`; default `sigmoid` | Probability calibration method used when `calibrate` is true. |
| `test_size` | float from `0.0` to `0.9`; default `0.2` | Fraction reserved for evaluation when `test_path` is absent; use a positive value when a holdout is required. A chronological holdout is used when `time_column` is set; otherwise the split is random and classification is stratified when possible. |
| `random_state` | integer; default `123` | Seed for the random holdout split. It is not automatically injected into estimators; set estimator seeds in `model_params` when needed. |
| `predictions_filename` | string; default `predictions.csv` | Predictions filename inside each model subrun's `results/` directory. |
| `results_filename` | string; default `results.json` | Model-zoo summary filename inside each model subrun's `results/` directory. |
| `metrics_filename` | string; default `metrics.json` | Metrics filename inside each model subrun's `results/` directory. |
| `id_column` | string or `null`; default `null` | Source column copied into per-model predictions as `id`. Row indices are used when omitted. |
| `classification_thresholds` | list of floats or `null`; default `null` | Sorted cut points used with classification to bin a continuous target via `numpy.digitize`; N thresholds create N+1 integer classes. A factory confusion summary is emitted only for binary outcomes. |
| `time_column` | string or `null`; default `null` | Datetime-like column used for a chronological split when `test_path` is absent. The newest `test_size` fraction becomes the holdout. |
| `standardize_targets` | boolean; default `false` | Regression only. Standardize the target after any transform, fit in standardized space, then invert predictions and reported metrics to the original scale. |
| `positive_class` | integer or `null`; default `null` | Binary classification label treated as positive for probability and confusion diagnostics. Model-zoo subruns prefer class `1`, then the highest class, when omitted; the factory confusion summary defaults to `1`. |
| `log_targets` | boolean; default `false` | Deprecated compatibility alias for `target_transform: log`. Do not set it together with a non-`none` `target_transform`. |
| `target_transform` | `none`, `log`, or `log1p`; default `none` | Regression only. Transform targets before optional standardization and invert predictions back to the original scale. `log` requires values greater than zero; `log1p` requires values greater than -1. |
| `drop_non_positive_targets` | boolean; default `true` | Drop invalid training targets before `log` or `log1p` transformation. When false, invalid values fail the run. |

Model compatibility by task:

| Task | Models |
| --- | --- |
| Regression only | `ridge`, `lasso`, `elasticnet`, `bayesian_ridge`, `gaussian_process` |
| Classification only | `logistic_regression` |
| Both | `random_forest`, `extra_trees`, `svm`, `knn`, `xgboost` |

Outputs in the factory run directory:
- `summary.csv` and `summary.json`: one row per model, with `metric_*` columns, optional
  `uncertainty_correlation`, and binary `confusion_*` values when available. Regression summaries
  also add OLS slope, intercept, R2, and RMSE diagnostics.
- `<model-name>/`: the complete model-zoo subrun, including its configured predictions, metrics,
  results, logs, and artifact.
- `artifacts/<model-name>_model.joblib`: pooled copies of successfully produced model bundles.
- `results/result_record.json`: canonical factory result. Its paths expose `summary_csv`, `summary_json`,
  and `log`.

### Python API
```python
from cadd_ml.model_factory.workflow import ModelFactoryConfig, ModelFactoryWorkflow

cfg = ModelFactoryConfig(train_path="train.csv", target_column="y", task="regression")
wf = ModelFactoryWorkflow(cfg)
job = wf.prepare_job()
result = wf.run(job)
print(result.metrics)
```

## Explore Classifier Tool (alpha)

Interactive threshold tweaking for classification runs (model-zoo or model-factory).

```bash
cat > explore_classifier.yaml <<'YAML'
job_id: <existing_job_id>
run_base: runs
# model_name: random_forest   # optional; required for model-factory to preselect
positive_class: 1             # optional; defaults to 1
YAML

caddforge run ml explore-classifier --config explore_classifier.yaml
```

- Opens a Streamlit UI to select classifiers (for factory runs), adjust decision threshold (binary only), view confusion matrix and metrics, and save the chosen threshold.
- Saves threshold JSON under `artifacts/` and also updates the model artifact with the decision threshold when possible (used automatically by model-zoo/model-factory in predict mode).
- Provides metric vs threshold plots (0.01 steps) to visualize how metrics move with the decision threshold.
- Requires `streamlit` and `plotly` installed locally (install with the `ui` extra or add manually).

### Python API
```python
from cadd_ml.explore_classifier.workflow import ExploreClassifierConfig, ExploreClassifierWorkflow

cfg = ExploreClassifierConfig(job_id="existing_job_id", run_base="runs")
wf = ExploreClassifierWorkflow(cfg)
job = wf.prepare_job()
result = wf.run(job)  # launches Streamlit UI
```

## Next Steps
- Extend configs for model persistence and cross-validation.
- Add richer kernels and hyperparameter tuning helpers.
- Document feature preprocessing expectations for different kernels.
## Compressed inputs

Training, test, and PCA CSV inputs may be supplied as `.csv.gz`. They are expanded in job-local
runtime scratch, and saved configurations continue to reference the compressed source.

## Continuous integration and releases

`.github/workflows/release.yml` builds a wheel and source archive, validates
metadata, and tests the installed wheel on Linux with Python 3.10 and 3.12.
The lightweight test selection is recorded in `scripts/ci_tests.json`;
external scientific engines are validated separately.

Pushing a matching `v*` release tag publishes the tested artifacts to PyPI.
Branch pushes, pull requests and manual runs never publish. Configure a
PyPI Trusted Publisher for the `idorsia` repository, `release.yml` workflow
and `pypi` environment before pushing release tags. Runtime CADDForge
dependencies must already be available publicly; publish core first,
and publish docking/descriptors before MD.
