# Release 0.8.450

!!!tip inline end "Need Help?"
    The SuperCowPowers team is happy to give any assistance needed when setting up AWS and Workbench. So please contact us at [workbench@supercowpowers.com](mailto:workbench@supercowpowers.com) or on chat us up on [Discord](https://discord.gg/WHAJuz8sw8)

Training predictions split into two files and `full_cross_fold` becomes leak-free. Hyperparameter search now ranks whole trials.

## Out-of-fold and validation predictions are separate files

Training produces two kinds of scored rows, and they measure different things:

* **Out-of-fold** — each row scored by the one fold model that held it out. Single-model, leak-free.
* **Held-out validation** — your `validation_ids` rows, scored by the full ensemble.

You can't get a leak-free *ensemble* score on training rows — for any fold's held-out rows, the other four models trained on them. So the two are genuinely different estimators, and they now write to **`oof_predictions.csv`** and **`val_predictions.csv`** instead of sharing one file with a marker column.

**Why it matters:** `full_cross_fold` was computed over both row types at once. A model with 13,170 training rows and 9,442 validation rows drew ~42% of its reported metric from ensemble predictions. It now reads out-of-fold only.

Held-out rows also carry real uncertainty now — scored through the same path production inference uses for a new molecule — so `val_predictions.csv` is a direct read on whether production UQ is calibrated.

Models using `model_class` (scikit-learn) train on a single split, so they write an empty `oof_predictions.csv` and no longer produce a `full_cross_fold` capture.

## Hyperparameter search ranks whole trials

**Pruning is gone.** Every trial now trains its full ensemble, so every objective value sits on the same basis and the shortlist compares like with like. The objective is the pooled out-of-fold error (`cv_mae`), or `holdout_mae` when `validation_ids` are designated.

**Budgets are sized per framework** when you don't set `n_trials` — 40 ChemProp, 100 PyTorch, 250 XGBoost, scaled to what one trial costs.

**`trial_counts` reports failures.** `completed` should equal `attempted`; any `failed` means the budget was quietly spent on trials that died (CUDA OOM is the usual cause when trials share a GPU).

## Data capture

Endpoints declare CSV and JSON content types explicitly, and captured data is parsed in a single pass. Backup is now a thin `aws s3 sync` wrapper rather than a bespoke script.

## REPL and Bosco

Named sessions with a recency-sorted listing, launched Batch jobs watched and reported on completion, and light/dark REPL themes driven by `REPL_THEME`.

## Also

* `Model` exposes training-job utilization; training-job readers moved out of `model_utils`.
* Inference image carries the X11 and libexpat libraries ChemProp's cuik-molmaker needs.
* Security bumps: aiohttp 3.14.3, GitPython 3.1.58.

## Upgrade notes

* **Retrain when convenient, not urgently.** Models trained before this release are read through their old capture, filtered to the out-of-fold rows — so they report the *corrected* leak-free metric without retraining. Each read logs a warning naming the model.
* `Model.get_inference_predictions("model_training")` now resolves to out-of-fold predictions.
* Dashboard views read stored inference captures, so existing model pages are unaffected.
* No client code changes are required.

## Questions?
<img align="right" src="../../images/scp.png" width="180">

The SuperCowPowers team is happy to answer any questions you may have about AWS and Workbench. Please contact us at [workbench@supercowpowers.com](mailto:workbench@supercowpowers.com) or on chat us up on [Discord](https://discord.gg/WHAJuz8sw8)
