Metadata-Version: 2.4
Name: boulevard-boosting
Version: 0.1.0a1
Summary: Scikit-learn-compatible Boulevard and BRAT estimators with uncertainty intervals.
Author: Yichen Zhou, Kevin Tan, Giles Hooker, Haimo Fang
License-Expression: MIT
Project-URL: Homepage, https://github.com/boosting-inference/Boulevard
Project-URL: Repository, https://github.com/boosting-inference/Boulevard
Project-URL: Issues, https://github.com/boosting-inference/Boulevard/issues
Keywords: boosting,gradient boosting,uncertainty quantification,statistical inference,scikit-learn,machine learning
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.22
Requires-Dist: joblib>=1.3
Requires-Dist: scikit-learn<1.9,>=1.8
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: matplotlib>=3.8; extra == "dev"
Requires-Dist: nbformat>=5.9; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: black>=24.0; extra == "dev"
Provides-Extra: examples
Requires-Dist: matplotlib>=3.8; extra == "examples"
Dynamic: license-file

# Boulevard

Boulevard is a research-oriented Python package for sklearn-compatible
Boulevard / BRAT-style boosting estimators with uncertainty intervals.

The first release focuses on numeric squared-error regression with three public
estimators:

```python
import boulevard as bd

bd.DropoutBooster
bd.ParallelBooster
bd.ExplainableBooster
```

`DropoutBooster` and `ParallelBooster` are histogram-tree estimators built on
scikit-learn's histogram gradient boosting internals. `ExplainableBooster` is a
main-effect additive estimator in the EBM family, implemented directly in this
package.

There are no backwards-compatible aliases for older pre-release class names.

## Install

The package name is `boulevard-boosting`; the import name is `boulevard`.

```bash
pip install boulevard-boosting
```

For local development:

```bash
pip install -e ".[dev]"
```

## Quickstart

```python
import boulevard as bd

model = bd.DropoutBooster(
    max_iter=700,
    learning_rate=0.8,
    dropout_rate=0.1,
    subsample_rate=0.8,
    max_depth=6,
    max_leaf_nodes=64,
    min_samples_leaf=2,
    max_bins=64,
    random_state=0,
)

model.fit(X_train, y_train)
model.prepare_inference(X_calib, y_calib)

pred = model.predict(X_test)
lower, upper, pred = model.predict_intervals(
    X_test,
    level=0.95,
    mode="confidence",
)
```

`prepare_inference` estimates residual variance and builds the cached linear
algebra used by interval methods. If it is omitted, interval methods prepare
inference on first use with the training data. A held-out calibration set is
usually preferable for variance estimation.

Runnable examples are included in the source repository:

```bash
git clone https://github.com/boosting-inference/Boulevard.git
cd Boulevard
pip install -e ".[examples]"
python examples/quickstart.py
```

Open the broader demo notebook from the source repository:

```text
examples/boulevard_boosters_demo.ipynb
```

## Recommended Use

Use `DropoutBooster` or `ParallelBooster` for low-dimensional smooth regression
signals. When dimension grows and an additive/main-effect model is scientifically
reasonable, start with `ExplainableBooster` and inspect its feature-level
intervals.

For interaction-heavy higher-dimensional functions, treat signal confidence
intervals as diagnostic unless you validate coverage in a problem-specific
simulation. Prediction intervals are currently the safer interval type for noisy
outcome uncertainty.

## TabArena Regression Benchmark

We also ran a local performance-positioning benchmark against common tree-based
tabular models on TabArena's regression/lite datasets. This benchmark is meant
to show where the current Boulevard estimators sit relative to existing
tree-based prediction tools; it is not an official TabArena leaderboard
submission.

Setup:

- Data: 13 TabArena regression/lite datasets from the `r0f0` split:
  `Another-Dataset-on-used-Fiat-500`, `Food_Delivery_Time`, `QSAR-TID-11`,
  `QSAR_fish_toxicity`, `airfoil_self_noise`,
  `concrete_compressive_strength`, `diamonds`,
  `healthcare_insurance_expenses`, `houses`, `miami_housing`,
  `physiochemical_protein`, `superconductivity`, and `wine_quality`.
- Evaluation: each TabArena training fold was split into an inner training set
  and validation set. The TabArena test fold was not used.
- Tuning: 30 Optuna trials per model on the same 13 datasets.
- Score: mean normalized validation RMSE. For each dataset,
  `normalized_RMSE = RMSE / std(y_val)`, then scores are averaged across
  datasets. Lower is better.
- Baselines: `HGBR` is sklearn's `HistGradientBoostingRegressor`. Additive
  InterpretML EBM is the InterpretML EBM with `interactions=0`.

Best tuned scores via Optuna, 30 trials per model:

| Model | Mean Normalized Validation RMSE | Mean Rank | Best Trial Total Fit Seconds |
| --- | ---: | ---: | ---: |
| CatBoost | 0.432 | 4.31 | 305.9 |
| XGBoost | 0.433 | 3.38 | 61.2 |
| LightGBM | 0.437 | 3.77 | 25.9 |
| HGBR | 0.441 | 5.31 | 265.5 |
| `ParallelBooster` | 0.447 | 5.69 | 197.2 |
| Random Forest | 0.448 | 5.23 | 469.7 |
| Extra Trees | 0.449 | 5.23 | 267.9 |
| InterpretML EBM | 0.455 | 5.62 | 692.7 |
| `DropoutBooster` | 0.466 | 8.00 | 892.1 |
| No-Interaction Additive InterpretML EBM | 0.528 | 8.54 | 72.3 |
| `ExplainableBooster` | 0.680 | 10.92 | 297.6 |

![Best tuned score across TabArena regression/lite datasets](assets/tabarena-regression-lite/best_score_bar.png)

![Accuracy-time tradeoff across tuned models](assets/tabarena-regression-lite/accuracy_time_scatter.png)

![Per-dataset rank heatmap](assets/tabarena-regression-lite/per_dataset_rank_heatmap.png)

![Optuna best-so-far curves](assets/tabarena-regression-lite/best_so_far_curve.png)

The main takeaway is that `ParallelBooster` is the strongest current Boulevard
predictive model in this benchmark, close to sklearn HGBR and random forests but
behind CatBoost/XGBoost/LightGBM. `ExplainableBooster` is intentionally
main-effect-only; its weaker score on general TabArena regression tasks should
be read as an additive-model limitation rather than a failure of the interval
API.

## Common Interval API

All three estimators support:

```python
model.fit(X_train, y_train)
model.prepare_inference(X_calib, y_calib)

lower, upper, pred = model.predict_intervals(
    X_test,
    level=0.95,
    mode="confidence",
)
```

Modes:

- `"confidence"`: uncertainty for the fitted signal.
- `"prediction"`: signal uncertainty plus residual noise variance.
- `"reproduction"`: repeated-training signal uncertainty.

`prepare_inference` estimates `sigma_hat2_` as centered residual variance on the
supplied calibration data. If no calibration data is supplied, it uses training
data.

## DropoutBooster

`DropoutBooster` is the sklearn-compatible BRAT-D / Boulevard dropout estimator.

```python
model = bd.DropoutBooster(
    max_iter=700,
    learning_rate=0.8,
    dropout_rate=0.1,
    subsample_rate=0.8,
    max_depth=6,
    max_leaf_nodes=64,
    min_samples_leaf=2,
    max_bins=64,
    random_state=0,
)
```

Important parameters:

- `max_iter`: number of trees.
- `learning_rate`: Boulevard/BRAT-D signal multiplier.
- `dropout_rate`: probability of dropping an old tree when constructing the
  next residual. Must be in `[0, 1)`.
- `subsample_rate`: row subsampling fraction for each new tree.
- `max_depth`, `max_leaf_nodes`: tree complexity controls.
- `min_samples_leaf`: minimum leaf size. Larger values smooth the fit.
- `max_bins`: number of histogram bins per feature. scikit-learn currently
  supports at most `255`.

Useful methods:

```python
pred = model.predict(X_test)
norms = model.weight_norms(X_test)
cells = model.apply_cell_indices(X_test)
ci_lower, ci_upper, _ = model.predict_intervals(X_test, mode="confidence")
pi_lower, pi_upper, _ = model.predict_intervals(X_test, mode="prediction")
ri_lower, ri_upper, _ = model.predict_intervals(X_test, mode="reproduction")
```

## ParallelBooster

`ParallelBooster` is the sklearn-compatible BRAT-P estimator. It uses
deterministic tree slots instead of random dropout.

```python
model = bd.ParallelBooster(
    n_rounds=70,
    trees_per_round=6,
    subsample_rate=0.8,
    max_depth=8,
    max_leaf_nodes=64,
    min_samples_leaf=4,
    max_bins=64,
    drop_first_round=True,
    n_jobs=1,
    random_state=0,
)
```

Important parameters:

- `n_rounds`: number of completed-history BRAT-P rounds.
- `trees_per_round`: number of deterministic tree slots per round. Total tree
  count is `n_rounds * trees_per_round`.
- `subsample_rate`: row subsampling fraction for each slot tree.
- `n_jobs`: optional joblib worker count for slot fitting. Use `1` or `None`
  for serial fitting.
- `drop_first_round`: if `False`, the first round is sequentially warm-started.
  If `True`, the first round uses the same completed-history shape as later
  rounds.
- `max_depth`, `max_leaf_nodes`, `min_samples_leaf`, `max_bins`: histogram-tree
  complexity controls.

The interval API matches `DropoutBooster`.

## ExplainableBooster

`ExplainableBooster` is an EBM-family additive estimator with feature-level
intervals. It is implemented in Boulevard and does not depend on InterpretML.

```python
model = bd.ExplainableBooster(
    max_rounds=160,
    max_bins=32,
    learning_rate=0.6,
    subsample_rate=1.0,
    warmup_rounds=10,
    max_depth=4,
    min_samples_leaf=8,
    leave_one_out=False,
    random_state=0,
)
```

Important parameters:

- `max_rounds`: number of additive update rounds.
- `max_bins`: number of one-dimensional bins per feature.
- `learning_rate`: additive update multiplier.
- `subsample_rate`: row subsampling for each feature update.
- `warmup_rounds`: number of early rounds before Boulevard-style averaging.
- `max_depth` or `max_leaves`: one-dimensional tree complexity. Specify only
  one.
- `min_samples_leaf`: minimum leaf size.
- `leave_one_out`: whether each feature update drops that feature's current
  contribution from the residual.

Feature-level interval API:

```python
term_lower, term_upper, term_pred = model.predict_feature_intervals(
    feature_idx=0,
    x_k=X_test[:, 0],
    level=0.95,
    mode="confidence",
)
```

## Hyperparameter Recommendations

These recommendations are empirical starting points, not formal guarantees.
They are based on synthetic diagnostics with known regression signals, Gaussian
noise with standard deviation `0.2`, and separate train / calibration / test
splits.

Coverage labels:

| Label | Meaning |
| --- | --- |
| `PASS` | signal CI coverage >= 0.90 |
| `OK` | signal CI coverage >= 0.80 |
| `LOW` | signal CI coverage < 0.80 |

Best trial per estimator and scenario:

| Scenario | Estimator | Status | RMSE vs Signal | Signal CI Coverage | PI Coverage | Median CI Width |
| --- | --- | --- | ---: | ---: | ---: | ---: |
| `smooth_1d` | `DropoutBooster` | PASS | 0.049 | 0.960 | 0.965 | 0.210 |
| `smooth_1d` | `ParallelBooster` | PASS | 0.053 | 0.980 | 0.970 | 0.250 |
| `smooth_1d` | `ExplainableBooster` | OK | 0.061 | 0.855 | 1.000 | 0.181 |
| `additive_3d` | `DropoutBooster` | OK | 0.146 | 0.824 | 0.976 | 0.400 |
| `additive_3d` | `ParallelBooster` | OK | 0.129 | 0.858 | 0.976 | 0.408 |
| `additive_3d` | `ExplainableBooster` | LOW | 0.071 | 0.782 | 1.000 | 0.176 |
| `interaction_5d` | `DropoutBooster` | LOW | 0.134 | 0.523 | 0.961 | 0.198 |
| `interaction_5d` | `ParallelBooster` | LOW | 0.149 | 0.606 | 0.958 | 0.253 |
| `interaction_5d` | `ExplainableBooster` | LOW | 0.077 | 0.603 | 1.000 | 0.139 |

The main takeaways are:

- 1D smooth problems are easy for all three estimators.
- 3D additive problems need more capacity than the original prototype defaults.
- 5D interaction problems remain hard for asymptotic signal CIs. Predictive RMSE
  can be reasonable while signal CI coverage is poor.
- Prediction intervals were close to or above 95% coverage in these diagnostics.
  This does not imply signal CIs are automatically calibrated.

### Dimension And Capacity

As dimension increases, use more samples and more model capacity.

| Effective Dimension | Suggested Samples | Tree/Partition Capacity |
| --- | ---: | --- |
| 1 | 500-1000 | shallow to moderate trees |
| 2-3 | 2000-4000 | deeper trees and more leaves |
| 4-8 | 5000+ | high capacity, then validate coverage |

Tune in this order:

1. Increase tree count or rounds.
2. Increase `max_depth` and `max_leaf_nodes`.
3. Decrease `min_samples_leaf` if the fit is over-smoothed.
4. Increase `max_bins` if the fitted function is too coarse.

If RMSE is high and signal CI coverage is low, tune the predictive model first.
The interval formula does not automatically correct approximation bias.

The estimators warn when tree-capacity settings are internally capped. For
example, increasing `max_leaf_nodes` will not change a tree if `max_depth`,
`min_samples_leaf`, subsampling, or one-dimensional `max_bins` already imposes a
smaller effective leaf budget.

### DropoutBooster Tuning

Low-dimensional starting point:

```python
model = bd.DropoutBooster(
    max_iter=700,
    learning_rate=0.8,
    dropout_rate=0.1,
    subsample_rate=0.8,
    max_depth=6,
    max_leaf_nodes=64,
    min_samples_leaf=2,
    max_bins=64,
    random_state=0,
)
```

Higher-dimensional additive starting point:

```python
model = bd.DropoutBooster(
    max_iter=1000,
    learning_rate=3.0,
    dropout_rate=0.1,
    subsample_rate=1.0,
    max_depth=10,
    max_leaf_nodes=256,
    min_samples_leaf=2,
    max_bins=128,
    random_state=0,
)
```

Recommended tuning:

- `max_iter`: use `200-700` for simple low-dimensional signals; use `700-1000`
  when dimension or curvature increases.
- `learning_rate`: start around `0.8-1.5` for 1D; larger values such as `3.0`
  were selected for the 3D additive search.
- `dropout_rate`: start around `0.0-0.1` for accuracy. Higher dropout is
  stronger regularization and can worsen signal bias.
- `subsample_rate`: start around `0.8`; use `1.0` when the model underfits.
- `max_depth` and `max_leaf_nodes`: increase together as dimension grows.
- `min_samples_leaf`: use small values such as `2-4` when bias dominates.
- `max_bins`: use `64` for smooth 1D signals; try `128-255` for higher
  dimension or sharper structure.

### ParallelBooster Tuning

Low-dimensional starting point:

```python
model = bd.ParallelBooster(
    n_rounds=70,
    trees_per_round=6,
    subsample_rate=0.8,
    max_depth=8,
    max_leaf_nodes=64,
    min_samples_leaf=4,
    max_bins=64,
    drop_first_round=True,
    n_jobs=1,
    random_state=0,
)
```

Higher-dimensional additive starting point:

```python
model = bd.ParallelBooster(
    n_rounds=70,
    trees_per_round=6,
    subsample_rate=1.0,
    max_depth=10,
    max_leaf_nodes=256,
    min_samples_leaf=4,
    max_bins=64,
    drop_first_round=True,
    n_jobs=1,
    random_state=0,
)
```

Recommended tuning:

- Total tree budget is `n_rounds * trees_per_round`.
- Increase `n_rounds` when the fit underfits.
- Increase `trees_per_round` when signal intervals are too narrow, but expect
  some tradeoff with fit time and possibly RMSE.
- `drop_first_round=True` keeps the first BRAT-P round closer to the frozen
  parallel training interpretation.
- Tune tree complexity the same way as `DropoutBooster`.
- Use `n_jobs=1` by default. Parallel scheduling only helps when
  `trees_per_round` is large enough.

### ExplainableBooster Tuning

Additive 3D starting point:

```python
model = bd.ExplainableBooster(
    max_rounds=160,
    max_bins=32,
    learning_rate=0.6,
    subsample_rate=0.8,
    warmup_rounds=10,
    max_depth=4,
    min_samples_leaf=8,
    leave_one_out=False,
    random_state=0,
)
```

Recommended tuning:

- `max_rounds`: start around `160`; increase toward `320` for harder additive
  functions.
- `max_depth`: start around `2-4`. Stumps are often too biased for nonlinear
  partial functions.
- `max_bins`: start with `32-64`; increase only when partial effects look too
  coarse and sample size is sufficient.
- `learning_rate`: start around `0.6-1.0`.
- `warmup_rounds`: `0-20` is a reasonable starting range.
- `leave_one_out=False` was selected in the best runs above.

`ExplainableBooster` can have strong RMSE while its signal CI coverage remains
below target. Treat current intervals as diagnostic and validate them on held-out
or simulated truth before relying on them.

## Practical Workflow

For a new dataset:

1. Split into train, calibration, and test or validation sets.
2. Fit a moderate starting model from the recommendations above.
3. Run `prepare_inference(X_calib, y_calib)`.
4. Check RMSE, residual variance, interval width quantiles, PI coverage for
   noisy outcomes, and CI coverage for known simulation truth if available.
5. If signal CI coverage is low and RMSE is high, increase model capacity.
6. If signal CI coverage is low but RMSE is already good, treat the signal CI as
   under-calibrated for that problem class.
7. Keep a held-out diagnostic script or notebook for every real analysis.

## Citation

If you use Boulevard, cite the software and the method paper most closely tied
to the estimator you use. A machine-readable citation file is provided in
`CITATION.cff`.

- Boulevard regularization:
  Yichen Zhou and Giles Hooker. 2022. "Boulevard: Regularized Stochastic
  Gradient Boosted Trees and Their Limiting Distribution." Journal of Machine
  Learning Research, 23(183):1-44.
- `DropoutBooster` and `ParallelBooster`:
  Haimo Fang, Kevin Tan, and Giles Hooker. 2025. "Statistical Inference for
  Gradient Boosting Regression." Advances in Neural Information Processing
  Systems 38.
- `ExplainableBooster`:
  Haimo Fang, Kevin Tan, Jonathan Pipping, and Giles Hooker. 2026. "Statistical
  Inference for Explainable Boosting Machines." AISTATS 2026.
- TabArena benchmark results:
  Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmueller, Prateek
  Mutalik Desai, David Salinas, and Frank Hutter. 2025. "TabArena: A Living
  Benchmark for Machine Learning on Tabular Data." NeurIPS 2025.
- Optuna tuning runs:
  Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori
  Koyama. 2019. "Optuna: A Next-generation Hyperparameter Optimization
  Framework." KDD 2019.
