Metadata-Version: 2.4
Name: shap-recommender
Version: 0.3.0
Summary: Exclusion / non-linearity / interaction recommendations from saved SHAP attribution files, and application of them to a design matrix.
Author: Kaylee
License-Expression: MIT
Project-URL: Homepage, https://pypi.org/project/shap-recommender/
Keywords: shap,feature-selection,interaction-detection,interpretability,machine-learning
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Intended Audience :: Science/Research
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: scipy>=1.9
Requires-Dist: statsmodels>=0.13
Requires-Dist: patsy>=0.5
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# shap-recommender

Exclusion / non-linearity / interaction recommendations from saved SHAP
attribution files, and application of them to a design matrix.

One shared idea runs through two of the three rules. A feature's own
contribution is represented flexibly (indicator columns when it takes few
values, a restricted cubic spline when it is continuous), and a model built
on that flexible basis is compared against a straight line in the feature.
That comparison answers two different questions:

- **non-linearity** -- does the attribution deviate from a linear function
  of the feature? This directly tests the linear-trend assumption, rather
  than relying on a raw correlation coefficient, which conflates "no effect"
  with "non-linear effect".
- **interaction** -- does the attribution vary among subjects who share the
  same feature value? Under additivity the attribution is a deterministic
  function of the feature, so residual dispersion implies effect
  modification.

Stratification for the interaction screen is always applied to the observed
feature value at a pre-specified cut point, never to the attribution itself
-- splitting on the attribution would condition on the candidate modifier.
Within strata, the contrast `E[phi_y | y=1] - E[phi_y | y=0]` is compared,
which (unlike the marginal attribution distribution) is invariant to
stratum composition under additivity. The p-value for that contrast comes
from a bootstrap (`n_boot_screen` resamples), not an analytic standard
error, since the analytic version does not account for the stratum-specific
baseline itself being estimated and understates the true sampling
variability under an additive null. The flagging threshold (`min_abs_effect`)
is applied to the contrast difference scaled by the typical attribution
magnitude across both strata, rather than by the two contrasts' own
magnitude, since a ratio to the contrasts themselves blows up whenever both
are near zero.

## Install

```bash
pip install shap-recommender
```

## Expected input files

For each dataset "tag" you want to load, `Recommender.load(tag)` (and the
CLI's `--tags`) expects two tab-separated files in `res_dir`:

- `shap_values_<tag>.tsv` -- SHAP values, one row per subject, one column
  per feature, first column = row index.
- `sel_data_<tag>.tsv` -- the corresponding feature values (design matrix),
  same row index.

## Command-line use

```bash
shap-recommender \
    --res-dir ./shap_results \
    --tags cohort_a cohort_b \
    --nonlinear-candidates age bmi creatinine \
    --out ./recommendations
```

This writes `exclusion_tests.tsv`, `exclusion_sensitivity.tsv`,
`nonlinear_tests.tsv`, `interaction_tests.tsv`, `attribution_patterns.tsv`,
and `recommendations.json` to `--out`. Run `shap-recommender --help` for all
options (thresholds, bootstrap count, spline degrees of freedom, a
`--cutpoints` JSON file for pre-specified stratification cut points, etc).

## Library use

```python
from shap_recommender import Recommender

rec = Recommender(res_dir="./shap_results")
recommendations = rec.generate(
    candidates_nonlinear=["age", "bmi", "creatinine"],
    tags=["cohort_a", "cohort_b"],
    out="./recommendations",
)

# apply the recommendations to a design matrix
X_train_adj, X_test_adj = Recommender.apply(
    X_train, X_test, recommendations, variant="all",
)
```

`Recommender.apply(..., variant=...)` accepts `"baseline"`, `"exclusion"`,
`"nonlinear"`, `"interaction"`, or `"all"`, so each rule's effect on
downstream model performance can be evaluated separately.

## Validating the interaction rule

`Recommender.null_sim()` runs a small simulation under an additive null
(no true interaction with the feature being tested) and reports the type-I
error rate of the within-stratum contrast used by `stratified_screen`,
compared against naively partitioning on the attribution itself:

```python
from shap_recommender import Recommender
Recommender(res_dir=".").null_sim()
```

## License

MIT
