Metadata-Version: 2.4
Name: stat-context
Version: 0.5.0
Summary: Grouped target statistics for tabular regression with cross-fitting and leakage diagnostics
Author: Mateusz Stachowicz, Stanisław Halkiewicz
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/joint-hubs/sce
Project-URL: Documentation, https://joint-hubs.github.io/sce
Project-URL: Repository, https://github.com/joint-hubs/sce.git
Project-URL: Bug Tracker, https://github.com/joint-hubs/sce/issues
Project-URL: Changelog, https://github.com/joint-hubs/sce/blob/main/CHANGELOG.md
Keywords: machine-learning,feature-engineering,statistical-context,regression,hierarchical-features,xgboost
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.21.0
Requires-Dist: pandas<3.0.0,>=1.3.0
Requires-Dist: scikit-learn<2.0.0,>=1.4.0
Requires-Dist: xgboost<3.0.0,>=1.5.0
Requires-Dist: joblib>=1.1.0
Requires-Dist: toml>=0.10.2
Requires-Dist: tomli>=2.0.0; python_version < "3.11"
Requires-Dist: pyarrow>=10.0.0
Provides-Extra: models
Requires-Dist: lightgbm>=4.0.0; extra == "models"
Requires-Dist: catboost>=1.2.0; extra == "models"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: isort>=5.12.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Provides-Extra: data
Requires-Dist: huggingface-hub>=0.19.0; extra == "data"
Requires-Dist: kaggle>=1.7.4; extra == "data"
Requires-Dist: requests>=2.31.0; extra == "data"
Requires-Dist: tqdm>=4.66.0; extra == "data"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.5.0; extra == "viz"
Requires-Dist: seaborn>=0.12.0; extra == "viz"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5.0; extra == "docs"
Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
Provides-Extra: equity
Requires-Dist: yfinance<2,>=0.2.40; extra == "equity"
Requires-Dist: pandas_market_calendars<6,>=4.3.0; extra == "equity"
Requires-Dist: pandera<1,>=0.20.0; extra == "equity"
Provides-Extra: sentiment
Requires-Dist: transformers>=4.30; extra == "sentiment"
Requires-Dist: torch; extra == "sentiment"
Requires-Dist: vaderSentiment>=3.3.2; extra == "sentiment"
Provides-Extra: all
Requires-Dist: stat-context[data,dev,docs,equity,models,sentiment,viz]; extra == "all"
Dynamic: license-file

# Statistical Context Engineering (SCE)

**Cross-fitted, leakage-aware target statistics for tabular regression.**

[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Tests](https://github.com/joint-hubs/sce/actions/workflows/ci.yml/badge.svg)](https://github.com/joint-hubs/sce/actions/workflows/ci.yml)
[![PyPI version](https://badge.fury.io/py/stat-context.svg)](https://pypi.org/project/stat-context/)
[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://joint-hubs.github.io/sce)

SCE enriches a tabular dataset with group-level statistics of the target
variable (means, medians, quantiles, spread, counts) computed over categorical
columns — the "context" each row lives in. The hard part of target encoding is
doing it **without leaking the target**: SCE uses out-of-fold cross-fitting on
the training set, supports temporal fold layouts for time-ordered data, and
ships diagnostics that help you *check* enrichment behavior on your data (no
automated check can prove the absence of leakage).

## Installation

```bash
pip install stat-context
```

```python
import sce  # package name on PyPI is stat-context, import name is sce
```

Optional extras:

```bash
pip install stat-context[models]     # LightGBM + CatBoost backends
pip install stat-context[data]       # remote dataset download helpers
pip install stat-context[dev]        # tests, linters
pip install stat-context[equity]     # experimental equity research stack (see below)
pip install stat-context[sentiment]  # experimental sentiment scorers (FinBERT, VADER)
```

## 60-second quickstart

Copy-paste runnable — synthetic data, no files needed:

```python
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import root_mean_squared_error
from xgboost import XGBRegressor

from sce import StatisticalContextEngine, ContextConfig, AggregationMethod

# --- toy data: price depends on a hidden per-city level the model can't see
rng = np.random.default_rng(42)
n = 8000
cities = rng.choice([f"city_{i}" for i in range(40)], size=n)
city_level = {f"city_{i}": rng.normal(100, 30) for i in range(40)}
df = pd.DataFrame({
    "city": cities,
    "sqft": rng.uniform(20, 120, size=n),
    "rooms": rng.integers(1, 6, size=n),
})
df["price"] = (
    df["city"].map(city_level) * 10 + df["sqft"] * 50 + rng.normal(0, 300, size=n)
)

# --- split FIRST, then enrich (this is the recommended order)
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)

config = ContextConfig(
    target_col="price",
    categorical_cols=["city"],
    aggregations=[AggregationMethod.MEAN, AggregationMethod.STD, AggregationMethod.COUNT],
    use_cross_fitting=True,   # train rows get out-of-fold stats
    n_folds=5,
)
engine = StatisticalContextEngine(config)
train_enriched = engine.fit_transform(train_df)                       # OOF stats
test_enriched = engine.transform(test_df.drop(columns=["price"]))     # full-train stats

# --- single model on the enriched features
features_sce = ["sqft", "rooms"] + [c for c in train_enriched.columns
                                    if c.startswith("city_price_")]
model = XGBRegressor(n_estimators=200, random_state=42)
model.fit(train_enriched[features_sce], train_enriched["price"])
rmse = root_mean_squared_error(
    test_df["price"], model.predict(test_enriched[features_sce])
)
print(f"test RMSE: {rmse:,.0f}")
```

This example shows the mechanics: split first, fit the engine on train only,
transform test without the target. It deliberately does **not** stage a
baseline contest. If you add one, keep it fair: `city` is visible to any
baseline too, so a meaningful comparison must let the baseline encode it (e.g.
`OneHotEncoder` on `city`, or a target encoder with its own out-of-fold
machinery) — a numeric-only baseline is not evidence that enrichment helps.

New features follow the pattern `{column}_{target}_{stat}`, e.g.
`city_price_mean`, `city_price_std`, `city_price_count`, plus fold-variance
features (`*_fold_std`, `*_fold_lower`, `*_fold_upper`) that quantify how
stable each statistic is across folds. To see what was added, diff the
DataFrame columns before and after enrichment.

### Time-ordered data

`cross_fit_strategy` separates two different temporal layouts; both require
`time_col` and build folds in chronological order (rows are sorted by
`time_col` before splitting, and each validation fold follows its training
window):

- `"time"` — expanding-window temporal folds (`TimeSeriesSplit`): each
  validation fold is later than its training data, and the training window
  grows fold by fold.
- `"rolling"` — same ordering, but the training window is capped
  (`rolling_max_train_size`), so old observations drop out of the statistics;
  an optional `rolling_gap` can additionally separate train and validation by
  a buffer.

```python
config = ContextConfig(
    target_col="sales",
    categorical_cols=["store", "dept"],
    use_cross_fitting=True,
    cross_fit_strategy="rolling",   # or "time" for expanding folds
    time_col="date",
    n_folds=5,
)
```

Caveat: folds are split by row position after sorting, so rows sharing an
identical timestamp can land on both sides of a fold boundary. The engine
also does not check that labels were observable when each row is scored —
ensuring a chronologically sound split and point-in-time label availability
is your responsibility.

## Current capabilities

- **Out-of-fold cross-fitting** with four strategies: `random` (shuffled
  K-fold), `time` (expanding temporal folds), `rolling` (capped sliding
  temporal window), and `off` (no cross-fitting).
- **Grouped target summaries** — per group: mean, median, std, var, CV,
  quantiles, count.
- **Interactions** — 2-way categorical crosses (`city×property_type`) by
  default (`include_interactions=True`, `max_interaction_depth=2`).
- **Rare-group backoff** — groups smaller than `min_group_size` (default 5)
  fall back to global statistics instead of producing noisy estimates.
- **Fold-stability features** (`include_fold_variance=True` by default) —
  `*_fold_std`/`*_fold_cv` and `*_fold_lower`/`*_fold_upper` (fold mean ± 2×
  fold std) describe how stable each statistic is across folds. They are
  stability diagnostics, **not calibrated uncertainty or confidence bounds**.
- **Model factory** — `sce.build_model("ridge", ...)` now returns a `RidgeCV`
  that selects its regularization over a built-in alpha grid (a scalar `alpha`
  argument is ignored — pass `alphas` to override). The pure-scikit-learn
  estimators (ridge, random forest, extra trees, gradient boosting) are
  wrapped in a median-imputing `Pipeline`, so the NaN-heavy feature blocks SCE
  produces for small groups fit cleanly. Details:
  [CHANGELOG.md](https://github.com/joint-hubs/sce/blob/main/CHANGELOG.md).
- **Optional cleanup** — `FeatureCleanupPipeline` for pruning low-value or
  unstable context features after enrichment.

Scope warnings:

- **Relative features are opt-in and leakage-prone.**
  `include_relative_features` defaults to `False`, and when enabled, `transform`
  requires the target column at inference time (the engine raises if the target
  is missing). SCE is therefore **not suitable for ordinary predictive
  deployments** with that option on — it is a research/analysis tool in that
  mode.
- **No guaranteed improvements.** Enrichment helps when the categorical
  hierarchy is informative; it can do nothing or hurt otherwise. There is no
  guarantee in either direction.

## Configuration reference

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `target_col` | `str` | required | Target variable column |
| `categorical_cols` | `list[str]` | `None` | Grouping columns (auto-detected if `None`) |
| `aggregations` | `list[AggregationMethod]` | `[MEAN, MEDIAN, STD, Q05, Q20, Q80, Q95, COUNT]` | Statistics to compute |
| `use_cross_fitting` | `bool` | `True` | Out-of-fold statistics on train |
| `cross_fit_strategy` | `str` | `"random"` | `"random"`, `"time"`, `"rolling"`, or `"off"` (requires `use_cross_fitting=False`) |
| `time_col` | `str` | `None` | Required for `"time"` and `"rolling"` |
| `n_folds` | `int` | `5` | Cross-fitting folds |
| `min_group_size` | `int` | `5` | Groups smaller than this back off to global stats |
| `include_global_stats` | `bool` | `True` | Dataset-wide statistics |
| `include_interactions` | `bool` | `True` | 2-way cross-column groups (`city×type`) |
| `include_fold_variance` | `bool` | `True` | Fold-stability features (`*_fold_std`, ...) |
| `include_relative_features` | `bool` | `False` | Z-score/ratio features; **requires target at inference, leakage-prone** |
| `rolling_max_train_size` | `int` | `None` | Rolling train window size (rows), `None` = auto |
| `rolling_gap` | `int` | `0` | Rows skipped between train and validation in `"rolling"` |

```python
class StatisticalContextEngine(BaseEstimator, TransformerMixin):
    def fit(self, X: pd.DataFrame, y: Optional[pd.Series] = None) -> "StatisticalContextEngine": ...
    def transform(self, X: pd.DataFrame) -> pd.DataFrame: ...
    def fit_transform(self, X: pd.DataFrame, y: Optional[pd.Series] = None) -> pd.DataFrame: ...
```

Scikit-learn compatible; `sce.pipeline.create_sce_pipeline(config, model)`
wraps the engine and an optional model into a `Pipeline` fit on
`(X, y)`. Full API docs: [joint-hubs.github.io/sce](https://joint-hubs.github.io/sce/api/)

## How it works

1. **Grouping** — categorical columns (and optionally their pairwise
   interactions) define groups: `city`, `city×property_type`, …
2. **Aggregation** — for each group, compute the configured target statistics.
3. **Cross-fitting** — on the training set, each row receives statistics
   computed *without its own fold* (random K-fold by default, or
   expanding/rolling temporal folds for time-ordered data). Small groups back
   off to global statistics.
4. **Inference** — test/production rows receive statistics computed from the
   full training set; the target is not needed at transform time (unless
   relative features are explicitly enabled).

The recommended evaluation protocol is split-first: the train/test split
happens **before** any enrichment, encoders and pruning are fit on train
only, and the test set is touched once, at scoring time.

## Evidence status

The numbers in this repository's `results/` directories come from
**historical, exploratory runs** on a small set of datasets. They are
illustrative only: they are not a curated benchmark, they predate the current
code, and negative or null results were not systematically collected or
reported. Do not treat any figure there as an expected gain.

- Exploratory result directories (not checked into git): `results/`,
  `results_gcp_*/`
- Dataset configurations (including ones excluded from that set):
  [`configs/`](https://github.com/joint-hubs/sce/blob/main/configs/),
  [`configs/experimental/`](https://github.com/joint-hubs/sce/blob/main/configs/experimental/)
- Run metadata (`git` SHA, config hash, seed, metrics, diagnostics) is written
  to `metadata.json` in each results directory, so individual runs remain
  traceable to an exact commit and config.

## Reproducing experiments

Requires a repo clone (datasets are rebuilt locally — Kaggle sources cannot be
redistributed):

```bash
git clone https://github.com/joint-hubs/sce.git && cd sce
pip install -e .[dev,data]

# rebuild the Kaggle-derived datasets (needs Kaggle API credentials)
python scripts/prepare_new_datasets.py
python scripts/prepare_m5_dataset.py --download

sce datasets list                                  # see what is available
python scripts/run.py --dataset rental_poland_short   # single experiment
python scripts/run.py --all                        # full sweep
```

### Diagnostics

```bash
python -m scripts.diagnostics.permuted_target  --dataset rossmann_daily
python -m scripts.diagnostics.shuffled_groups  --dataset rossmann_daily
python -m scripts.diagnostics.crossfit_ab      --dataset rossmann_daily
```

| Diagnostic | Question it probes | Reading |
|---|---|---|
| `permuted_target` | Does SCE "improve" on pure noise? | advantage < 1% on shuffled targets |
| `shuffled_groups` | Does SCE need real group structure? | no advantage with shuffled categories |
| `crossfit_ab` | What does cross-fitting cost/protect? | informational |

These checks reduce the risk of silent leakage but do not prove its absence.

## Experimental: equity research package

The 0.5.0 release also ships an **experimental** `equity` package —
point-in-time OHLCV ingestion for equity universes (e.g. S&P 500), SCE
feature enrichment, and a walk-forward forecaster with sector-head,
residual, and quantile heads. Install its dependencies with the
`[equity]` extra; a companion `[sentiment]` extra adds FinBERT/VADER
sentiment scoring for the same pipeline.

A single-fold smoke run (requires a `prices.parquet` in the canonical schema
documented in
[`equity/data/README.md`](https://github.com/joint-hubs/sce/blob/main/equity/data/README.md)):

```bash
python -m equity.forecaster.run_smoke --prices prices.parquet --sectors sectors.csv
```

Walk-forward evaluation is available via
`python -m equity.forecaster.run_walk_forward`. This is research tooling for
studying SCE on market data: it makes **no investment-performance claims**
and is not trading software.

## Research status and roadmap

An earlier version of the accompanying manuscript was submitted to ICML 2026
and was **not accepted**. There is no publication, acceptance, or
under-review status to report.

Planned work:

- **Leakage audit**, including nested evaluation (selection and reporting on
  the same fold can itself leak).
- **Matched comparisons** against hierarchical/mixed-model baselines:
  target encoding variants, hierarchical GLMMs, and gradient-boosted trees
  with native categorical support (e.g. CatBoost) under identical splits and
  budgets.
- **Multi-domain regression benchmark** with pre-registered inclusion rules so
  negative results are part of the published set.
- **Reproducibility and negative results** as first-class outputs.
- **Conditional shrinkage and compact context selection** — pursued only if
  supported by the audit and comparison results above.

## Project layout

```
sce/                  # the library (pip install stat-context)
├── engine.py         #   StatisticalContextEngine
├── config.py         #   ContextConfig, AggregationMethod
├── models.py         #   model factory (xgboost core; lightgbm/catboost optional)
├── stats.py          #   aggregation functions, backoff
├── cleanup.py        #   FeatureCleanupPipeline
└── io/               #   dataset registry + loading
configs/              # dataset configs (TOML)
configs/experimental/ # configs excluded from the exploratory set
equity/               # experimental equity research package (see above)
scripts/              # experiment runner, diagnostics, dataset preparation
tests/                # pytest suite (incl. leakage guard tests)
```

## License

This software is released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) — free for academic, research, and commercial use.

The accompanying research manuscript remains under CC BY-NC 4.0.

## Citation

If you use SCE in academic work, please cite the software:

```bibtex
@misc{stat_context_2026,
  title        = {Statistical Context Engineering (stat-context)},
  author       = {Stachowicz, Mateusz and Halkiewicz, Stanisław},
  year         = {2026},
  url          = {https://github.com/joint-hubs/sce},
  note         = {Python software}
}
```

## Links

- [Source code](https://github.com/joint-hubs/sce)
- [Issue tracker](https://github.com/joint-hubs/sce/issues)
- [Documentation](https://joint-hubs.github.io/sce/)
- [Contributing](https://github.com/joint-hubs/sce/blob/main/CONTRIBUTING.md)
