Metadata-Version: 2.4
Name: gsrfs
Version: 0.16.2
Summary: Research prototype for geometry-supported residual unsupervised feature selection
License-Expression: MIT
Keywords: feature-selection,unsupervised-learning,machine-learning,dimensionality-reduction,scikit-learn
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: scikit-learn>=1.3
Requires-Dist: pandas>=2.0
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Provides-Extra: benchmark
Requires-Dist: openml>=0.15; extra == "benchmark"
Requires-Dist: statsmodels>=0.14; extra == "benchmark"
Dynamic: license-file

# GSR-FS v0.16 — Geometry-Supported Residual Feature Selection

**Status: research candidate with a completed 30-dataset heterogeneous canonical evidence panel. Minimum evidence/provenance gate PASS; strong superiority FAIL; formal OpenML-CC18 and FGMRW-2026 empirical gates pending. Not yet released to PyPI.**

GSR-FS is a fully label-blind unsupervised feature-selection framework. Each feature is represented
by its additive contribution to pairwise sample geometry. Structural support is calibrated with
within-feature permutations, geometry concentrated on very few observations is downweighted by
Sample-Geometric Coverage (SGC), and a non-negative residual term discourages selecting geometry
already represented by previously selected features.

The selector never uses `y`, class labels, or the ground-truth number of classes.

## Core score

For feature `j` and observations `a,b`,

\[
g_j(a,b)=(x_{aj}-x_{bj})^2.
\]

The frozen greedy score is

\[
Score(j\mid S)=EffectiveSupport_j\;R(j\mid S)^{0.5},
\]

where `EffectiveSupport` is permutation-calibrated structural support multiplied by SGC and
`R(j|S)` is the fraction of feature-geometry not explained by a non-negative projection onto the
already selected geometry profiles.

`residual_power = 0.5` was frozen in v0.5 before later external evidence and remains unchanged through v0.16.

## Install the research wheel

```bash
pip install gsrfs-0.16.0-py3-none-any.whl
```

## Minimal use

```python
from gsrfs import GSRSelector

selector = GSRSelector(
    n_features="auto",
    n_permutations=200,
    alpha=0.05,
    random_state=42,
)

X_selected = selector.fit_transform(X)
print(selector.get_support(indices=True))
print(selector.get_feature_names_out())
print(selector.get_feature_report())
```

`y` is accepted only for scikit-learn estimator compatibility and is ignored.

## What changed in v0.7

The feature-selection objective is **unchanged** from the frozen v0.6 estimator. v0.7 adds public-use and publication-safety infrastructure around it.

### 1. Command-line interface

After installation, numeric CSV/NPY matrices can be selected without writing Python:

```bash
gsrfs-select data.csv --exclude-columns id --n-features auto \
  --output selected.csv --report gsr_report.json
```

The CLI never accepts a target variable. Its JSON report explicitly records `selection_used_y=false`.

### 2. External author-code provenance schema

`gsrfs.external` defines a strict JSON contract for rankings generated by MATLAB/Python/R author code. It requires an explicit complete zero-based ranking, feature count, dataset identifier, source metadata, and flags declaring whether labels or the ground-truth class count were used. A benchmark can additionally bind the ranking to the exact feature matrix with a SHA-256 fingerprint.

### 3. Automated publication gate

`evaluate_publication_gate()` checks minimum external-dataset coverage, number of competitors, presence of canonical/author-code methods, method failure rates, GSR label-blindness audit evidence, frozen hyperparameters and canonical source locking. The heterogeneous 30-dataset minimum-evidence/provenance gate now **passes**. This does not imply superiority: the strong-superiority gate remains **FAIL**, and the formal OpenML-CC18 standardized-suite gate remains pending.

### 4. Benchmark implementation provenance

Publication-style result rows now distinguish `native_or_local`, `canonical`, and `author_code` implementations, preventing local development baselines from being mislabeled as canonical results.

### 5. v0.6 stochastic/runtime improvements retained

#### Stochastic reproducibility audit

Pair sampling and permutation calibration now use independent RNG substreams. A common bank of
sample-index permutations is reused across features. This removes a pair-budget/permutation
confound and makes the Monte-Carlo statistic equivariant to a pure reordering of feature columns
(up to deterministic tie handling).

#### Balanced nested pair sampling

When all `n(n-1)/2` pairs cannot be used, the new default `pair_sampler="balanced"` uses a
randomized round-robin edge schedule. Sample endpoint counts are nearly equal, pair samples are
unique, and smaller pair budgets are exact prefixes of larger budgets for the same seed.

Against exact all-pairs geometry on Iris, Wine and Breast Cancer, mean convergence was:

| pairs | balanced Jaccard | balanced rank rho | random Jaccard | random rank rho |
|---:|---:|---:|---:|---:|
| 5,000 | 0.611 | 0.802 | 0.325 | 0.606 |
| 10,000 | 0.652 | 0.910 | 0.448 | 0.849 |
| 20,000 | 0.778 | 0.948 | 0.714 | 0.888 |
| 50,000 | 0.867 | 0.974 | 0.867 | 0.970 |

These results support balanced sampling as the default approximation layer, but they also show that
sampled-pair feature subsets can still differ from exact all-pairs results in moderate-dimensional
data. Publication runs should therefore report the pair budget and sensitivity checks.

#### Residual runtime cliff removed

The auto residual engine now considers both feature count and pair count. On Breast Cancer with
10 permutations and 8 selected features, the batched coordinate-descent solver produced the same
selection path as SciPy NNLS with maximum score differences below `2.5e-14`, while runtime was:

| pairs | NNLS s | batched-CD s | speedup |
|---:|---:|---:|---:|
| 10,000 | 0.247 | 0.358 | 0.69x |
| 20,000 | 6.256 | 0.640 | 9.78x |
| 50,000 | 6.446 | 1.274 | 5.06x |

The auto rule therefore keeps NNLS for small geometry matrices and switches to batched-CD for tall
or wide geometry matrices.

#### Canonical comparator bridges

`ScikitFeatureLaplacianSelector`, `ScikitFeatureSPECSelector`, and
`ScikitFeatureMCFSSelector` call an independently installed scikit-feature implementation. GSR-FS
does **not** vendor that GPL source. `CANONICAL_SOURCE_LOCK.json` records the external file blob
SHAs used for validation.

The canonical MCFS bridge is explicitly marked `ranking_depends_on_k=True`, because its LARS
cardinality constraint depends on the requested number of selected features. The flexible benchmark
runner therefore re-fits canonical MCFS for every fixed `k` instead of incorrectly reusing a single
ranking.

#### Frozen OpenML-CC18 manifest

`CC18_TASKS.json` freezes all 72 task IDs from OpenML-CC18 (suite 99). The v0.6 CC18 runner uses
that manifest, uniform label-independent row subsampling, fixed X-only preprocessing, checkpoints,
and either development or canonical baseline sets.

## Frozen automatic-selection calibration retained from v0.6

With `alpha=0.05`, 100 permutations, balanced 5,000-pair sampling and frozen `gamma=0.5`:

- **200 independent-null runs:** 13/200 runs selected at least one feature, empirical FWER **0.065**;
  exact Clopper-Pearson 95% CI **0.035–0.109**, which contains the nominal 0.05.
- **30 redundant-signal runs:** exactly 2 selected features in every run; mean latent-group coverage
  **1.00**, relevant precision **1.00**, redundancy **0.00**.
- **15 coherent-outlier diagnostic runs:** mean **21.8** selected features, coverage **1.00** but
  relevant precision only **0.092**. This remains a major identifiability limitation: a fully
  unsupervised method cannot generally know whether coherent rare structure is nuisance
  contamination or a genuine rare subgroup without another modeling assumption.

The null result is consistent with approximate 5% family-wise calibration but is **not a proof of
finite-sample FWER control**.

## Frozen small real-data screen retained from v0.6

On Iris, Wine and Breast Cancer (two seeds, dataset-balanced aggregation, local development
comparators), mean NMI was:

- Laplacian Score development implementation: **0.729**
- **GSR-FS: 0.660**
- label-blind MCFS development implementation: **0.622**
- MaxVariance: **0.558**
- SPEC development implementation: **0.449**

GSR-FS therefore remains competitive in the small screen but does **not** beat Laplacian Score.
These are not publication-grade SOTA results because the real-data set is tiny and the development
comparators are not the final author-code/canonical benchmark.

## Claim gates

Three claim levels are deliberately separated:

1. **Minimum evidence/provenance — PASS.** The heterogeneous source-audited panel contains 30 datasets and real canonical scikit-feature comparators.
2. **Strong superiority/replacement — FAIL.** Friedman and Holm-corrected comparisons do not establish overall dominance, so do not claim that GSR-FS replaces, outperforms, or is state of the art.
3. **Formal standardized-suite — PENDING.** The heterogeneous panel must not be described as an official OpenML-CC18 suite-99 execution. The pinned GitHub Actions workflow is prepared for that run.

Any future standardized-suite or recent-author result must use the frozen objective and source-integrity rules without retuning on final labels.

## Research files

- `V0.7_RESEARCH_REPORT.md`
- `V0.6_RESEARCH_REPORT.md`
- `PUBLICATION_PROTOCOL.md`
- `HYPERPARAMETER_FREEZE.md`
- `BASELINE_VALIDATION.md`
- `CANONICAL_SOURCE_LOCK.json`
- `CC18_TASKS.json`
- `PRIOR_ART_NOTES.md`
- `METHOD_REGISTRY.json`
- `PUBLICATION_GATE_CURRENT.json`
- `EXTERNAL_RANKING_SCHEMA.md`
- `benchmarks/results/v06/`

## License

MIT for GSR-FS code. Third-party implementations are not bundled and remain subject to their own
licenses.

## v0.8 offline evidence update

v0.8 keeps the GSR-FS mathematical objective frozen and adds a fold-wise, non-transductive downstream
validation protocol. Seven real datasets are available offline; three (ANES96, ModeChoice, Spector) are an
extension beyond the datasets used for the residual-power development decision.

Across the seven-dataset screening panel, GSR-FS achieved mean balanced accuracy 0.7126, compared with
0.7289 for the local MCFS-eigengap development baseline, 0.7065 for MaxVariance, 0.6875 for SPEC-dev, and
0.6742 for LaplacianScore-dev. The Friedman test was non-significant (p≈0.590). These local comparators are
**not canonical third-party implementations**, and this small panel does not clear the publication claim gate.

See `V0.8_RESEARCH_REPORT.md`, `OFFLINE_PANEL_FREEZE.md`, `OFFLINE_PANEL_MANIFEST.json`, and
`PUBLICATION_GATE_CURRENT_V08.json` for the exact protocol and boundaries.


## v0.9 experimental partition branch

The frozen `GSRSelector` remains unchanged. v0.9 adds an **experimental fixed-k** selector,
`GSRPartitionSelector`, that fuses structural-support rank with a new marginal
**Persistent Balanced-Gap Evidence (PBGE)** rank. PBGE scans interior order-statistic gaps and rewards
only gaps that are balanced and remain large relative to neighboring spacings across multiple local windows.
It uses no labels, pseudo-labels, mutual information, information gain, classifier, Hartigan Dip statistic,
Hopkins statistic, or MST clustering.

Synthetic-only development selected `partition_weight=0.35`: mean latent-group recovery increased from
0.771 for frozen GSR to 0.813 for GSR-PBGE, while redundancy fell from 0.021 to 0.000 in the four-scenario
development panel. However, a fresh three-dataset downstream diagnostic (Fair, CCard, Heart) did **not**
confirm a general improvement: mean balanced accuracy was 0.610 for GSR-PBGE versus 0.642 for frozen GSR.
PBGE improved CCard but degraded Fair and especially Heart.

Therefore PBGE is **not promoted into the main GSR-FS objective**. It remains a documented experimental
branch and negative/partial result. v0.9 supports fixed `n_features` only; automatic PBGE stopping is
deliberately disabled because a separate null model for the marginal partition term has not been validated.


## v0.10 experimental stability branch

The frozen `GSRSelector` objective remains unchanged. v0.10 adds `GSRConcordanceSelector`, an experimental fixed-k branch using **Jackknife Geometry Concordance (JGC)** only as a near-tie resolver. Samples are divided into disjoint blocks, leave-block-out geometry ranks are computed without labels, and a conservative lower-envelope rank is used when candidate GSR scores fall within a frozen 10% near-optimal band.

Synthetic-only confirmation across redundant blocks, three-factor blobs and nonlinear moons increased mean exact-feature subsample Jaccard from **0.353** (frozen GSR) to **0.420** (GSR-JGC), while mean latent-group coverage increased from **0.978** to **1.000**. However, a fresh Titanic fold-wise diagnostic was neutral: both GSR-base and GSR-JGC achieved mean balanced accuracy **0.752** and mean fold-selection Jaccard **0.511**.

Therefore JGC is **not promoted to the default GSRSelector**. It remains experimental. See `V0.10_RESEARCH_REPORT.md`, `JGC_FREEZE.md`, and `JGC_PRIOR_ART_BOUNDARY.md`.


## v0.11 contextual substitution diagnostic

The frozen `GSRSelector` remains unchanged. v0.11 adds `GSRSubstitutionSelector`, which performs the same GSR feature selection and then builds **diagnostic Contextual Geometry Substitution (CGS) groups** around selected representatives. An unselected feature is grouped only when it can both be represented by the selected geometry basis and replace a representative without materially increasing non-negative pair-geometry reconstruction error.

The CGS threshold `0.85` was frozen from synthetic + independent-null development before a fresh real label-free panel. In the synthetic threshold study, exact-column Jaccard was 0.400 versus substitution-aware Jaccard 0.987, with 100% known redundant-group recovery and 0% noise assignment. In 50 independent-null datasets there were no substitution assignments.

On a fresh label-free real structural panel, mean exact Jaccard across Penguins/MPG/Diamonds was **0.712**, while CGS-aware Jaccard was **0.908**. Penguins was neutral; MPG grouped `cty` with `hwy`, and Diamonds grouped `x/y/z`. These results support the narrower interpretation that some apparent GSR instability is representative swapping within redundant information groups. They do **not** establish a new general feature-grouping method or improve downstream performance, because the selected subset is intentionally unchanged.

Python:

```python
from gsrfs import GSRSubstitutionSelector

sel = GSRSubstitutionSelector(
    n_features=5,
    substitution_threshold=0.85,
    random_state=42,
)
sel.fit(X)
print(sel.get_feature_names_out())       # same frozen GSR selection
print(sel.get_substitution_groups(names=True))
print(sel.get_group_report())
```

CLI:

```bash
gsrfs-select data.csv --n-features 5 --diagnose-substitutes \
  --substitution-threshold 0.85 --report report.json
```

See `V0.11_RESEARCH_REPORT.md`, `CGS_FREEZE.md`, and `CGS_PRIOR_ART_BOUNDARY.md`.


## v0.12 structural-group output

The frozen `GSRSelector` objective is still unchanged. v0.12 adds
`GSRStructuralGroupSelector`, an interpretation layer built on the frozen v0.11
Contextual Geometry Substitution (CGS) diagnostic. Each selected feature is
reported as a **structural-group representative** together with contextual
substitutes, while `transform()` continues to return only the original selected
columns. No averages, projections, latent variables, synthetic components, or
feature-extraction outputs are created.

Example:

```python
from gsrfs import GSRStructuralGroupSelector

sel = GSRStructuralGroupSelector(
    n_features=4,
    substitution_threshold=0.85,
    random_state=42,
)
sel.fit(X)
print(sel.get_feature_names_out())       # unchanged frozen GSR representatives
print(sel.get_structural_groups())       # SG01, SG02, ... + contextual substitutes
print(sel.get_structural_group_report())
print(sel.get_group_summary())
```

CLI:

```bash
gsrfs-select data.csv --n-features 4 --structural-groups --report groups.json
```

The frozen CGS diagnostic default is now consistently `0.85` in the Python class
and CLI. Earlier v0.11 package documentation froze 0.85, but the class constructor
still had a legacy 0.95 default; v0.12 corrects that metadata/API inconsistency.
This does not change the core GSR selected subset.

### v0.12 group-output audit

A fresh audit used 30 independent-null datasets and 20 known redundant-block
synthetic datasets. At threshold 0.85, the null panel produced **0 false
substitution assignments**, while the redundant-block panel recovered **100%**
of known groups with **0 false assignments**.

A fresh, label-free real structural panel produced:

| Dataset | Exact Jaccard | Group-aware Jaccard | Gain | Non-singleton groups |
|---|---:|---:|---:|---:|
| Longley exogenous variables | 0.485 | 0.823 | +0.338 | 1 |
| Macrodata numeric variables | 0.727 | 0.924 | +0.197 | 1 |
| Diabetes features | 0.598 | 0.598 | 0.000 | 0 |

The neutral Diabetes result is important: structural-group output does not force
a group when the frozen CGS threshold finds no contextual substitutes. Longley
and Macrodata groups should be interpreted as geometry-level substitution under
the fitted dataset, not as causal, semantic, or universally interchangeable
variables.

See `V0.12_RESEARCH_REPORT.md`, `STRUCTURAL_GROUP_OUTPUT.md`, and
`SGO_PRIOR_ART_BOUNDARY.md` for the exact scope and evidence.

## v0.13 evidence-integrity gate

v0.13 adds a dataset source-integrity contract for external benchmarking. A dataset counts
toward the 30+ external-dataset publication gate only when its source audit is `PASS`.
Target-informed pre-selector operations (for example classifier-based feature filtering or
response-informed row reduction) make a dataset ineligible even if the downstream GSR-FS call
itself never receives `y`.

```python
from gsrfs import (
    build_source_integrity_record,
    audit_source_integrity_manifest,
    evaluate_publication_gate,
)

record = build_source_integrity_record(
    dataset="task31_credit-g",
    source_uri="https://www.openml.org/t/31",
    feature_filtering_used_y=False,
    row_sampling_used_y=False,
)
audit = audit_source_integrity_manifest([record])
```

See `SOURCE_INTEGRITY_POLICY.md` and `EXTERNAL_SOURCE_AUDIT_V013.json`.

## v0.14 canonical evidence milestone

v0.14 leaves the frozen GSR-FS objective unchanged and adds real canonical/author-code evidence. Independently fetched upstream `scikit-feature` code was executed for Laplacian Score, SPEC, and MCFS; that GPL source is not vendored in the MIT package.

The heterogeneous source-audited evidence panel now contains **30 datasets**. Canonical comparisons are complete on **29 datasets**. On balanced accuracy, GSR-FS has mean **0.6031** and mean rank **2.862**, versus SPEC mean **0.6019** and mean rank **2.690**. Friedman `p=0.488`; no GSR-vs-canonical comparison is Holm-significant.

Accordingly:

- `PUBLICATION_GATE_CURRENT_V014.json`: **PASS** (minimum evidence/provenance only);
- `SUPERIORITY_GATE_CURRENT_V014.json`: **FAIL**;
- `STANDARD_BENCHMARK_GATE_CURRENT_V014.json`: **FAIL** (the 30-dataset panel is not a formal CC18 suite-99 run).

Do **not** describe v0.14 as state of the art, statistically superior, or a proven replacement for existing UFS methods. The supported statement is that frozen GSR-FS is competitive with classic canonical baselines on the current heterogeneous evidence panel while optimizing a distinct label-free structural objective.

See `V0.14_RESEARCH_REPORT.md` for details.

## v0.16 standardized-suite and recent-author infrastructure

v0.16 leaves the frozen GSR-FS objective unchanged. The release adds two network-reproducible evidence paths:

1. **Official OpenML-CC18 suite 99 workflow.** `.github/workflows/cc18_v016.yml` installs the pinned scikit-feature commit, runs the frozen 72-task manifest, records source-integrity for every accepted task, and finalizes a formal standardized-suite gate.
2. **FGMRW-UFS TKDE 2026 bridge.** `FGMRWAuthorSelector` downloads a pinned public author-source snapshot only when explicitly used for benchmarking, verifies Git blob SHA-1 values, performs training-X-only MinMax scaling, and invokes the author algorithm function with no labels or class counts. No FGMRW source is vendored.

Current status: the heterogeneous 30-dataset minimum-evidence gate remains **PASS** from v0.14; the strong superiority gate remains **FAIL**; the formal CC18 and recent-author execution gates remain **not yet executed** because the present runtime has no external DNS/network access. See `V0.16_RESEARCH_REPORT.md`, `STANDARDIZED_SUITE_PROTOCOL_V016.md`, and `FGMRW_2026_SOURCE_AUDIT_V016.md`.


## v0.16 CI-readiness release

v0.16 leaves the frozen GSR-FS method unchanged and fixes/validates the networked standardized-benchmark workflow before GitHub upload. A static preflight and workflow-contract test now verify result paths, pinned commits, source-vendoring boundaries, and required runner files. See `V0.16_RESEARCH_REPORT.md` and `GITHUB_REPOSITORY_BOOTSTRAP_V016.md`.
