Metadata-Version: 2.5
Name: binspect-regression
Version: 0.2.2
Summary: Binned scatterplots that audit the regression behind them
Project-URL: Homepage, https://github.com/joshuamyers22/binspect
Project-URL: Repository, https://github.com/joshuamyers22/binspect
Project-URL: Issues, https://github.com/joshuamyers22/binspect/issues
Project-URL: Changelog, https://github.com/joshuamyers22/binspect/blob/main/CHANGELOG.md
Author: Josh Myers
License: MIT License
        
        Copyright (c) 2026 Josh Myers
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: binscatter,diagnostics,econometrics,regression,visualization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Python: >=3.10
Requires-Dist: matplotlib>=3.7
Requires-Dist: numpy>=1.24
Requires-Dist: polars>=1.0
Requires-Dist: scipy>=1.10
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: hypothesis; extra == 'dev'
Requires-Dist: import-linter; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pandas-stubs; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: pypdf>=6.18.1; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest-mpl; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material; extra == 'docs'
Requires-Dist: mkdocstrings[python]; extra == 'docs'
Provides-Extra: dpi
Requires-Dist: binsreg>=3.2.1; extra == 'dpi'
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == 'pandas'
Provides-Extra: validation
Requires-Dist: statsmodels==0.15.0; extra == 'validation'
Description-Content-Type: text/markdown

# binspect

**Binned scatterplots for linear specification diagnostics.**

`binspect` estimates binned conditional means and compares them with a linear fit to
the underlying observations. The bin means are the fitted values from the saturated
model `OLS(y ~ C(bin))`. Their weighted deviations from the line provide a descriptive
linear specification diagnostic.

Version 0.2.2 is **Polars-native**, with pandas input compatibility.
Result tables are Polars DataFrames; use `result.to_pandas()` for an explicit
pandas projection. NumPy/SciPy handle numerical estimation. See the
[input and export contract](docs/INPUT_OUTPUT_CONTRACT.md) and
[compatibility policy and migration examples](docs/COMPATIBILITY.md).

The [user guide](docs/site/index.md) covers the supported workflows, with an
[API reference](docs/site/reference/estimation.md) generated by MkDocs from source.
Run `uv sync --frozen --all-extras` and `make docs` to execute examples and build
the site, then `uv run --frozen --all-extras mkdocs serve` to preview it locally.
The site is prepared for review; hosting/publication is not configured by this work.

The [reproducible gallery](docs/site/guide/gallery.md) includes seven seeded Polars
examples: linear, nonlinear, heteroskedastic, clustered, weighted, discrete x and
grouped controls, each explaining useful and misleading interpretations. Run
`uv run --frozen --all-extras python examples/gallery.py --output .work/gallery`
to retain figures and a manifest of seeds, versions and source/result hashes.

![binspect](docs/hero.png)

```python
import numpy as np
import polars as pl
import binspect

rng = np.random.default_rng(11)
n = 5000
age = rng.uniform(18, 70, n)
tenure = rng.uniform(0, 10, n)
df = pl.DataFrame(
    {
        "age": age,
        "tenure": tenure,
        "sales": 4 * np.log(age) + tenure + rng.normal(size=n),
        "region": np.where(np.arange(n) % 2, "east", "west"),
        "sample_weight": rng.uniform(0.5, 2, n),
        "firm_id": np.arange(n) // 50,
    }
)
bs = binspect.binscatter(df, y="sales", x="age", bins=20)

bs.table  # Polars: per-bin means, SDs, standard errors, intervals
bs.summary_frame()  # one-row model and diagnostic table
bs.to_pandas()  # optional pandas projection
bs.to_json()  # deterministic, versioned strict JSON
print(bs.summary())
bs.plot(theme="paper")
bs.audit(theme="paper")  # plot plus marginal distributions and residuals
```

Adjust both variables for numeric or categorical controls with FWL residualization:

```python
adjusted = binspect.binscatter(
    df,
    y="sales",
    x="age",
    controls=["region", "tenure"],
    bins=20,
    ci=None,  # adjusted bins are descriptive; slope uncertainty remains available
)
adjusted.fit.slope  # age coefficient from OLS(sales ~ age + region + tenure)
adjusted.plot()  # axes are explicitly labelled as adjusted
```

Residualized variables retain their original means, keeping the plot on a familiar
scale. Categorical controls are indicator-encoded and a constant is included
automatically. With `weights=`, the projection uses the same reliability weights.
FWL preserves the coefficient fitted to observation-level residuals. A regression
on the displayed bin means generally has a different slope.

Zero-weight observations are retained by default: they can affect bin boundaries,
unweighted bin counts, and stored descriptive arrays, but never point estimates or
degrees-of-freedom corrections. To make them fully equivalent to omitted rows, set
`zero_weight="drop"`:

```python
trimmed = binspect.binscatter(
    df, y="sales", x="age", weights="sample_weight", zero_weight="drop"
)
```

For comparisons across groups, pooled bin edges are used by default so facets refer
to the same intervals of `x`:

```python
comparison = binspect.compare(
    df,
    y="sales",
    x="age",
    group="region",
    bins=20,
)

comparison.table  # one row per group and bin
comparison.summary_frame()  # one row per group
comparison.plot(sharex=True, sharey=True)
```

Pass `common_bins=False` to select bins separately within each group. The pooled
estimate remains available as `comparison.pooled`.

When using `controls=`, set `common_bins=False`. Pooled and group fits adjust
separately and restore their own means, so their adjusted coordinates do not share
a common partition. Shared bins with controls raise an explicit error until a
common adjusted-coordinate contract is validated.

Tables include only occupied intervals. With shared bins, the `bin` ID and
`x_lo`/`x_hi` bounds refer to the same original interval across groups; IDs can
have gaps where a group has no observations. With independent bins, IDs are local
to each partition and should not be used to join groups as matching x ranges.
`binning.partition_edges` preserves the complete partition and
`binning.interval_ids` maps the compact estimation arrays to those intervals.
The older `binning.edges` remains a compressed partition that folds in empty
intervals; use the table or full partition for interval comparisons. JSON includes
the complete partition and interval IDs.

Use `cluster=` when observations share shocks within a firm, person, location, or
other sampling unit:

```python
clustered = binspect.binscatter(
    df,
    y="sales",
    x="age",
    controls=["region", "tenure"],
    cluster="firm_id",
    bins=20,
)
```

This applies CR1 cluster-robust standard errors to the fitted slope and, without
controls, bin means. Bin-mean intervals use a t reference distribution based on
the number of clusters represented in each bin. Bins containing fewer than two
positive-weight clusters have undefined intervals.

Few or highly uneven clusters can produce severe undercoverage even when CR1
arithmetic is correct. In a controlled development case with three clusters sized
480/60/60, nominal 95% bin intervals covered the population target only 78.2% of
the time. Cluster count alone is not a reliability guarantee; see the
[expanded coverage evidence](docs/expanded-coverage-review.md).

Unadjusted bin intervals are **approximate, pointwise and conditional on the
observed partition**. They omit uncertainty from choosing bins and provide no
simultaneous coverage guarantee.

With `controls=`, bin standard errors and confidence limits are unavailable
(NaN in tables, null in JSON) because fitted-control uncertainty is not validated.
Bin means, dispersion and slope uncertainty remain available. Requesting intervals
issues `AdjustedInferenceWarning`; pass `ci=None` for descriptive adjusted bins
without that warning. Controlled simulations found 87.4% coverage at nominal 95%;
see the [withdrawal decision](docs/adjusted-inference-boundary-review.md). An `x`
that adds no numerical rank beyond the controls on positive-weight observations
raises `InsufficientDataError`.

`bs.inference` (also in JSON) reports covariance, degrees of freedom and these
limitations. Classical slope SEs require their variance model; with weights this
is inverse-variance WLS. Independent weighted bin SEs instead use reliability
variance and effective sample size. HC1 is not supported by `binscatter`/`compare`;
the DPI selector's internal covariance setting does not change that. See the
[statistical analysis plan](docs/STATISTICAL_ANALYSIS_PLAN.md) for the exact contracts.

## Related packages


`binsreg` (Cattaneo, Crump, Farrell, and Feng) provides formal binscatter inference.
`binspect` delegates optimal bin selection and, through the separate adapter below,
pointwise function inference to it when requested. Use `binsreg` directly when
uniform confidence bands or formal shape-restriction tests are required.

## Adjusted function inference with binsreg

The optional adapter fits binsreg's function in the original x
coordinates, jointly with numeric or categorical controls:

```python
# Install this checkout with: pip install -e ".[dpi]"
function = binspect.binsreg(
    df, y="sales", x="age", controls=["region", "tenure"], bins="dpi"
)
function.dots  # degree-0 dot estimates
function.intervals  # pointwise limits and their own fitted centers
function.metadata  # target, control evaluation, covariance, actual method, issues
print(function.summary())
function.plot()
```

Controls are evaluated at positive-weight sample means by default. Use `at="zero"`
or an explicit vector in `metadata["control_columns"]` order to choose fixed
encoded-control values. The full coefficient covariance includes estimated-control
uncertainty (`asyvar=False`); uncertainty in the chosen evaluation values themselves
is omitted. The normal path uses degree-1 intervals and HC1 covariance. `weights=`
and `cluster=` pass through to binsreg after consistent complete-case filtering;
zero-weight rows are always dropped and counts are exported.

Use an integer `bins=` for a fixed count and `binning="equal_width"` for equal-width
spacing. Fixed counts can leave approximation bias; upstream warnings are exposed
as `BinsregWarning` and controlled issue codes. With few clusters, binsreg may
reduce bins and return constant-fit intervals. These have `limited_support`
status, explicit actual settings and **no few-cluster coverage guarantee**.
Fallback may also change knot placement; `actual_binning=None` reports that the
requested spacing is not certified in that path.
Unrecognized warnings or an unvalidated backend version give `unverified_method`.
The locked reference version is binsreg 3.2.1. See the
[adapter protocol](docs/BINSREG_ADAPTER_PLAN.md) for the target and coverage scope.
Prespecified development coverage at nominal 95% was 93.6% for adjusted iid DPI,
94.3% with 60 balanced clusters and 42.4% for the three-uneven-cluster fallback.
See the [results and limitations](docs/binsreg-adapter-review.md).

`BinsregResult` has separate copied dot/interval tables, metadata and strict-JSON
`to_dict()` output. It supplies function estimates without an FWL slope or gap
verdict. Its intervals do not apply to the residualized bins from `binscatter`.

## Result ownership

Use eager Polars or pandas dataframes, Series, or array inputs. All inputs align
by row position; pandas indexes do not trigger joins. Collect LazyFrames explicitly.
Polars tables are returned for both input backends. The pandas compatibility extra
is optional, and native estimation/export/plotting do not require pandas.
`result.to_pandas("summary")` and `result.to_pandas("decomposition")` provide the
other single-result tables; collections also support the summary conversion.

Results own snapshots of their numeric data. Changing input arrays, dataframe
columns, weights or custom edges after estimation does not change the result.
`result.x`, `result.y`, `result.weights`, and arrays in `binning` and `estimates`
are read-only. Use `result.x.copy()` when you need an editable array; to change
an estimate, run estimation again with the changed inputs. Array access shares
immutable numeric storage but returns a fresh array header, so changing a
returned array's shape or dtype also leaves the result intact.

Grouped results own a read-only copy of the group mapping and share immutable
single-result objects. Use immutable hashable group labels, such as strings,
numbers or dates; mutable custom label objects are outside this contract.
Tables, summaries, inference dictionaries and `to_dict()` exports are independent
editable projections. The binsreg adapter similarly returns copied dot/interval
tables and nested metadata. Public result access does not expose writable stored
numeric data; deliberate private-attribute or native-memory tampering is outside
this API contract.

Snapshot construction copies the retained numeric buffers once per container;
reading an array does not copy its values. This trades memory for stable results.
See the [ownership verification and memory measurements](docs/result-ownership-review.md).

## Versioned exports and provenance

`to_dict()` and deterministic `to_json()` exports include schema version 1,
input/retained/dropped row counts, encoded control identity and coordinates,
selection provenance, inference limitations and diagnostic policies. Undefined
numeric values encode as JSON null. Raw observations are omitted.

`to_evidence(provenance=..., exported_at=...)` adds caller-supplied references to
the analysis plan, inputs, software lock and code. References are never fetched,
read or hashed implicitly. An optional caller timestamp stays outside the
deterministic payload. See the [schema, examples and migration guide](docs/INPUT_OUTPUT_CONTRACT.md).

## Choosing bins with DPI

For `binscatter`/`compare`, install `binspect-regression[dpi]` and use `bins="dpi"` for binsreg's direct
plug-in count for a piecewise-constant fit. Both quantile and equal-width spacing
are supported. Selection uses the full retained sample with mass-point checks.
It currently requires no `weights`, `controls`, or `cluster`; those combinations
raise an error rather than passing an incomplete specification to the selector.
Use an integer, custom edges, or `bins="auto"` for those estimates.

If DPI cannot produce a finite integer count between two and the retained sample
size, selection raises `InvalidBinningError`. There is no rule-of-thumb fallback
or silent count clipping. binspect constructs its own edges and may merge bins
for tied/discrete x; it does not promise identical knots or intervals to binsreg.

`bs.bin_rule`, `bs.binning.requested_bins`, and `bs.n_bins` distinguish the rule,
selected count, and realized count. JSON and summary exports include this metadata
and a `None` fallback. Groups sharing pooled edges report `bin_rule="pooled"` and
the originating rule in `bs.binning.source_rule`; the pooled result retains the
original selection count.

## What it draws

The default plot presents the estimates, uncertainty, linear fit, lack of fit, and
distribution of the exogenous variable as separate layers.

| Layer | What it shows | Default |
|---|---|---|
| `bins` | Bin means — the saturated-model fitted values | on |
| `ci` | Confidence bar per bin mean | on |
| `fit` | OLS line through the underlying data | on |
| `deviation` | Signed departures between bin means and the line; not squared gap or area | on |
| `rug` | x-density, so quantile bins can't hide their own imbalance | on |
| `sd_line` | Signed SD reference; OLS slope equals SD slope times `abs(r)` | off |
| `smooth` | Local-linear smoother through the bin means | off |
| `raw` | Underlying observations at low alpha | off |

Three themes are included: `notebook` (default), `paper` (thin, serif), and `deck`
(larger marks and type). Themes are scoped; importing `binspect` does not modify
global `rcParams`. D2 adds local PNG/PDF/SVG checks and
[grayscale/color-vision review with explicit limits](docs/site/guide/figure-exports.md).
Existing presets are unsuitable for dark figure/axes backgrounds; general
accessibility and maintainer visual acceptance remain open.
See the [public plotting inventory](docs/API_INVENTORY.md) for
layer options, draw order and return types.

Use `bs.audit()` for a composed diagnostic figure with the unchanged binscatter in
the central panel, marginal histograms, and OLS residuals against fitted values.
Either companion view can be omitted with `marginals=False` or `residuals=False`.
These panels describe the stored estimate; they do not add a formal specification
test.

## One thing to know about η²

The bin-indicator model does not nest the linear model. Consequently, η² can be below
the linear R² when bins are coarse, and their difference is not a valid curvature
measure. `binspect` reports normalized lack of fit,

```
SS_lof = Σⱼ nⱼ (ȳⱼ − ŷ(x̄ⱼ))²      gap = SS_lof / SS_total
```

which is nonnegative by construction and corresponds to the deviations shown in the
plot. This quantity is descriptive and is not a formal test of linearity.

## Diagnostic policy

Verdicts are configurable descriptive heuristics. The defaults use a gap threshold
of 0.02 and require at least 30 effective rows in every bin. `linear` means the gap
falls below that chosen cutoff; it is not evidence from a specification test.
`limited support` replaces the former `underpowered bins` label. Constant outcomes
and clustered results without an explicit cluster threshold are `not assessed`.

```python
policy = binspect.DiagnosticPolicy(gap_threshold=0.05, min_bin_effective_n=40)
screened = binspect.binscatter(df, y="sales", x="age", diagnostic_policy=policy)
descriptive = binspect.binscatter(df, y="sales", x="age", diagnostic_policy=None)
```

`compare` applies the same policy to pooled and group results. An explicit
`min_bin_clusters` enables clustered classification using both cluster and effective
row thresholds; it does not validate confidence coverage. Policy values, support
minima and decision reasons are exported in decomposition/summary records.
`n_obs` and bin `n` count retained rows; `n_positive` and `n_effective` separately
report positive-weight and Kish effective rows. Cluster counts remain separate.
Retained zero-weight rows cannot supply diagnostic support.

The signed SD reference obeys `OLS slope = abs(correlation) * SD slope`, including
negative relationships. At exactly zero covariance its orientation is positive;
constant y gives zero slope. Deviation marks show signed departures; displayed
area or length does not equal the weighted squared gap.

## Status

Package version is `0.2.2`. The 0.2 series makes Polars tables, versioned exports,
result ownership changes and the optional function adapter available to callers. See the
[compatibility policy](docs/COMPATIBILITY.md) for supported options and migration.
The distribution name is `binspect-regression`; the import remains `binspect`.

**Not yet implemented:** uniform confidence bands and quantile regression.
Without controls, weights or clusters, bin standard errors are `sd/√n` and assume
independent observations. Weighted/clustered conventions and the adjusted-bin
uncertainty restriction are described above.

For workload measurements, see [performance and limits](docs/site/guide/performance.md).
Clustered bin scores use occupied-pair storage rather than dense bin-by-cluster
arrays. The measured grid covers 10k–1M synthetic Polars rows; control width,
group count and plot layers still affect cost. Baseline/runner acceptance remains
pending, and 10M rows are unqualified. Run `make benchmark` separately from the
portable checks on the recorded host.

## Install

```bash
python -m pip install -e .
```

Run that command from the checked-out repository for native Polars use. Install
`.[pandas]` for explicit pandas projections or `.[dpi]` for optional binsreg methods.
To run every README/guide example, use `uv sync --frozen --all-extras` and `make docs`.
Python ≥3.10 is required; isolated minimum/current checks and the Python 3.10–3.13
scope are described in [dependency configurations](docs/site/guide/dependencies.md).
The DPI extra requires binsreg ≥3.2.1. Dependency qualification is described in
the compatibility policy. NumPy/SciPy, Polars and Matplotlib are required; pandas
is optional for native use.

Install the published package with `python -m pip install binspect-regression` and
continue to write `import binspect`. See [CONTRIBUTING.md](CONTRIBUTING.md) for
development checks.

For contributing, release checks, and development conventions, see
[CONTRIBUTING.md](CONTRIBUTING.md). Please report vulnerabilities privately as
described in [SECURITY.md](SECURITY.md).

Maintainer release instructions are in [RELEASING.md](RELEASING.md).

## License

MIT. Dependencies retain their own licenses. See the
[supply-chain scope and approved license decisions](docs/SUPPLY_CHAIN.md),
including the optional binsreg backend's GPL-3.0-only metadata.

## Citation

The methodology this package leans on is Cattaneo, M. D., Crump, R. K., Farrell,
M. H., & Feng, Y. (2024). "On Binscatter." *American Economic Review*, 114(5),
1488–1514. If you use binned scatterplots for inference, cite that paper and consider
using `binsreg` directly.
