Metadata-Version: 2.4
Name: pysmartcor
Version: 1.2.0
Summary: Correlation methods matched to variable types, with full inference
Author-email: "M. Harshvardhan" <harshvardhan@aus.edu>, Pritam Ranjan <pritamr@iimidr.ac.in>
Maintainer-email: "M. Harshvardhan" <harshvardhan@aus.edu>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://harshvardhaniimi.github.io/smartcor/
Project-URL: Documentation, https://harshvardhaniimi.github.io/smartcor/
Project-URL: Repository, https://github.com/harshvardhaniimi/smartcor
Keywords: correlation,statistics,polychoric,ordinal,categorical
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: src/pysmartcor/LICENSES/semopy.txt
Requires-Dist: numpy>=1.21
Requires-Dist: scipy>=1.9
Requires-Dist: pandas>=2.1
Requires-Dist: statsmodels>=0.14
Provides-Extra: viz
Requires-Dist: matplotlib>=3.5; extra == "viz"
Requires-Dist: seaborn>=0.12; extra == "viz"
Dynamic: license-file

# pysmartcor

`pysmartcor` detects variable types and selects a suitable correlation method for each pair. It supports continuous, count, binary, ordinal, and categorical variables and returns the estimate, inference, selected method, and rationale.

Before choosing a method, it checks the pair for outliers, normality, latent normality, and linearity. It chooses among 18 methods, following the decision framework of Harshvardhan and Ranjan (2026). The R package [smartcor](https://harshvardhaniimi.github.io/smartcor/) implements the same rules.



## Install

The package is on [PyPI](https://pypi.org/project/pysmartcor/):

```bash
pip install pysmartcor
```

Install the optional plotting dependencies when needed:

```bash
pip install 'pysmartcor[viz]'
```

## Quick start

```python
import numpy as np
from pysmartcor import smart_cor

rng = np.random.default_rng(42)
education_years = rng.integers(8, 22, size=300)
income = 1500 * education_years + rng.normal(0, 8000, size=300)

result = smart_cor(income, education_years, x_name="income", y_name="education_years", verbose=False)
print(result)
```

Inspect `result.method`, `result.rationale`, and `result.diagnostics` to see how the diagnostics informed the choice.

## Load the example data

The package includes `gss_2024_casestudy.csv`. This CSV is frozen for reproducibility. The examples below read the bundled copy and do not download data.

```python
from importlib.resources import as_file, files
import pandas as pd

csv_resource = files("pysmartcor").joinpath("data/gss_2024_casestudy.csv")
with as_file(csv_resource) as csv_path:
    gss = pd.read_csv(csv_path)
```

## Correlate one pair

```python
from pysmartcor import smart_cor

result = smart_cor(
    gss["coninc"],
    gss["sex"],
    x_name="coninc",
    y_name="sex",
    verbose=False,
)

print(result)
result.estimate
result.method
result.p_value
result.ci_lower, result.ci_upper
result.diagnostics
```

Income is far from normal within each sex, so the Jarque-Bera test rejects and the rank-biserial correlation is selected.

Set `assume_latent_normal` to control pairs that can use a latent-variable method:

```python
smart_cor(gss["degree"], gss["happy"], assume_latent_normal="auto")
smart_cor(gss["degree"], gss["happy"], assume_latent_normal=True)
smart_cor(gss["degree"], gss["happy"], assume_latent_normal=False)
```

The default, `"auto"`, runs the likelihood-ratio test of latent normality for each affected pair. `True` and `False` fix the answer; the outlier and Jarque-Bera checks still run. A binary-by-binary table is saturated, so automatic selection uses phi; `True` gives tetrachoric and `False` gives Yule's Q.

## Diagnostics

Each diagnostic is also available on its own. All tests use `alpha=0.05` unless you pass another value to `smart_cor()`, `smart_cormat()`, `smart_cor_df()`, or `compare_methods()`.

```python
from pysmartcor import detect_outliers, test_bivariate_normality, test_linearity

detect_outliers(gss["coninc"])                       # outer fences, Q1 - 3 IQR and Q3 + 3 IQR
test_bivariate_normality(gss["age"], gss["coninc"])  # Mardia's test
test_bivariate_normality(gss["coninc"], gss["degree"])  # Jarque-Bera and the LR test
test_bivariate_normality(gss["degree"], gss["happy"])   # LR test of the threshold model

diagnostics = test_linearity(gss["coninc"], gss["age"])
diagnostics.decision
diagnostics.p_value
```

`result.diagnostics` is a dictionary with the keys `outliers`, `normality`, `latent`, and `linearity`; a test that did not run is `None`. The tidy output summarises them in its last four columns: `outlier_flag`, `normality_p`, `latent_p`, and `linearity_p`.

For two continuous/count variables without outliers, `smart_cor()` runs `test_linearity()`. The test regresses each variable on the other, orders the residuals by the regressor (averaging residuals at tied values), and applies the Box-Pierce, Ljung-Box, BDS, Breusch-Pagan, and RESET rule in each direction. A direction rejects linearity when a residual structure test and RESET both reject. Spearman is selected only when both directions reject; otherwise Pearson is retained. Each direction's p-value is `max(min(p_bp, p_lb, p_bds), p_reset)`, the combined `p_value` is the larger of the two, and the BDS p-value is the minimum over the distance thresholds. An explicit `method` skips all diagnostics.

Because both directions are tested and tied values are averaged, the decision does not depend on argument order or row order. An inconclusive result retains Pearson and reports the reason. When the rule recommends robust/bootstrap Pearson inference, `bootstrap="auto"` uses a paired-bootstrap interval; the p-value remains the standard Pearson t-test. `bootstrap=False` disables that bootstrap. The diagnostics use `statsmodels`; BDS uses the sample and distance conventions of R's `fNonlinear::bdsTest(res, m=3)`.

## Build a matrix

```python
from pysmartcor import smart_cormat

columns = ["age", "coninc", "degree", "happy", "sex", "region"]
matrix = smart_cormat(gss[columns], rng=1, verbose=False)

matrix.correlations
matrix.methods
matrix.types
matrix.to_long()
```

Use `smart_cor_df()` when a script needs plain pandas DataFrames:

```python
from pysmartcor import smart_cor_df

plain = smart_cor_df(gss[columns])
plain["correlations"]
plain["methods"]
```

Set `nearPD=True` when downstream analysis requires a positive-definite matrix:

```python
adjusted = smart_cormat(gss[columns], nearPD=True, verbose=False)
adjusted.correlations
adjusted.original_correlations
adjusted.nearPD
plain = smart_cor_df(gss[columns], nearPD=True)
```

The default is `nearPD=False`. A NumPy implementation follows `Matrix::nearPD(..., corr=TRUE)`, including Dykstra's correction, default tolerances, and the final eigenvalue adjustment. It retains a diagonal of ones. Methods, pairwise details, and `to_long()` retain the original estimates and inference. Missing or infinite matrix entries must be resolved before adjustment.

## Compare methods

```python
from pysmartcor import compare_methods

comparison = compare_methods(
    gss["degree"],
    gss["happy"],
    bootstrap=False,
    verbose=False,
)
comparison.results
```

## Plot a matrix

Plotting requires the optional `viz` dependencies.

```python
from pysmartcor.viz import cor_heatmap, method_heatmap

cor_heatmap(matrix)
method_heatmap(matrix)
```

## Method selection

| Variable types | Default method | Alternative |
|:--|:--|:--|
| *Correlation methods (signed, from -1 to +1)* | | |
| Continuous + continuous | Pearson (linear, no outliers) | Spearman (outliers or nonlinear), Kendall's tau |
| Continuous + binary | Biserial (if latent normality holds) | Point-biserial; rank-biserial (outliers or non-normal) |
| Binary + binary | Pearson = phi | Tetrachoric (latent normal), Yule's Q (no latent continuum) |
| Continuous + ordinal | Polyserial (if latent normality holds) | Kendall's tau (m <= 5) or Spearman |
| Ordinal + ordinal | Polychoric (if latent normality holds) | Kendall's tau (m <= 5) or Spearman; Goodman-Kruskal gamma |
| Binary + ordinal | Polychoric (if latent normality holds) | Rank-biserial; Kendall's tau, Spearman |
| *Association measures (unsigned, from 0 to 1)* | | |
| Continuous + categorical | Correlation ratio eta-squared | Kruskal-Wallis eta-squared (outliers or non-normal); Cramer's V (binned) |
| Binary + categorical | Cramer's V | -- |
| Ordinal + categorical | Freeman's theta | Cramer's V |
| Categorical + categorical | Cramer's V | Theil's U, Tschuprow's T |

m is the number of categories of the ordinal variable (the smaller of the two for an ordinal pair). Count variables follow the continuous rows.

## Methods

The package implements 18 methods: Pearson, Spearman, Kendall's tau, point-biserial, biserial, rank-biserial, phi, tetrachoric, Yule's Q, polychoric, polyserial, Goodman-Kruskal's gamma, the correlation ratio (eta-squared), Kruskal-Wallis eta-squared, Freeman's theta, Cramer's V, Tschuprow's T, and Theil's U. Each comes with a confidence interval and a p-value.

Required dependencies are NumPy, SciPy, pandas, and statsmodels. Matplotlib and seaborn are optional plotting dependencies.

## Paper

The accompanying paper, *smartcor: Intelligent Correlation Method Selection for Mixed Variable Types* by M. Harshvardhan and Pritam Ranjan (2026), is available as an arXiv preprint: [arXiv:2607.22285](https://arxiv.org/abs/2607.22285) ([doi:10.48550/arXiv.2607.22285](https://doi.org/10.48550/arXiv.2607.22285)).
The vignettes of the R package and the articles on the package website cover the same material.

## Authors

M. Harshvardhan (maintainer) and Pritam Ranjan.

## License

GPL (>= 3)
