Metadata-Version: 2.5
Name: sparse-count-pca
Version: 1.0.0
Summary: PCA of implicitly represented sparse count transforms
Project-URL: Homepage, https://sparse-count-pca.readthedocs.io/
Project-URL: Documentation, https://sparse-count-pca.readthedocs.io/en/latest/
Project-URL: Repository, https://github.com/jgarthur/sparse_count_pca
Project-URL: Issues, https://github.com/jgarthur/sparse_count_pca/issues
Project-URL: Changelog, https://github.com/jgarthur/sparse_count_pca/blob/main/CHANGELOG.md
Author: Joey Arthur
License-Expression: MIT
License-File: LICENSE
Keywords: anndata,bioinformatics,dimensionality reduction,principal component analysis,single-cell,sparse matrix
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: anndata>=0.11.4
Requires-Dist: numpy>=2.2.6
Requires-Dist: scikit-learn>=1.7.2
Requires-Dist: scipy>=1.15.3
Provides-Extra: docs
Requires-Dist: ipykernel; extra == 'docs'
Requires-Dist: jupytext; extra == 'docs'
Requires-Dist: mkdocs-material; extra == 'docs'
Requires-Dist: mkdocstrings-python; extra == 'docs'
Requires-Dist: nbconvert; extra == 'docs'
Requires-Dist: ruff; extra == 'docs'
Requires-Dist: scanpy>=1.10; extra == 'docs'
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Requires-Dist: ruff; extra == 'test'
Requires-Dist: scanpy>=1.10; extra == 'test'
Requires-Dist: zarr; extra == 'test'
Description-Content-Type: text/markdown

# sparse-count-pca

## [Read the documentation →](https://sparse-count-pca.readthedocs.io/en/latest/)

Single-cell transcriptomics data reaches the user as a large, very sparse
matrix of integer counts. Many normalization methods assign nonzero values to
observed zeros, making the transformed matrix dense and potentially increasing
entry storage by an order of magnitude or more.

Most workflows do not need every normalized entry at once, because their
primary consumer is PCA. `sparse-count-pca` represents supported count
transforms exactly as a sparse matrix plus a small low-rank term. This
factorization provides efficient matrix-vector and matrix-matrix products
without materializing the full transformed matrix. The package uses those
products to compute PCA of the column-centered transform through SciPy's
`LinearOperator` interface and truncated SVD.

The implementation is pure Python and is tested against independent
dense-matrix implementations and pinned external reference outputs. It covers
PCA of GLM model residuals, size-factor-normalized log1p counts, shifted
centered log-ratio (CLR) coordinates, classical correspondence analysis, and
several more experimental options.

## Installation

Python 3.11 or newer is required.

```bash
pip install sparse-count-pca
```

## Quick start

The primary API writes Scanpy-compatible PCA outputs to an `AnnData` object:

```python
import sparse_count_pca as scp

scp.residual_pca(
    adata,
    layer="counts",
    n_comps=50,
    model="poisson",
    residual="pearson",
)
```

Cell embeddings/scores are written to `adata.obsm["X_pca"]`, component vectors
to `adata.varm["PCs"]`, and variance statistics and parameters to
`adata.uns["pca"]`. The full matrix of normalized values is not stored, by
design. For plotting those values and other uses, see
[normalized values and transform reuse][transform-reuse].

Pass raw counts: finite, nonnegative, and integer-valued. Floating-point
counts are accepted within `1e-8` of an integer, and `check_values=False`
skips that check. Remove cells with no counts first, for example with
`sc.pp.filter_cells(adata, min_counts=1)`. See [input
validation][input-validation].

[Run a complete example with simulated data](https://sparse-count-pca.readthedocs.io/en/latest/getting-started/)

## Supported transforms

| Transform | AnnData function | Main use |
| --- | --- | --- |
| Residual PCA | `residual_pca(...)` | Pearson or deviance residual PCA under Poisson, binomial, or scaled-NB (see note below) |
| Size-factor-normalized log1p | `log1p_norm_pca(...)` | PCA of `log1p` counts after size-factor normalization; matches Scanpy's `normalize_total` + `log1p` |
| Count-scale shifted CLR | `shifted_clr_pca(...)` | Within-observation log ratios; includes the PFlog parameterization of Booeshaghi et al. (2026) |
| Composition-scale shifted CLR | `proportion_shifted_clr_pca(...)` | CLR after a fixed shift on the composition scale |
| Correspondence analysis | `correspondence_analysis(...)` | Classical contingency-table ordination |

[Compare transform assumptions, formulas, provenance, and API
maturity](https://sparse-count-pca.readthedocs.io/en/latest/transforms/)

Already using Scanpy? See [coming from Scanpy][scanpy-compatibility] for what
each call maps to, including the `normalize_total` + `log1p` + `pca` pipeline.

`scaled_nb` is a package-specific name for a negative-binomial model that
scales overdispersion inversely with cell depth, as in the sSeq model from
[Yu, Huber, and Vitek (2013)](https://doi.org/10.1093/bioinformatics/btt143). It
is not an implementation of
[SCTransform](https://doi.org/10.1186/s13059-019-1874-1); its Pearson residuals
agree with SCTransform's only under conditions that ordinary data does not meet.
See the [scaled-NB null model][scaled-nb-model].

There is also an experimental scaled-NB extension to correspondence analysis;
see the
[correspondence-analysis guide][correspondence-analysis-guide].

## How it works

Supported dense transforms have an exact representation,

```text
transformed matrix = sparse matrix + low-rank baseline,
```

which allows efficient factored matrix multiplication without materializing the
actual matrix. Selecting PCA variables preserves this form, and column centering
adds one rank-one term. This package uses `scipy.sparse.linalg.LinearOperator`
to expose matrix-vector and matrix-matrix products to
`scipy.sparse.linalg.svds`, using the ARPACK solver. See
[sparse plus low rank][sparse-plus-low-rank] for more details.

## Performance

Residual PCA never stores the dense residual matrix, so its memory grows with
the number of stored counts rather than with cells × genes. Scanpy's normalized
`log1p` is also sparse, so there the two packages perform similarly.

On 22,111 cells × 29,157 genes with 72 million stored counts, pooled from
10x's 5k Human PBMC 3' GEM-X donors [1][pbmc-1], [2][pbmc-2], [3][pbmc-3], and
[4][pbmc-4], `sparse-count-pca` 1.0.0 and Scanpy 1.12.4 computed 30 components
in float64 on one thread of a Linux server:

| Calculation | Scanpy | sparse-count-pca |
| --- | --- | --- |
| Pearson residual PCA | 35.6 s, 20.8 GiB | 15.6 s, 1.5 GiB |
| Normalized `log1p` PCA | 10.1 s, 2.7 GiB | 10.8 s, 1.5 GiB |

Figures here and below are medians of three runs. Memory is the process's peak
resident memory, including the loaded counts. Residuals are clipped at Scanpy's
±√n_cells threshold in both packages.

To measure scaling, each package selected 5,000
[highly variable genes][hvg-selection] by Pearson residual variance and
computed PCA on them, in float32 with a 26 GiB memory limit. Inputs are random
subsets of 10x's
[1.3 million brain cells from E18 mice][brain-data]; quality filtering removed
about 0.3% of each subset's cells.

| Cells sampled | Scanpy `recipe_pearson_residuals` | sparse-count-pca |
| ---: | --- | --- |
| 200,000 | 90 s, 23.5 GiB | 55 s, 7.4 GiB |
| 400,000 | out of memory | 120 s, 14.6 GiB |
| 600,000 | out of memory | 161 s, 21.7 GiB |

## API structure

- **One-step AnnData API.** Functions such as `residual_pca`,
  `shifted_clr_pca`, and `correspondence_analysis` write results directly into
  an `AnnData` object, as in the quick start above. Counts may come from `.X`
  or a layer. See [AnnData workflows][anndata-workflows].
- **One-step matrix API.** The corresponding `_matrix` functions take dense or
  sparse matrices outside AnnData and return a result object:

  ```python
  result = scp.residual_pca_matrix(counts, ...)

  result.scores       # observations by components
  result.components   # components by variables
  result.components.T # variables by components
  ```

- **Two-step transform API.** `transform` fits a normalization once, so
  normalized values can be plotted or inspected in bounded slices, per-gene
  variances can rank highly variable genes, or several PCA masks can share one
  fitted state. See
  [normalized values and transform reuse][transform-reuse].

## Important distinctions

- **A gene mask usually selects PCA columns, not normalization inputs.** Slice
  the count matrix first if excluded genes should not affect cell totals, gene
  proportions, or CLR row means. Correspondence analysis is the exception: its
  mask defines the contingency table, so margins are recomputed after masking.
  See [normalization, masking, and centering][normalization-masking-centering].
- **Scaled-NB residuals are not a drop-in SCTransform v2 implementation.** Do
  not expect default outputs to match. See the
  [compatibility reference][sctransform-compatibility] for the conditions under
  which equality can be tested.
- **Residuals are clipped by default,** as Seurat and Scanpy do, at Seurat's
  threshold ±`sqrt(n_obs / 30)` (`clip="seurat"`), so a few extreme values
  cannot dominate PCA. This symmetric clipping can make memory use depend on
  the data values, not just the matrix's size and sparsity, so the package
  caps that growth and raises `RuntimeError` beyond it. Pass
  `clip_max_nnz_ratio=None` to remove the cap. See
  [clipping and precision][clipping-and-precision] for details and
  alternatives.
- **Keep the default `dtype="float64"`** unless memory requires otherwise; see
  [calculation dtype][calculation-dtype] for what `float32` changes.

## Related work

The [`cleartools`](https://github.com/cleartools) projects provide sparse
Rust-backed tooling for shifted CLR (PFlog) workflows: the Python
[`scclr`](https://github.com/cleartools/scclr) package, built on the
[`runorm`](https://github.com/cleartools/runorm) normalization crate. See the
[compatibility reference][pflog-compatibility] for the scope of the
relationship.

[`10XGenomics/scan-rs`](https://github.com/10XGenomics/scan-rs) is a Rust
library used by Cell Ranger and contains similar matrix-representation and
normalization machinery.

## Citation

If you use `sparse-count-pca` in published work, please cite the version you
used. Citation metadata is in
[`CITATION.cff`](https://github.com/jgarthur/sparse_count_pca/blob/main/CITATION.cff);
the **Cite this repository** button on the GitHub page formats it as APA or
BibTeX. Each GitHub release is archived on Zenodo with its own DOI. Please also
cite the methods behind the transforms you use; the
[transform catalog][transform-catalog] lists each one's origin.

## Development

```bash
uv sync --extra test --extra docs
uv run python -m pytest
uv run ruff check .
uv run ruff format --check .
uv run mkdocs build --strict
```

See
[CONTRIBUTING.md](https://github.com/jgarthur/sparse_count_pca/blob/main/CONTRIBUTING.md)
for project invariants and documentation expectations.

[anndata-workflows]: https://sparse-count-pca.readthedocs.io/en/latest/guides/anndata-workflows/
[brain-data]: https://www.10xgenomics.com/datasets/1-3-million-brain-cells-from-e-18-mice-2-standard-1-3-0
[calculation-dtype]: https://sparse-count-pca.readthedocs.io/en/latest/concepts/clipping-and-precision/#calculation-dtype
[clipping-and-precision]: https://sparse-count-pca.readthedocs.io/en/latest/concepts/clipping-and-precision/
[correspondence-analysis-guide]: https://sparse-count-pca.readthedocs.io/en/latest/guides/correspondence-analysis/
[hvg-selection]: https://sparse-count-pca.readthedocs.io/en/latest/guides/transform-reuse/#rank-genes-by-variance
[input-validation]: https://sparse-count-pca.readthedocs.io/en/latest/development/specification/#input-validation
[normalization-masking-centering]: https://sparse-count-pca.readthedocs.io/en/latest/concepts/normalization-masking-and-centering/
[pbmc-1]: https://www.10xgenomics.com/datasets/5k_Human_Donor1_PBMC_3p_gem-x
[pbmc-2]: https://www.10xgenomics.com/datasets/5k_Human_Donor2_PBMC_3p_gem-x
[pbmc-3]: https://www.10xgenomics.com/datasets/5k_Human_Donor3_PBMC_3p_gem-x
[pbmc-4]: https://www.10xgenomics.com/datasets/5k_Human_Donor4_PBMC_3p_gem-x
[pflog-compatibility]: https://sparse-count-pca.readthedocs.io/en/latest/reference/compatibility/#shifted-clr-pflog-and-cleartools
[scaled-nb-model]: https://sparse-count-pca.readthedocs.io/en/latest/reference/scaled-nb-model/
[scanpy-compatibility]: https://sparse-count-pca.readthedocs.io/en/latest/reference/compatibility/#coming-from-scanpy
[sctransform-compatibility]: https://sparse-count-pca.readthedocs.io/en/latest/reference/compatibility/#sctransform-v2
[sparse-plus-low-rank]: https://sparse-count-pca.readthedocs.io/en/latest/concepts/sparse-plus-low-rank/
[transform-catalog]: https://sparse-count-pca.readthedocs.io/en/latest/transforms/
[transform-reuse]: https://sparse-count-pca.readthedocs.io/en/latest/guides/transform-reuse/
