Metadata-Version: 2.4
Name: sirna-data-grabber
Version: 0.1.3
Summary: Reusable loader for the siRNA knockdown-efficacy dataset (siRNAEfficacyDB + supplementary sources), plus the fetchers that built it.
Author-email: Brandon Walker <brandon.walker@ucsf.edu>
License: MIT
Project-URL: Homepage, https://github.com/BrandonWalk/sirna-data-grabber
Project-URL: Repository, https://github.com/BrandonWalk/sirna-data-grabber
Project-URL: Issues, https://github.com/BrandonWalk/sirna-data-grabber/issues
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE.md
Requires-Dist: pandas>=2.0
Requires-Dist: requests>=2.32
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Provides-Extra: lint
Requires-Dist: ruff>=0.6; extra == "lint"
Requires-Dist: mypy>=1.10; extra == "lint"
Requires-Dist: pandas-stubs; extra == "lint"
Requires-Dist: types-requests; extra == "lint"
Dynamic: license-file

# sirna-data-grabber

A standalone siRNA knockdown-efficacy dataset: the raw data files, full
provenance/license documentation, and a small reusable Python package
(`sirna_data`) for loading it -- and, since `pip install sirna-data-grabber`
alone can't ship most of this non-commercial data, a bundled `sirna-data-fetch`
command that re-fetches it from its original sources. Any project that wants
this dataset can depend on this repo (or just the PyPI package) rather than
vendoring a copy of the data or the loading code.

## License

**The code in this repo (`sirna_data`, `tests/`) is MIT licensed** — see
[`LICENSE`](LICENSE). Use it, modify it, ship it commercially, whatever you
want.

**The data in `data/raw/` is NOT covered by that license.** It's redistributed
under each original source's own terms, and most of those sources are
**non-commercial only** (CC BY-NC / CC BY-NC-ND). Loading the data with this
permissively-licensed code does not lift those restrictions — you still have
to comply with them separately. See [`NOTICE.md`](NOTICE.md) for the
per-source summary and [`data/DATA_SOURCES.md`](data/DATA_SOURCES.md) for
full terms before using the data itself, especially commercially.

## What's here

```
LICENSE                    MIT license -- covers the code only, not data/raw/
NOTICE.md                  per-source data license summary (see License section above)
data/
  raw/                       fetched CSVs + FASTA transcripts (the actual dataset)
  DATA_SOURCES.md            full provenance + license terms for every source
  DATA_SOURCE_LEDGER.md      audit: what's trainable, what's not, and why
  CMSIRNADB_FULL_RETRIEVAL.md   detail on the CMsiRNAdb full-database retrieval
  DEMETER2_README.txt        upstream release notes for DepMap DEMETER2 (investigated, not included -- see FUNCTIONAL_GENOMICS_SCREENS.md)
  FUNCTIONAL_GENOMICS_SCREENS.md   notes on functional-genomics screen sources considered
  POTENTIAL_DATA_SOURCES.md  landscape of sources investigated
  sirecords_overlap_analysis.md    siRecords overlap/dedup analysis
  data_source_ledger.csv     machine-readable companion to DATA_SOURCE_LEDGER.md
  *.png                      figures referenced by the docs above
src/sirna_data/
  raw_loader.py               load + merge every source into SiRNARecord rows
  ncbi_fetch.py                fetch a gene's RefSeq mRNA transcript by symbol
  __init__.py                  public API
  fetch/                       sirna-data-fetch CLI + per-source fetchers (see Install below)
    cli.py                       `sirna-data-fetch` entry point ([project.scripts])
    sirna_efficacy.py            siRNAEfficacyDB + NCBI -> sirna_efficacy.csv, mrna_transcripts.fasta
    monopoli.py                  Monopoli et al. 2023 supplementary data -> monopoli_*
    shabalina.py                 Shabalina et al. 2006 supplementary data -> shabalina_*
    cmsirnadb.py                 CMsiRNAdb + NCBI -> cmsirnadb_full_raw.tsv, cmsirnadb*_transcripts.fasta
tests/
  test_raw_loader.py          unit tests for raw_loader.py (fixtures, no real data needed)
  test_ncbi_fetch.py          unit tests for ncbi_fetch.py (mocked HTTP calls)
  conftest.py                 shared pytest fixtures
```

Start with [`data/DATA_SOURCES.md`](data/DATA_SOURCES.md) for what's in the
dataset and where it came from; [`data/DATA_SOURCE_LEDGER.md`](data/DATA_SOURCE_LEDGER.md)
for the bottom-line audit (6,577 trainable records across 87 genes, 4
sources — 16,178 records / 97 genes if the optional CMsiRNAdb full-database
retrieval is also included). Primary source is **siRNAEfficacyDB** (Zhang
et al. 2024, CC BY-NC); see the docs for the rest and their individual
license terms before reusing this data outside this project.

## Install

`sirna-data-grabber` is [on PyPI](https://pypi.org/project/sirna-data-grabber/),
so most users just need:

```
pip install sirna-data-grabber
```

That installs the `sirna_data` package plus the `sirna-data-fetch` command
(no extras needed). Since the PyPI package can't ship most of this
non-commercial data, use `sirna-data-fetch` to reconstruct it from its
original sources into a local directory, then point `SIRNA_DATA_DIR` at it:

```
sirna-data-fetch --dest ./my_data
export SIRNA_DATA_DIR=./my_data
```

`sirna-data-fetch --only sirna_efficacy monopoli` fetches a subset instead of
all four sources; see `sirna-data-fetch --help`.

### From a git checkout

If you're working from this repo instead (e.g. to browse `data/raw/` and the
provenance docs alongside the code, or to contribute):

```
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
```

This installs `sirna_data` in editable mode, so it resolves `data/raw/`
relative to the checkout automatically -- no `sirna-data-fetch` or
`SIRNA_DATA_DIR` needed if `data/raw/` already has the files. If you copy the
`data/` folder somewhere else, point at it explicitly instead:

```
export SIRNA_DATA_DIR=/path/to/data/raw
```

`SIRNA_DATA_DIR` is just the default -- if you'd rather not set an env var at
all, pass the directory straight to `load_records(data_dir=...)` (see Usage
below).

## Usage

```python
from sirna_data import load_records, fetch_mrna_by_gene

records = load_records()  # list[SiRNARecord], reads from SIRNA_DATA_DIR / default data/raw/
# Or point directly at a directory -- no env var needed:
records = load_records(data_dir="./my_data")
print(len(records), "records across", len({r.gene for r in records}), "genes")

r = records[0]
r.guide_seq       # siRNA antisense strand
r.mrna_window      # local mRNA context around the real target site
r.label            # experimental %knockdown / %inhibition
r.source           # provenance, e.g. "siRNAEfficacyDB"

# Look up any gene's RefSeq transcript live from NCBI:
transcript = fetch_mrna_by_gene("TP53")
transcript.accession, transcript.sequence
```

`load_records()` takes `include_sirna_efficacy` / `include_monopoli` /
`include_shabalina` / `include_cmsirnadb` / `include_cmsirnadb_full` flags
(all default `True`) to include or exclude any individual source, including
the primary siRNAEfficacyDB set -- no source is loaded unconditionally.

`data_dir` (a `Path` or `str`) points every source at a specific directory of
fetched files, as a plain function argument -- no `SIRNA_DATA_DIR` export
required. It falls back to `SIRNA_DATA_DIR` if set, then the package's
default relative `data/raw/` location, in that order.

## Using this from another project

Install as a sibling checkout in editable mode:

```
pip install -e ../sirna-data-grabber
```

That gives you `import sirna_data` with no other coupling — this repo only
depends on pandas and requests, and knows nothing about any particular
downstream model or feature-engineering pipeline.

## Tests

```
pip install -e ".[test]"
pytest
```

Tests run entirely against small in-memory/tmp-dir fixtures (see
`tests/conftest.py`) and mocked HTTP calls, so they don't touch the real
dataset or the network.

## Linting and type checking

```
pip install -e ".[lint]"
ruff check .
mypy
```

Both run in CI on every pull request (`.github/workflows/tests.yml`), alongside
the test matrix.
