Metadata-Version: 2.4
Name: datacache
Version: 1.10.1
Summary: Helpers for transparently downloading datasets
Author-email: Alex Rubinsteyn <alex.rubinsteyn@unc.edu>
Project-URL: Homepage, https://github.com/openvax/datacache
Project-URL: Bug Tracker, https://github.com/openvax/datacache/issues
Project-URL: Documentation, https://github.com/openvax/datacache#readme
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=0.15.2
Requires-Dist: appdirs>=1.4.0
Requires-Dist: requests>=2.5.1
Requires-Dist: typechecks>=0.0.2
Provides-Extra: progress
Requires-Dist: tqdm>=4.64; extra == "progress"
Provides-Extra: html
Requires-Dist: lxml>=4.9; extra == "html"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: ruff; extra == "test"
Requires-Dist: tqdm>=4.64; extra == "test"
Requires-Dist: lxml>=4.9; extra == "test"
Dynamic: license-file

# DataCache

[![Tests](https://github.com/openvax/datacache/actions/workflows/tests.yml/badge.svg)](https://github.com/openvax/datacache/actions/workflows/tests.yml)
[![PyPI](https://img.shields.io/pypi/v/datacache.svg)](https://pypi.org/project/datacache/)

Download, verify, transform, and cache datasets for Python applications,
including OpenVax libraries such as pyensembl. DataCache provides streaming
downloads, gzip/ZIP decompression, reusable local paths, offline inspection,
and SQLite caches built from pandas DataFrames.

## Install

Python 3.9 or newer is required.

```sh
python -m pip install datacache
# Optional progress bars and HTML-table conversion:
python -m pip install "datacache[progress,html]"
```

Progress is opt-in with `show_progress=True`. Normal use does not import tqdm
or configure your application's logging.

The existing pandas dependency range is unchanged; upgrading DataCache does
not introduce a pandas 1.5 requirement. CI covers pandas 1.4.4, 1.5.3, and
current releases on supported Python versions.

## Quickstart

This example runs entirely offline and cleans up after itself:

```python
import gzip
import hashlib
from pathlib import Path
from tempfile import TemporaryDirectory

from datacache import Cache

with TemporaryDirectory() as directory:
    root = Path(directory)
    contents = b">reference\nACGT\n"
    source = root / "reference.fa.gz"
    source.write_bytes(gzip.compress(contents))

    cache = Cache("references", cache_root=root / "cache")
    path = cache.fetch(
        source.as_uri(),  # HTTP, HTTPS, and FTP URLs also work.
        filename="reference.fa",
        expected_sha256=hashlib.sha256(contents).hexdigest(),
    )
    assert Path(path).read_bytes() == contents
    source.unlink()
    assert cache.fetch(source.as_uri(), filename="reference.fa") == path
    print(cache.inspect(filename="reference.fa").status)  # available
```

For real releases, get the expected hash from trusted release metadata. Hashes
describe the installed bytes **after** decompression or conversion. A hash
computed from an untrusted download does not establish authenticity.

To install at an exact path instead of using a cache key:

```python
from datacache import fetch_file

path = fetch_file(
    "https://example.org/releases/v1/records.tsv.gz",
    destination="references/v1/records.tsv",
    decompress=True,
    timeout=30,
    show_progress=True,  # Requires datacache[progress].
)
```

Replace the example URL with your dataset URL. Existing files are reused;
`force=True` explicitly replaces them. When integrity expectations are supplied,
an invalid cache hit raises `FileValidationError` instead of silently replacing
the file. Failed downloads leave the previous file intact.

## Choose the right API

| Task | API | Result |
| --- | --- | --- |
| Download or reuse one file | `fetch_file(...)`, `Cache.fetch(...)` | Local path string |
| Compute a path without filesystem access | `expected_path(...)`, `Cache.local_path(...)` | Path string |
| Check presence | `file_exists(...)`, `Cache.exists(...)` | Boolean; does not establish integrity |
| Validate bytes, raising on failure | `validate_file(...)` | Path string |
| Inspect without repair or network access | `inspect_file(...)`, `Cache.inspect(...)` | `FileInspection` |
| Inspect a set of required files | `inspect_files(root, files)` | `CacheInspection` |
| Explicitly share an existing private file | `make_file_readable(...)`, `Cache.make_readable(...)` | Path string; POSIX only |
| Download and parse CSV/TSV | `fetch_csv_dataframe(...)` | pandas DataFrame |
| Cache a custom file transformation | `fetch_and_transform(...)` | Transformer/loader result |
| Create a SQLite cache | `db_from_dataframe(...)`, `db_from_dataframes(...)` | Open SQLite connection |
| Download CSV into SQLite | `fetch_csv_db(...)` | Open SQLite connection |
| Reopen a database with matching metadata | `connect_if_correct_version(...)` | Connection or `None` |

The library does not export `fetch_fasta_dict` or `fetch_fasta_db`. Download
FASTA files with `fetch_file`, then parse them in the consuming library.

## Guides

- [Downloads and cache inspection](docs/downloads.md): destinations, naming,
  decompression, integrity, retries, concurrency, and downstream compatibility.
- [Progress and logging](docs/progress.md): tqdm, callbacks, retries, and
  independent download options for CSV helpers.
- [SQLite and transformations](docs/data.md): numeric fidelity, column names,
  versioning, rollback, connection ownership, and custom transformations.
- [Shared caches](docs/shared-caches.md): permissions, read-only use, and
  troubleshooting existing installations.
- [Downstream integration](docs/integration.md): contracts for consuming libraries.
- [Release notes](CHANGELOG.md) and [release procedure](RELEASING.md).

## Guarantees and limits

Downloads are staged privately and published atomically after validation.
New files respect the process umask; replacements preserve existing access
permissions. This includes pyensembl's private download helpers.

Custom single-file transformations publish only successful output. Existing
SQLite caches rebuild in a transaction: failure rolls back both schema and
rows. New databases are built privately before publication. Cached data is
reused by path or database version; DataCache does not automatically discover
remote changes or repair previously corrupted caches.

Upgrades keep existing cache names and database metadata compatible. Valid
cache hits do not rewrite files, change permissions, or apply new schema
constraints. See [upgrading existing caches](docs/data.md#upgrading-existing-caches)
and [sharing old private files](docs/shared-caches.md#files-already-downloaded-as-0600).

File publication requires local filesystem support for atomic replacement;
new SQLite database publication also requires hard links. SQLite locking and
transactions govern database rebuilds. These are single-file guarantees, not
a multi-file release installer or a distributed lock service.

## Development

```sh
python -m pip install -e ".[test]"
./lint-and-test.sh
python -m examples.basic_usage
```

Tests use local files, mocked responses, and local HTTP servers. They do not
depend on external dataset servers. See CI for the Python, dependency, and
operating-system combinations exercised.
