Metadata-Version: 2.4
Name: scigantic-transfer
Version: 0.1.2
Summary: Mirror a Hugging Face dataset into a region-local S3 bucket and load it into memory fast (with HF fallback + benchmark).
Author: Scigantic
License: Proprietary — Evaluation Use Only. See LICENSE.
Keywords: huggingface,s3,datasets,gpu,mirror,transfer,benchmark
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.26
Requires-Dist: huggingface_hub>=0.20
Provides-Extra: fast
Requires-Dist: hf_transfer>=0.1.5; extra == "fast"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# scigantic-transfer

Mirror a Hugging Face dataset into a region-local S3 bucket, then load it into
memory fast on a GPU instance in that region.

## Install

```
pip install scigantic-transfer
```

## Mirror a dataset (Hugging Face to S3)

```
scigantic-transfer --region REGION --name HF_DATASET --bucket-name BUCKET [--include GLOB] [--endpoint-url URL]
```

Flags:

```
--region         region of the bucket and the GPU instance (must match)
--name           Hugging Face dataset id
--bucket-name    destination S3 bucket; data lands at s3://BUCKET/NAME/
--include GLOB   files to mirror, repeatable; default '*' (whole repo)
--endpoint-url   S3-compatible endpoint for neoclouds; omit on AWS (see below)
```

Examples:

```
scigantic-transfer --region us-east-1 --name org/ds --bucket-name my-mirror
scigantic-transfer --region us-east-1 --name org/ds --bucket-name my-mirror --include 'data/*'
```

## Neoclouds (Nebius and other non-AWS GPU clouds)

`--endpoint-url` is for neoclouds. The speed win only exists when the bucket and
the GPU instance are in the same cloud and region, so on a neocloud you mirror
into that cloud's own S3-compatible object storage and point the tool at its
endpoint. On AWS, omit the flag.

```
scigantic-transfer --region eu-north1 --name org/ds --bucket-name my-mirror \
    --endpoint-url https://storage.eu-north1.nebius.cloud
```

Works against any S3-compatible store (Nebius, MinIO, Cloudflare R2). Credentials
are read the usual way (env vars / config) and apply to that endpoint.

## Load into memory

```python
from scigantic_transfer import load

result = load("org/ds", region="us-east-1", bucket="my-mirror")
data = result.blob
```

## Benchmark

```
scigantic-transfer-bench --size-gib 1
scigantic-transfer-bench --live --region us-east-1 --name org/ds --bucket-name my-mirror
```

## License

Proprietary, evaluation use only. No redistribution or derivative works.
See LICENSE. Contact support@scigantic.com.
