Metadata-Version: 2.4
Name: dexverse
Version: 0.1.2
Summary: DexVerse dataset downloader for S3-compatible robot training datasets
Author: DexVerse
License: Apache-2.0
Keywords: dataset,dexverse,downloader,minio,robotics,s3,s5cmd
Requires-Python: >=3.9
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: tomlkit>=0.12
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: boto3>=1.34; extra == 'dev'
Requires-Dist: fastapi>=0.110; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: moto[s3]>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: mock
Requires-Dist: fastapi>=0.110; extra == 'mock'
Requires-Dist: uvicorn>=0.27; extra == 'mock'
Provides-Extra: s3
Requires-Dist: boto3>=1.34; extra == 's3'
Description-Content-Type: text/markdown

# DexVerse Downloader (`dexverse`)

Presigned-URL batch downloader for robot-training datasets.

```text
manifest -> dexverse-dl -> httpx -> presigned URLs -> local file landing
```

The primary input is the DexVerse backend's raw presigned-URL JSON. For those
payloads, `storage_path` is the remote object key and `file_name` is the local
landing path under `--out`.

Canonical manifests are still supported. Their local paths come from
`relativePath`, falling back to `originalPath`, then `objectKey`:

```text
localPath = out / (file.relativePath || file.originalPath || file.objectKey)
```

## Install

```bash
pip install "dexverse[s3,mock]"          # boto3 + fastapi mock server
# optional legacy S3 backend:
#   install s5cmd (https://github.com/peak/s5cmd) and put it on PATH
```

Development:

```bash
uv venv && . .venv/bin/activate          # Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"
```

## Quick Start

If your backend issues short-lived presigned URLs per file, save that response
as `download.json` and run:

```bash
dexverse-dl download --manifest ./download.json --out D:\DexVerse
```

No login, no `AWS_*` credentials, no s5cmd, no backend flag, and no TLS flag are
required for the common internal/self-signed endpoint case. The default backend
is `http`, and TLS verification is disabled by default for http/presigned
downloads. Use `--secure` when the endpoint has a valid certificate and you want
certificate validation.

Raw backend output is accepted directly:

```json
[{
  "download_url": "https://host/bucket/datasets/.../20260429_180359.hdf5?X-Amz-Signature=...",
  "storage_bucket": "dexverse",
  "storage_path": "datasets/.../20260429_180359.hdf5",
  "file_name": "20260429_180359.hdf5",
  "file_size": 14994636
}]
```

Result:

```text
D:\DexVerse\20260429_180359.hdf5
```

Presigned URLs expire, often after 1 hour. Have the backend regenerate the
manifest right before each download; expired URLs surface as per-file `HTTP 403`
errors.

## CLI

```bash
dexverse-dl login   --server https://dexverse.xxx.com [--token TOKEN]
dexverse-dl download --manifest ./download.json --out /data/dexverse [--parallel 16] [--secure] [--backend http|python|s5cmd]
dexverse-dl download --task dl_xxx --out /data/dexverse [--server URL]
dexverse-dl verify  --manifest PATH --out /data/dexverse [--deep]
dexverse-dl repair  --manifest PATH --out /data/dexverse [--backend http|python|s5cmd]
```

## Python SDK

```python
from dexverse import DexVerseClient, download_manifest

client = DexVerseClient(server="https://dexverse.xxx.com", token="xxx")
local_root = client.download_task(task_id="dl_xxx", out="/data/dexverse", parallel=16)

# or from a local manifest file
download_manifest(manifest_path="./download.json", out="/data/dexverse")
```

## Layout

State files live under `<out>/.dexverse/` and never mix into the dataset tree:

```text
/data/dexverse/
  .dexverse/tasks/<taskId>/
    manifest.json
    download-state.json
    download.log
  20260429_180359.hdf5
```

## Backends

| backend  | engine            | notes                                   |
|----------|-------------------|-----------------------------------------|
| `http`   | httpx             | default; presigned URLs, no creds/login |
| `python` | boto3             | legacy S3 fallback                      |
| `s5cmd`  | peak/s5cmd binary | legacy S3 high-throughput backend       |
