Metadata-Version: 2.4
Name: dexverse
Version: 0.1.0
Summary: DexVerse dataset downloader for S3-compatible robot training datasets
Author: DexVerse
License: Apache-2.0
Keywords: dataset,dexverse,downloader,minio,robotics,s3,s5cmd
Requires-Python: >=3.9
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: tomlkit>=0.12
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: boto3>=1.34; extra == 'dev'
Requires-Dist: fastapi>=0.110; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: moto[s3]>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: mock
Requires-Dist: fastapi>=0.110; extra == 'mock'
Requires-Dist: uvicorn>=0.27; extra == 'mock'
Provides-Extra: s3
Requires-Dist: boto3>=1.34; extra == 's3'
Description-Content-Type: text/markdown

# DexVerse Downloader (`dexverse`)

S3-compatible batch downloader for robot-training datasets.

```text
manifest → dexverse-dl → s5cmd → MinIO / S3 → original-path landing
```

The downloader **never reorganizes** your dataset directory. Local paths come from the
manifest's `relativePath` (falling back to `originalPath`, then `objectKey`):

```text
localPath = out / (file.relativePath || file.originalPath || file.objectKey)
```

## Install

```bash
pip install "dexverse[s3,mock]"          # boto3 + fastapi mock server
# optional, recommended download kernel:
#   install s5cmd (https://github.com/peak/s5cmd) and put it on PATH
```

Development:

```bash
uv venv && . .venv/bin/activate          # Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"
```

## Quick start — Local Manifest mode (Phase 0)

```bash
export AWS_ACCESS_KEY_ID="minio_access_key"
export AWS_SECRET_ACCESS_KEY="minio_secret_key"
export AWS_REGION="us-east-1"

dexverse-dl download --manifest ./manifest.json --out /data/dexverse
```

Result:

```text
/data/dexverse/datasets/aaa_test/v1/episodes/episode_000001.hdf5
```

If `s5cmd` is not installed, fall back to the pure-Python backend:

```bash
dexverse-dl download --manifest ./manifest.json --out /data/dexverse --backend python
```

## Quick start — Presigned-URL mode (no login, no S3 creds)

If your backend issues **short-lived presigned URLs** per file (self-authenticating
HTTPS links), put them in the manifest and use the `http` backend. No login, no
`AWS_*` credentials, no s5cmd:

```bash
dexverse-dl download --manifest ./manifest_presigned.json --out D:\DexVerse \
  --backend http --insecure
```

`--insecure` (`-k`) skips TLS verification for self-signed/internal endpoints
like `https://192.168.x.x:8443` — drop it when the endpoint has valid TLS.

Each manifest file just needs `objectKey`, `relativePath`, `size`, and `presignedUrl`:

```json
{
  "taskId": "dl_presigned",
  "files": [{
    "objectKey": "datasets/.../20260429_180359.hdf5",
    "relativePath": "datasets/.../20260429_180359.hdf5",
    "size": 14994636,
    "presignedUrl": "https://host/bucket/datasets/.../20260429_180359.hdf5?X-Amz-Signature=..."
  }]
}
```

> Presigned URLs expire (often 1h). Have the backend regenerate the manifest
> right before each download; expired URLs surface as per-file `HTTP 403`.

## Quick start — Mock Backend mode (Phase 1)

```bash
# terminal 1 — run the mock server
dexverse-mock-server --host 0.0.0.0 --port 18080

# terminal 2 — login + download by taskId
dexverse-dl login --server http://localhost:18080 --token dev-token
dexverse-dl download --task dl_demo_001 --out /data/dexverse
```

## CLI

```bash
dexverse-dl login   --server https://dexverse.xxx.com [--token TOKEN]
dexverse-dl download --manifest ./manifest.json --out /data/dexverse [--parallel 16] [--backend s5cmd|python]
dexverse-dl download --task dl_xxx --out /data/dexverse [--server URL]
dexverse-dl verify  --manifest PATH --out /data/dexverse [--deep]
dexverse-dl repair  --manifest PATH --out /data/dexverse [--backend python]
```

## Python SDK

```python
from dexverse import DexVerseClient, download_manifest

client = DexVerseClient(server="https://dexverse.xxx.com", token="xxx")
local_root = client.download_task(task_id="dl_xxx", out="/data/dexverse", parallel=16)

# or from a local manifest file
download_manifest(manifest_path="./manifest.json", out="/data/dexverse", backend="s5cmd")
```

## Layout of the download root

State files live under `<out>/.dexverse/` and never mix into the dataset tree:

```text
/data/dexverse/
  .dexverse/tasks/<taskId>/
    manifest.json
    download.s5cmd
    download-state.json
    download.log
  datasets/aaa_test/v1/episodes/episode_000001.hdf5   # original paths preserved
```

## Backends

| backend  | engine            | notes                                    |
|----------|-------------------|------------------------------------------|
| `s5cmd`  | peak/s5cmd binary | default, high-throughput, needs binary   |
| `python` | boto3             | fallback, no external binary required    |
| `http`   | httpx             | presigned URLs only — no creds, no login |
