Metadata-Version: 2.4
Name: hubdl
Version: 0.1.2
Summary: Multi-hub model downloader (Hugging Face + ModelScope) with resume, rate-limit, L3/L4 verify
Author: hubdl contributors
License: Apache-2.0
Project-URL: Homepage, https://github.com/mrhegit/hubdl
Project-URL: Documentation, https://github.com/mrhegit/hubdl#readme
Project-URL: Documentation (中文), https://github.com/mrhegit/hubdl/blob/main/README.zh-CN.md
Project-URL: Bug Tracker, https://github.com/mrhegit/hubdl/issues
Project-URL: Source, https://github.com/mrhegit/hubdl
Project-URL: Changelog, https://github.com/mrhegit/hubdl/blob/main/CHANGELOG.md
Project-URL: Security, https://github.com/mrhegit/hubdl/blob/main/SECURITY.md
Project-URL: Publishing, https://github.com/mrhegit/hubdl/blob/main/docs/PUBLISHING.md
Keywords: huggingface,modelscope,aria2,download,llm,checksum,resume
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Utilities
Classifier: Natural Language :: English
Classifier: Natural Language :: Chinese (Simplified)
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# hubdl

[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![PyPI version](https://img.shields.io/pypi/v/hubdl.svg)](https://pypi.org/project/hubdl/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](./LICENSE)
[![Platform](https://img.shields.io/badge/platform-Linux-lightgrey.svg)](#requirements)

**English** | [简体中文](./README.zh-CN.md)

**hubdl** is a multi-hub model and dataset downloader for **Linux servers**, distributed on **[PyPI](https://pypi.org/project/hubdl/)**.

It combines:

- **Hugging Face** listing + aria2 workflow (inspired by [hfd](https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f))
- **ModelScope** API listing (from battle-tested local `msd` tooling)
- A **shared core**: integrity-aware resume, hard rate limits, path sandbox, directory flock, L3 SHA-256 and L4 repository reports

> **Disclaimer:** hubdl is **not** affiliated with Hugging Face, Inc. or ModelScope / Alibaba DAMO.  
> You are responsible for complying with each model's license and each Hub's terms of service.

---

## Install & use

**Official install:** create a venv, then install from **PyPI**. Install system **aria2** for downloads.  
Do **not** install into the system/global Python (`sudo pip` is discouraged).

```bash
# 1) System packages (Debian / Ubuntu)
sudo apt update
sudo apt install -y python3 python3-venv aria2 ca-certificates

# 2) Install from PyPI into a venv
python3 -m venv .venv
source .venv/bin/activate          # Windows: .\.venv\Scripts\Activate.ps1
pip install -U pip
pip install hubdl

# 3) Verify
hubdl --version
hubdl hf gpt2 --local-dir ./gpt2 --dry-run
```

| Task | Command |
|------|---------|
| **Install (official)** | `pip install hubdl` |
| Pin a version | `pip install "hubdl==0.1.2"` |
| Upgrade | `pip install -U hubdl` |

Package: https://pypi.org/project/hubdl/ · CLI entry points: `hubdl`, `hfdx`, `msd`

---

## Table of contents

- [Install & use](#install--use)
- [Features](#features)
- [CLI entry points](#cli-entry-points)
- [Requirements](#requirements)
- [Installation](#installation)
- [Quick start](#quick-start)
- [Architecture](#architecture)
- [CLI reference](#cli-reference)
- [Profiles](#profiles)
- [Workflow & modes](#workflow--modes)
- [Integrity (L0–L4)](#integrity-l0l4)
- [Delete policy](#delete-policy)
- [Metadata directory](#metadata-directory)
- [Environment variables](#environment-variables)
- [Hub differences](#hub-differences)
- [Security](#security)
- [Troubleshooting](#troubleshooting)
- [Development](#development)
- [Roadmap](#roadmap)
- [Credits & license](#credits--license)

---

## Features

| Feature | Description |
|---------|-------------|
| Multi-hub | Hugging Face (`hf`) and ModelScope (`ms`) behind one core |
| Dual CLI aliases | `hfdx` → HF, `msd` → ModelScope (drop-in style entrypoints) |
| Fast listing | HF: siblings fast path + resumable tree walk; MS: paginated `repo/files` |
| Download engine | Default **aria2c** (multi-connection, `.aria2` resume); optional **wget** |
| Rate limit | `--rate-mbps` (megabit/s) → aria2 overall limit |
| Integrity | L0 resume → L1 size → L3 SHA-256 (when available) → L4 repo report |
| Safe resume | Incomplete files with `size < expected` are **not** deleted |
| Path sandbox | Reject `..`, absolute paths, and writes into control dirs (`.hubdl`, …) |
| Single instance | `flock` on `.hubdl/hubdl.lock` (Linux) |
| Ops modes | `--dry-run`, `--verify-only`, `--offline`, `--refresh` |
| Structured outputs | `plan.json`, `verify_report.json`, `last_run.json`, `hash_cache.tsv` |

---

## CLI entry points

| Command | Hub | Notes |
|---------|-----|--------|
| `hubdl hf <REPO>` | Hugging Face | Primary unified CLI |
| `hubdl ms <REPO>` | ModelScope | Primary unified CLI |
| `hfdx <REPO>` | Hugging Face | Alias of `hubdl hf` |
| `msd <REPO>` | ModelScope | Alias of `hubdl ms` |

Also accepted: `hubdl huggingface …`, `hubdl modelscope …`.

**Current version:** `0.1.2` ([PyPI](https://pypi.org/project/hubdl/) · [CHANGELOG](./CHANGELOG.md))  
**Metadata directory:** `<local-dir>/.hubdl/`  
**Manifest format:** `hubdl-manifest-v1` — `size` TAB `sha256|-` TAB `path`

---

## Requirements

### System (Linux recommended)

| Package | Required | Role |
|---------|----------|------|
| Python **≥ 3.10** | Yes | Runtime |
| **aria2c** | Recommended | Default download engine |
| wget | Optional | `--tool wget` |
| flock / fcntl | Recommended | Single-instance lock (Linux) |
| ca-certificates | Recommended | HTTPS |

```bash
# Debian / Ubuntu
sudo apt update
sudo apt install -y python3 python3-venv python3-pip aria2 ca-certificates
# optional
sudo apt install -y wget
```

### Python

No third-party runtime dependencies (stdlib only). Dev extras: `pytest`, `pytest-cov`.

---

## Installation

| Method | Use when |
|--------|----------|
| **PyPI** `pip install hubdl` | **Normal use (official)** |
| GitHub Release wheel | Offline / no PyPI access |
| `pip install git+https://…` | Install from a specific commit or tag |
| Editable clone | Developing hubdl itself |

Release process for maintainers: [docs/RELEASE.md](./docs/RELEASE.md) · [docs/PUBLISHING.md](./docs/PUBLISHING.md).

### From PyPI (official)

```bash
sudo apt install -y python3 python3-venv aria2 ca-certificates   # Linux, once
python3 -m venv .venv
source .venv/bin/activate   # Windows: .\.venv\Scripts\Activate.ps1
pip install -U pip
pip install hubdl
hubdl --version
hfdx --help
msd --help
```

- PyPI: https://pypi.org/project/hubdl/  
- Console scripts: `hubdl`, `hfdx`, `msd`  
- Python runtime dependencies: **none** (stdlib only). Downloader: system **`aria2c`**.

### From a GitHub Release wheel (offline)

Download the `.whl` from [Releases](https://github.com/mrhegit/hubdl/releases):

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install ./hubdl-0.1.2-py3-none-any.whl
hubdl --version
```

### From Git (optional)

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install "git+https://github.com/mrhegit/hubdl.git@v0.1.2"
# or track main:
# pip install "git+https://github.com/mrhegit/hubdl.git"
hubdl --version
```

### Editable install (contributors)

```bash
git clone https://github.com/mrhegit/hubdl.git && cd hubdl
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
hubdl --version
pytest -q
```

Without activating the venv: `.venv/bin/hubdl` (Linux) or `.venv\Scripts\hubdl.exe` (Windows).  
`.venv/` is gitignored and must not be committed.

---

## Quick start

Assumes **[Install & use](#install--use)** is done and the venv is active (`hubdl --version` works).

### Hugging Face

```bash
# Plan only (no download, no delete)
hubdl hf gpt2 --local-dir ./gpt2 --dry-run
# equivalent alias
hfdx gpt2 --local-dir ./gpt2 --dry-run

# Full download (L3 default = auto)
hubdl hf Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05

# Private / gated model
export HF_TOKEN=hf_xxxxxxxx
hubdl hf meta-llama/Llama-3.2-1B \
  --local-dir ./llama \
  --hf_username YOUR_HF_NAME

# Mirror endpoint
export HF_ENDPOINT=https://hf-mirror.com
hubdl hf gpt2 --local-dir ./gpt2
```

### ModelScope

```bash
hubdl ms Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05 --dry-run
# equivalent alias
msd Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05 --dry-run

# Shared LAN: hard cap 35 Mb/s, single connection
hubdl ms deepseek-ai/DeepSeek-V4-Flash-0731 \
  --local-dir /data1/models/DeepSeek-V4-Flash-0731 \
  --profile lan-shared
```

### Verify only

```bash
# Re-check integrity (verify-only maps auto → full)
hubdl hf owner/model --local-dir ./m --verify-only --verify-hash on
hubdl ms owner/model --local-dir ./m --verify-only --offline
```

### Background job (nohup)

```bash
mkdir -p /data1/model_download_logs
LOG=/data1/model_download_logs/hubdl_$(date +%Y%m%d_%H%M%S).log
nohup hubdl ms deepseek-ai/DeepSeek-V4-Flash-0731 \
  --local-dir /data1/models/DeepSeek-V4-Flash-0731 \
  --profile lan-shared \
  --progress none \
  > "$LOG" 2>&1 &
echo $! > /data1/model_download_logs/hubdl.pid
```

---

## Architecture

Option **A**: dual CLI + shared library.

```text
hubdl/
├── src/hubdl/
│   ├── cli.py                 # hubdl / hfdx / msd entrypoints
│   ├── core/                  # shared pipeline
│   │   ├── sandbox.py         # path jail, reserved dirs
│   │   ├── plan.py            # size + optional L3
│   │   ├── verify.py          # L3 + L4 report
│   │   ├── download.py        # aria2 / wget
│   │   ├── cache.py           # hash_cache.tsv
│   │   ├── httputil.py        # host pin, retries
│   │   └── pipeline.py        # orchestration
│   └── hubs/
│       ├── huggingface.py     # list + resolve URLs
│       └── modelscope.py
├── scripts/                   # repo wrappers without install
└── tests/
```

| Layer | Responsibility |
|-------|----------------|
| **CLI** | Parsing, profiles, hub selection, exit codes |
| **core** | Plan, verify, download, lock, atomic IO, hash policy |
| **hubs.*** | File listing + URL resolution **only** |

---

## CLI reference

### Positional

| Argument | Description |
|----------|-------------|
| `hub` | Only for `hubdl`: `hf` / `ms` (or long names) |
| `repo_id` | `owner/name`; HF also allows legacy single-segment ids (e.g. `gpt2`) |

### Common options

| Option | Default | Description |
|--------|---------|-------------|
| `--local-dir DIR` | `./<repo_name>` | Root for weights + `.hubdl/` |
| `--revision REV` | `main` (hf) / `master` (ms) | Branch / tag / commit |
| `--dataset` | off | Treat repo as a **dataset** |
| `--token TOKEN` | env (see below) | Access token |
| `--hf_token TOKEN` | | Alias of `--token` for HF |
| `--hf_username NAME` | | HF gated hint (Bearer still uses token) |
| `--endpoint URL` | hub default | Override API/download base |
| `--include PAT` | all | Include glob (`*`); repeatable |
| `--exclude PAT` | none | Exclude glob; repeatable |
| `--tool aria2c\|wget` | `aria2c` | Download backend |
| `-x N` | `4` | aria2 connections per file |
| `-j N` | `5` | aria2 concurrent files |
| `--rate-mbps N` | unlimited | Overall cap in **megabit/s** |
| `--max-download-limit L` | | Pass-through aria2 limit string (e.g. `4480K`) |
| `--dry-run` | | Fetch + plan only |
| `--verify-only` | | L3+L4 only; no download; no delete |
| `--offline` | | No network list (requires existing manifest); **only with `--verify-only`** |
| `--refresh` | | Force re-list (ignore fingerprint cache) |
| `--verify-hash MODE` | **hf: `auto`**, **ms: `on`** | `auto` \| `on` \| `off` |
| `--no-verify-hash` | | Same as `--verify-hash off` |
| `--verify-mode MODE` | `auto` | `full` \| `light` \| `auto` (`verify-only` + `auto` → `full`) |
| `--verify-jobs N\|auto` | auto | SHA-256 parallelism |
| `--no-fix-bad` | | Never delete bad files |
| `--trust-aria2-checksum` | off | Seed hash cache from size-aligned downloads (use carefully) |
| `--progress MODE` | `aria2` | `aria2` \| `none` (`line` reserved) |
| `--profile NAME` | | See [Profiles](#profiles) |
| `--allow-http` | off | Allow `http://` endpoints |
| `--proxy URL` | env (see below) | HTTP(S) proxy only (`http://host:port`); `none`/`direct`/`off`/`false`/`0` = force direct. **No SOCKS** |
| `--no-proxy LIST` | env | Comma-separated bypass hosts (aria2 + simple API match) |
| `-h` / `--help` | | Help |
| `--version` | | Print version |

### Rate limit note

Spoken “35 megabit/s” → `--rate-mbps 35` → roughly `4480K` in aria2 (`N × 1024 / 8` KiB/s).

---

## Profiles

| Profile | Effect |
|---------|--------|
| `lan-shared` | `--rate-mbps 35` if unset, `-x 1 -j 1` |
| `fast` | No rate limit, `-x 4 -j 5`; on HF sets `verify-hash=off` |
| `safe-verify` | `--verify-only`, `--verify-mode full`, `--verify-hash on`, no delete |

---

## Workflow & modes

### download (default)

```text
fetch list → plan (size + optional L3)
           → scrub orphan .aria2 (size-ok payload) → re-plan if needed
           → download round=1 → plan
           → if still need: download round=2 → plan
           → verify L3+L4
```

**ModelScope multi-conn notes (aria2 only):**

| Rule | Behavior |
|------|----------|
| Small files (`size ≤ 32 MiB` or unknown) | Per-URI `split=1` / single connection (avoids intermittent 404 under multi-range) |
| Large weights | Keep user `-x` multi-connection |
| Global `--min-split-size` | `16M` (HF + MS); reduces pointless splits on tiny objects |
| Round 2 (if need remains) | **MS:** force single-conn recovery (`-x1 -j1`); **HF:** same strategy retry |
| Orphan `.aria2` | If payload size already matches expected, control file is removed (size-only; L3 still owns content trust) |
| `wget` | No multi-range policy; prefer `aria2c` for MS reliability |

### dry-run

```text
fetch → plan → exit 0
```

### verify-only

```text
fetch | offline → verify → exit 0/2
```

(`auto` verify-mode becomes `full` so stale `plan.json` is not trusted lightly.)

### Exit codes

| Code | Meaning |
|------|---------|
| **0** | Success (download+verify, dry-run, or verify-only pass) |
| **1** | Download still incomplete (`need` non-empty) |
| **2** | L3/L4 verification failed |
| **3** | Directory lock conflict |
| **4** | Invalid args / runtime error |
| **130** | Interrupted (Ctrl+C) |

---

## Integrity (L0–L4)

| Level | Meaning |
|-------|---------|
| **L0** | Transfer resume: aria2 `.aria2` / wget `-c` |
| **L1** | Local file size equals expected size from listing |
| **L3** | Local SHA-256 equals expected (LFS / API when present) |
| **L4** | All listed files present, no leftover `.aria2`, counts and matched bytes consistent |

**`--verify-hash auto` (HF default):** enable L3 if any of:

- share of files with sha ≥ 30%, or  
- all files ≥ 20 MiB have sha, or  
- files covering top ~90% of total bytes all have sha  

**ModelScope default** is `on` (API usually provides `Sha256`).

Files without a digest are recorded as `size_ok_nohash` when L3 is enabled for the run.

---

## Delete policy

| Local state | Plan | Delete |
|-------------|------|--------|
| Missing | need | No |
| Has `.aria2` | need (in progress) | **No** (payload kept); size-ok orphan `.aria2` may be scrubbed |
| `size < expected` | need (resume) | **No** |
| `size > expected` | need | **Yes** if `fix_bad` |
| Size OK, hash OK | skip | No |
| Size OK, hash bad | need | **Yes** if `fix_bad` |
| Hash IO error | need | **No** |
| Path into `.hubdl` / `..` / absolute | rejected | No |
| `--dry-run` / `--verify-only` | — | **Forced no** |

Deletes only occur inside `--local-dir` after sandbox checks.

---

## Metadata directory

Path: **`<local-dir>/.hubdl/`** (mode preferably `0700`).

| File | Purpose |
|------|---------|
| `manifest` | Listing v1: size, sha256 or `-`, path |
| `repo_meta.json` | Counts, endpoint, hub, fingerprint fields |
| `list_fingerprint` | Skip re-list when options unchanged |
| `plan.json` | Last plan statistics |
| `need_download.txt` | Paths still needed |
| `last_need.txt` | Snapshot of last download batch |
| `aria2_urls.txt` | aria2 input (may briefly hold Authorization; cleared after download) |
| `hash_cache.tsv` | L3 cache (path, size, mtime, expected, digest) |
| `verify_report.json` | Last L3+L4 report |
| `download.log` | Download log (tee / append) |
| `progress.json` | Progress snapshot |
| `last_run.json` | Phase + exit code |
| `hubdl.lock` | Single-instance lock |
| `verify_hash.flag` | Last decided L3 on/off (CLI still re-evaluated each run) |
| `repo_metadata.json` | Raw HF metadata when using HF hub |
| `list_state` / `manifest.partial` | Temporary HF tree-walk checkpoints |

**Do not** publish or artifact-upload `.hubdl/` without scrubbing secrets.

---

## Environment variables

| Variable | Applies to | Description |
|----------|------------|-------------|
| `HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN` | HF | Access token |
| `HF_USERNAME` | HF | Username hint for gated repos |
| `HF_ENDPOINT` | HF | Mirror base (default `https://huggingface.co`) |
| `MODELSCOPE_API_TOKEN` / `MODELSCOPE_TOKEN` | MS | Access token |
| `MODELSCOPE_ENDPOINT` | MS | Default `https://www.modelscope.cn` |
| `HUBDL_PROXY` | both | Preferred hubdl proxy URL (or `none` to force direct) |
| `HUBDL_NO_PROXY` | both | Preferred no-proxy list |
| `https_proxy` / `HTTPS_PROXY` / `http_proxy` / `HTTP_PROXY` / `ALL_PROXY` / `all_proxy` | both | Standard proxy env (after `HUBDL_PROXY`); applied to **API + aria2 + wget** |
| `no_proxy` / `NO_PROXY` | both | Bypass list |
| `HUBDL_ALLOW_HTTP` | both | `1` allows plain `http://` endpoints |

### Proxy notes

Priority: **CLI `--proxy` > `HUBDL_PROXY` > `https_proxy` / `http_proxy` / `ALL_PROXY`**.

Only **HTTP(S) forward proxies** are supported (aria2 / urllib / wget common capability).  
**SOCKS is rejected** — point hubdl at a local HTTP port (Clash/V2Ray **mixed** / HTTP port), not `socks5://`.

```bash
# Recommended: local HTTP proxy (API + download, full path)
export https_proxy=http://127.0.0.1:7890
hubdl hf gpt2 --local-dir ./gpt2

# Explicit CLI (overrides env)
hubdl ms org/model --local-dir ./m --proxy http://127.0.0.1:7890 --no-proxy localhost,127.0.0.1

# Force direct when the shell has a global proxy
hubdl hf gpt2 --local-dir ./gpt2 --proxy none
# same: export https_proxy=none
```

- Child downloaders **do not inherit** ambient proxy env; hubdl injects or clears explicitly.
- aria2 also clears protocol-specific options so conf/`https_proxy` cannot override `--proxy none`.
- Credentials in the proxy URL are **redacted** in logs (still visible to same-uid `ps`); prefer unauthenticated localhost proxies.
- `last_run.json` stores scheme only (`http` / `https` / `none`).

---

## Hub differences

| | Hugging Face | ModelScope |
|--|--------------|------------|
| List API | metadata + siblings / recursive tree | `/api/v1/{models\|datasets}/…/repo/files` |
| Content hash source | Often LFS `sha256` / `oid` | API `Sha256` field |
| Default `--verify-hash` | `auto` | `on` |
| Default revision | `main` | `master` |
| Legacy repo id | Yes (`gpt2`) | `owner/name` only |
| Gated repos | Token required; username recommended | Token via env/`--token` |
| Large file counts | Resumable tree walk | Paged API (hard stop ~200 pages) |

Capabilities are **not** 100% symmetric; that is intentional.

---

## Security

Summary (see also [SECURITY.md](./SECURITY.md)):

1. **Sandbox:** repository paths cannot target `.hubdl`, `.hfdx`, `.msd`, `.hfd`, or `.git`.
2. **HTTP pin:** requests must stay on the configured endpoint host; cross-host redirects are blocked (reduces token exfiltration).
3. **aria2:** `--check-integrity=true`; URL list is wiped after download.
4. **Tokens:** prefer env vars; do not commit tokens or pack `.hubdl/` into CI artifacts.
5. **Mirrors:** a mirror can still observe your token on the first hop — treat mirrors as trusted parties.
6. **fix_bad:** can delete overflow/hash-bad files **inside** `local_dir` only; use `--no-fix-bad` when unsure.

---

## Troubleshooting

| Symptom | What to check |
|---------|----------------|
| Exit 3 / lock | Another hubdl on the same `--local-dir`; remove stale process before touching lock |
| Exit 2 / verify fail | Open `.hubdl/verify_report.json` → `bad` list; re-run download or `--verify-only` |
| Exit 1 / incomplete | Re-run the **same** command to resume; check `.hubdl/download.log` |
| Slow / limited speed | Is `--rate-mbps` or `lan-shared` set? Try adjusting `-x` / `-j` |
| 401 / gated | Token validity; HF username; accept model license on the website |
| Hash slow | First full L3 is disk-bound; raise `--verify-jobs`; rely on `hash_cache.tsv` later |
| Treesize WARN (HF) | Incomplete listing or filter; try `--refresh` or official endpoint |
| flock WARN on Windows | Expected; use Linux for production locks |
| aria2 not found | `sudo apt install aria2` or `--tool wget` |

```bash
tail -f <local-dir>/.hubdl/download.log
cat <local-dir>/.hubdl/last_run.json
cat <local-dir>/.hubdl/verify_report.json
cat <local-dir>/.hubdl/plan.json
```

---

## Development

```bash
source .venv/bin/activate   # or Windows Activate.ps1
pip install -e ".[dev]"
pytest -q
# with coverage:
pytest --cov=hubdl --cov-report=term-missing
python -m hubdl hf --help
```

Project layout for contributors mirrors the [Architecture](#architecture) section.

### Tests

| Layer | Coverage |
|-------|----------|
| Unit | sandbox, hash policy, sha normalize, patterns, HTTP host pin, atomic IO, cache, lock |
| Component | manifest IO, plan (missing/incomplete/overflow/hash/cache), verify pass/fail/light |
| Mocked hub | ModelScope page list, HF siblings + gated |
| Pipeline | offline `--verify-only`, dry-run, download incomplete (mocked aria2) |
| CLI | defaults, profiles, offline rules, entry aliases |

```bash
pytest -q
pytest --cov=hubdl --cov-report=term-missing
```

CI runs on Python 3.10–3.12 (GitHub Actions). Before a PR:

1. `pytest -q`
2. `hubdl hf gpt2 --dry-run --local-dir /tmp/hubdl-gpt2` (network)
3. `hubdl ms Qwen/Qwen2.5-0.5B-Instruct --dry-run --local-dir /tmp/hubdl-qwen --no-verify-hash`

---

## Roadmap

- [x] Shared core + HF / MS backends  
- [x] Dual CLI aliases (`hfdx`, `msd`)  
- [x] Official PyPI distribution (`pip install hubdl`)  
- [x] CI + Trusted Publishing (OIDC) to PyPI / GitHub Release  
- [ ] Stronger single-line progress HUD (hfd-style)  
- [ ] Optional migration from legacy `.msd` / `.hfdx` meta dirs  
- [ ] Broader integration tests against mocked Hub APIs  
- [ ] Optional standalone binary packaging

---

## Credits & license

**Inspired by / thanks to:**

- [hfd](https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f) by **padeoe** — Hugging Face listing + aria2 workflow  
- Local **msd** / **hfdx** tooling — L3/L4, sandbox, operational flags  

**License:** [Apache License 2.0](./LICENSE)  
**Attribution:** [NOTICE](./NOTICE)  
**Security policy:** [SECURITY.md](./SECURITY.md)

Model weights and Hub content remain under their respective licenses.

---

## Language

| Language | File |
|----------|------|
| **English (default)** | [README.md](./README.md) |
| **简体中文** | [README.zh-CN.md](./README.zh-CN.md) |
