Metadata-Version: 2.4
Name: hydrastream
Version: 1.2.0
Summary: Concurrent HTTP downloader with in-memory stream reordering (curl_cffi + uvloop).
Keywords: async,streaming,multiplexing,download,in-memory,bioinformatics,data-engineering,mlops,aimd,uvloop
Author: Valentin Zhukovetski
Author-email: Valentin Zhukovetski <zukovetski@gmail.com>
License-Expression: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Networking
Classifier: Topic :: Utilities
Requires-Dist: curl-cffi>=0.15.0
Requires-Dist: orjson>=3.11.7
Requires-Dist: pydantic>=2.13.3
Requires-Dist: pydantic-settings>=2.14.0
Requires-Dist: pygments>=2.20.0
Requires-Dist: rich>=14.3.2
Requires-Dist: typer>=0.24.1
Requires-Dist: uvloop>=0.22.1 ; sys_platform != 'win32'
Requires-Dist: pywin32 ; sys_platform == 'win32'
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/Zhukovetski/HydraStream
Project-URL: Repository, https://github.com/Zhukovetski/HydraStream
Project-URL: Issues, https://github.com/Zhukovetski/HydraStream/issues
Description-Content-Type: text/markdown

# HydraStream

[![PyPI version](https://badge.fury.io/py/hydrastream.svg)](https://pypi.org/project/hydrastream/)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Coverage: 75%](https://img.shields.io/badge/coverage-75%25-brightgreen.svg)](https://github.com/Zhukovetski/HydraStream)
[![Tests](https://github.com/Zhukovetski/HydraStream/actions/workflows/tests.yml/badge.svg)](https://github.com/Zhukovetski/HydraStream/actions/workflows/tests.yml)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/HydraStream/HydraStream)

<p align="center">
  <img src="https://raw.githubusercontent.com/Zhukovetski/HydraStream/main/assets/Demo.gif" alt="HydraStream Demo" width="800">
</p>

HydraStream is a concurrent HTTP downloader written in Python. It fetches file chunks concurrently using HTTP Range requests and utilizes an internal min-heap to reorder out-of-sequence chunks in memory. This enables the piping of large remote files directly to `stdout` without requiring intermediate disk storage.

## Core Characteristics

* **Chaos-Tested Resilience & "Laptop-Lid" Recovery**: Hardened against severe network and OS anomalies using continuous CI fault injection (`tc qdisc` and `iptables`). HydraStream guarantees deterministic recovery from total internet outages, massive packet loss, and OS-level process suspensions (e.g., closing your laptop lid). If a socket dies, the affected worker seamlessly requeues the chunk and resumes via HTTP `Range` requests without losing a single verified byte.
* **In-Memory Assembly & On-the-Fly Hashing**: Converts concurrent, out-of-order HTTP chunks into a sequential byte stream using a min-heap. Enables direct piping to `stdout`. Computes cryptographic hashes (MD5, SHA-256, BLAKE2, etc.) incrementally as the stream is yielded, ensuring integrity without buffering the full file.
* **POSIX-Compliant Telemetry**: Strictly adheres to the Unix philosophy by routing all diagnostic outputs, progress bars, and warnings to `stderr`. This guarantees a completely pure `stdout` for binary data pipes. Supports structured JSON Lines logging for CI/CD integration.
* **Network Resilience & Chaos-Tested Reliability**: Hardened against adversarial network conditions using Property-Based Testing (Hypothesis) and fault injection. Guarantees deadlock-free recovery, pipeline termination, and data integrity during `429`/`503` cascades, connection drops, and missing `Range` headers.
* **Strict Data Verification**: Enforces multi-layered integrity checks. Automatically validates payload size against remote metadata and performs strict post-download checksum validation to eliminate silent data corruption.
* **Auto-Scaling Concurrency & Throttling**: Implements an AIMD (Additive Increase/Multiplicative Decrease) algorithm to dynamically adjust active workers based on network health. Supports hard bandwidth throttling (`--limit`) for controlled environment execution.
* **Actor-Based Architecture & Lock-Free Synchronization**: Pipeline components (feeders, resolvers, dispatchers, workers) operate as isolated asynchronous actors (CSP pattern). Solves Fan-In/Fan-Out race conditions using prioritized sentinel values (poison pills) instead of shared-memory mutexes.
* **Zero-Lock Disk I/O**: Leverages `os.pwrite` within a dedicated thread pool to write scattered chunks concurrently. Completely bypasses GIL contention and traditional file locking mechanisms during disk operations.
* **Dry-Run Protocol**: Provides a safe simulation mode (`--dry-run`) to preemptively fetch remote metadata, verify available local disk space, and resolve target hashes without allocating space or initiating data transfer.
* **TLS Fingerprint Spoofing**: Integrates `curl_cffi` to mimic real browser TLS signatures (e.g., Chrome 120), bypassing strict WAFs (Web Application Firewalls) and Deep Packet Inspection (DPI) heuristics.
* **Layered Configuration & Domain-Driven Design**: Features a strict boundary between the core engine and CLI. Seamlessly merges CLI arguments and global TOML configurations (`~/.config/hydrastream/config.toml`) via a late-binding validation layer.


## Installation

Requires Python 3.12+.

```bash
uv tool install hydrastream
```
or
```bash
pipx install hydrastream
```

## Usage

### 1. Download to Disk
Downloads the specified file to the output directory using dynamically scaled threads.:
```bash
hs "https://ftp.ncbi.nlm.nih.gov/.../genome.fna.gz" -t 20 --output ./data
```
<p align="center">
  <img src="https://raw.githubusercontent.com/Zhukovetski/HydraStream/main/assets/HydraStream-Demo.gif" alt="HydraStream Demo" width="800">
</p>

### 2. Stream to stdout (Pipe)
Downloads the file in memory and streams binary data to `stdout`. The `--quiet` (`-q`) flag is used to suppress logging output to `stderr`.:
```bash
hs "https://ftp.ncbi.nlm.nih.gov/.../genome.fna.gz" -t 20 --stream -q | zcat | wc -l
```
<p align="center">
  <img src="https://raw.githubusercontent.com/Zhukovetski/HydraStream/main/assets/Pipeline-Streaming-Demo.gif" alt="Pipeline Streaming Demo" width="800">
</p>

### 3. Batch Processing
Reads target URLs from a local file.

```bash
hs --input urls.txt --threads 20 --output ./datasets
```

## Configuration

HydraStream supports layered configuration. Default parameters can be defined in a TOML file located at `~/.config/hydrastream/config.toml`. CLI arguments override these defaults.

```toml
# ~/.config/hydrastream/config.toml
threads = 128
output_dir = "~/downloads"
verify = true
speed_limit = 50.0
min-chunk-mb = 5
```

### 4. Python API

```python
import asyncio
from hydrastream import HydraDaemon, HydraConfig, UIConfig


async def main():
    config = HydraConfig(threads=20)
    ui_config = UIConfig(quiet=True)
    url = ["https://example.com/file1.gz"]

    async with HydraDaemon(config=config, ui_config=ui_config) as daemon:
        task_id = await daemon.add_download(url)

        if task_id is not None:
            file_stream = await daemon.get_stream(task_id)

        if file_stream is not None:
            # Returns an async generator yielding chunk_generator
            async for chunk in file_stream:
                sys.stdout.buffer.write(chunk)


if __name__ == "__main__":
    asyncio.run(main())
```

## CLI Options

HydraStream supports layered configuration. Options can be passed as CLI arguments or defined in `~/.config/hydrastream/config.toml`. CLI flags take precedence.

| Option | Shortcut | Default | Description |
| :--- | :---: | :---: | :--- |
| `LINKS` | - | `None` | One or multiple target URLs to download (positional argument). |
| `--input` | `-i` | `None` | Read URLs from a text file or `-` for stdin. |
| `--typehash` | `-th` | `md5` | Hash algorithm type (e.g., `md5`, `sha256`). |
| `--checksum` | `-c` | `None` | Expected hash checksum (applicable only for a single URL). |
| `--output` | `-o` | `downloads/` | Destination directory for downloaded files. |
| `--threads` | `-t` | `Auto` | Number of concurrent download connections (scales up to 128). |
| `--stream` | `-s` | `False` | Enable streaming mode (redirects binary data to `stdout`). |
| `--dry-run` | `-dr` | `False` | Simulate the process (fetch metadata, check disk space) without downloading. |
| `--min-chunk-mb` | `-mcm` | `1` | Minimum chunk size in Megabytes for standard disk downloads. |
| `--stream-chunk-mb` | `-scm` | `5` | Target chunk size in Megabytes for streaming mode. |
| `--buffer` | `-b` | `None` | Maximum stream buffer size in Megabytes to prevent OOM. |
| `--limit` | `-l` | `None` | Global download bandwidth throttle limit in MB/s. |
| `--no-ui` | `-nu` | `False` | Disable GUI (progress bars). Leaves plain text logs. |
| `--quiet` | `-q` | `False` | Dead silence. No console output at all. Logs are still written to file. |
| `--json` | `-j` | `False` | Output logs in structured JSON Lines format. |
| `--verify` / `--no-verify` | `-V` / `-N` | `True` | Verify the downloaded file hash. Use `--no-verify` to skip. |
| `--browser` | `-B` | `chrome120` | Browser TLS fingerprint to impersonate (e.g., `chrome120`, `safari153`). |
| `--debug` | `-d` | `False` | Enable debug mode (propagates full exception tracebacks). |
| `--version` | `-v` | - | Show application version and exit. |

## Roadmap

### **v2.0: Rust Core:**

Port the core engine to Rust (`tokio`/`reqwest`) with a `PyO3` wrapper to bypass the Python GIL and improve multi-core execution.

## License

MIT License. See the [LICENSE](LICENSE) file for details.
