Metadata-Version: 2.4
Name: mirror-url
Version: 3.1.17
Summary: Enterprise-grade remote directory mirroring tool with adaptive concurrency, integrity checks, and SSRF-hardened transport.
Author: BP
License: MIT License
        
        Copyright (c) 2026 BP
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/bpodlipnik/mirror-url
Project-URL: Repository, https://github.com/bpodlipnik/mirror-url
Project-URL: Issues, https://github.com/bpodlipnik/mirror-url/issues
Keywords: mirror,download,http,scraper,directory,async,httpx
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: System :: Archiving :: Mirroring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.24
Requires-Dist: pydantic>=2.0
Requires-Dist: PyYAML>=6.0
Provides-Extra: fast
Requires-Dist: stringzilla>=3.0; extra == "fast"
Requires-Dist: lxml>=4.9; extra == "fast"
Provides-Extra: progress
Requires-Dist: tqdm>=4.65; extra == "progress"
Provides-Extra: monitor
Requires-Dist: psutil>=5.9; extra == "monitor"
Provides-Extra: all
Requires-Dist: stringzilla>=3.0; extra == "all"
Requires-Dist: lxml>=4.9; extra == "all"
Requires-Dist: tqdm>=4.65; extra == "all"
Requires-Dist: psutil>=5.9; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: black>=24.0; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: types-PyYAML>=6.0; extra == "dev"
Requires-Dist: pre-commit>=3.5; extra == "dev"
Dynamic: license-file

# MirrorURL

[![CI](https://github.com/bpodlipnik/mirror-url/actions/workflows/ci.yml/badge.svg)](https://github.com/bpodlipnik/mirror-url/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.9%E2%80%933.12-blue)](https://www.python.org)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](./LICENSE)

Enterprise-grade remote directory mirroring tool. MirrorURL recursively
discovers files behind an HTTP(S) directory listing and mirrors them locally
with adaptive concurrency, resumable/partial downloads, integrity verification,
and an SSRF-hardened transport layer.

> **Status:** Refactor complete. The full implementation now lives in the
> [`src/mirror_url/`](./src/mirror_url) package (30 modules), migrated verbatim
> from the original single-file script per [`REFACTORING_PLAN.md`](./REFACTORING_PLAN.md).
> The legacy [`mirror_url.py`](./mirror_url.py) is retained only as a frozen
> reference and can be deleted once you've run the test suite against the package
> in an environment with the runtime dependencies installed (see Development).

## Features

- **Recursive discovery** of remote directory trees (BFS, depth/exclude limits, cycle-safe).
- **True parallel downloads** — multiple files and multiple chunks per file concurrently.
- **Adaptive async concurrency** that tunes itself to server RTT, throughput, and error rate.
- **Resumable & partial downloads** with HTTP range requests and chunk assembly.
- **Integrity checks** — size/timestamp comparison, ETag handling, content hashing.
- **Resilience** — per-domain circuit breakers, exponential backoff, rate limiting.
- **Security** — path-traversal and symlink-bomb defenses, private-IP/SSRF guards, URL-scope enforcement.
- **Operability** — metrics collection, multi-level progress, optional HTTP health-check server.
- **Caching** — filesystem and disk-backed indexes to skip unchanged content.

## Installation

```bash
# From source (editable, recommended during the refactor)
pip install -e ".[all,dev]"
```

Python 3.9+ is required. Core dependencies: `httpx`, `pydantic` (v2), `PyYAML`.
Optional extras: `fast` (stringzilla, lxml), `progress` (tqdm), `monitor` (psutil).

## Usage

Run via the console entry point or the module:

```bash
mirror-url https://example.com/files/ --output ./mirror
# or
python -m mirror_url https://example.com/files/ --output ./mirror
```

Run `mirror-url --help` for the full option list.

Configuration can also be supplied via a YAML file (see `MirrorConfig` /
`load_config_from_args`).

📖 **Full documentation:** see the [User Guide](./docs/USER_GUIDE.md)
([HTML version](./docs/USER_GUIDE.html)) for detailed installation, CLI
reference, config-file format, download modes, security notes, the Python API,
and troubleshooting.

🛠 **Contributing to the code?** The [Developer Guide](./docs/DEVELOPER_GUIDE.md)
([HTML version](./docs/DEVELOPER_GUIDE.html)) is an architecture deep-dive:
dependency layers, the `MirrorURL` mixin design, runtime data flow, and
step-by-step extension recipes.

## Development

```bash
pip install -e ".[dev]"
pre-commit install

ruff check .          # lint
black --check .       # format check
mypy                  # type-check the new package
pytest -m "not integration"   # fast test lane
pytest                # full suite (includes integration)
```

Continuous integration runs lint + tests across Python 3.9–3.12 (see
`.github/workflows/ci.yml`).

## Project layout

```
src/mirror_url/          # the package (30 modules, dependency-layered)
tests/                   # pytest suite
mirror_url.py            # legacy monolith (frozen reference, pending removal)
REFACTORING_PLAN.md      # module breakdown + migration roadmap
CHANGELOG.md             # notable changes (Keep a Changelog format)
CONTRIBUTING.md          # dev setup, checks, conventions
pyproject.toml           # packaging, deps, tool config
```

## Contributing

Contributions are welcome — see [CONTRIBUTING.md](./CONTRIBUTING.md) for the dev
setup, the checks CI runs, and the project conventions (dependency layering,
lint policy, behavior-preserving refactors). Notable changes are tracked in
[CHANGELOG.md](./CHANGELOG.md).

## License

[MIT](./LICENSE) © BP
