MirrorURL

📖 Overview

MirrorURL mirrors public HTTP(S) directory trees and downloads explicitly selected files. Mirror mode discovers a directory listing, downloads missing or changed files, and can preview, move, or delete obsolete local files. Download mode accepts file paths or URL lists without discovery or freshness checks. The default cleanup policy preserves obsolete files.

This page documents v3.8.0. See the User Guide, Developer Guide, and changelog for the full project documentation.

📦 Installation

Python 3.10 or newer is required. CI covers Python 3.10–3.14. Pip installs httpx[http2], aiohttp, pydantic v2, PyYAML, portalocker (with pywin32 on Windows), and tqdm. Both transfer backends and progress bars work in a standard installation.

Install from PyPI

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install mirror-url

# Verify the package and command
python -m pip check
mirror-url --version
mirror-url --help

On Windows, activate the environment with .venv\Scripts\Activate.ps1 in PowerShell.

Optional packages and upgrades

# Optional acceleration and monitoring packages
python -m pip install "mirror-url[all]"

# Upgrade the installed package
python -m pip install --upgrade mirror-url

Extras: fast (stringzilla + lxml), monitor (psutil), and all (all optional runtime packages). The progress and aiohttp extras remain compatibility aliases; their features are included in the base package. Use dev for contributor tools.

# Install this documented release in an activated environment
python -m pip install "git+https://github.com/bpodlipnik/mirror-url.git@v3.8.0"

# Or build a wheel from a checkout
git clone https://github.com/bpodlipnik/mirror-url.git
cd mirror-url
git checkout v3.8.0
python -m pip install build
python -m build
python -m pip install "dist/mirror_url-3.8.0-py3-none-any.whl[all]"

# Or install an isolated command with pipx
pipx install mirror-url

FROM python:3.12-slim
COPY dist/mirror_url-3.8.0-py3-none-any.whl /tmp/
RUN pip install --no-cache-dir "/tmp/mirror_url-3.8.0-py3-none-any.whl[all]"
ENTRYPOINT ["mirror-url"]

Allow space for the mirrored data and staging files. See download modes for temporary storage requirements.

⚡ Quick Start

Replace https://example.com/data/ with your archive's directory-listing URL. Keep logs outside the destination tree.

1. Inspect the archive

# Immediate child directories; no destination or log path required
mirror-url --url https://example.com/data/ --list-dirs

# Last five filenames in each directory, sorted lexicographically
mirror-url --url https://example.com/data/ --list-files 5

2. Preview a run

mirror-url --url https://example.com/data/ \
  --dest-path ./mirror/data --log-path ./logs --dry-run

3. Mirror using the default cleanup policy

mirror-url --url https://example.com/data/ \
  --dest-path ./mirror/data --log-path ./logs

Re-run the same command for incremental synchronization. Obsolete local files are preserved by default; files identified as changed may still be replaced.

4. Choose cleanup explicitly

# Preview obsolete-file actions without changing mirrored files
mirror-url --url https://example.com/data/ \
  --dest-path ./mirror/data --log-path ./logs --cleanup preview --dry-run

# Archive obsolete files in a sibling data_obsolete directory
mirror-url --url https://example.com/data/ \
  --dest-path ./mirror/data --log-path ./logs --cleanup move

For repeatable jobs, create a configuration file and run mirror-url --config mirror.yaml.

🪞 Mirroring & Cleanup

Cleanup reconciles obsolete local files within the current remote selection. Filters, directory exclusions, depth limits, and skipped duplicate subtrees define that selection; files outside it are preserved. This is one-way synchronization from the remote archive to local storage.

# Explicit deletion after reviewing the selected scope
mirror-url --config mirror.yaml --cleanup delete --confirm-delete

Cleanup preserves local symlinks and the reserved .mirror-url-state/ directory. An incomplete scan skips cleanup and fails the run; a complete empty scan can clean the selected local files. Cleanup failures also fail the run. Failed moves preserve their source files and never fall back to deletion.

Freshness checks use available ETags, sizes, and timestamps. They do not compare downloaded files with a remote cryptographic digest, and size/time checks can miss changes that preserve those values.

📄 Download Known Files

Use --mode download when you already know the files to fetch. It downloads every selected file without listing directories, checking freshness, or cleaning obsolete files. Choose positional files or one --url-list, not both.

One file with shortcut defaults

mirror-url --mode download https://example.com/data/a.fits

# Choose persistent paths and show progress
mirror-url --mode download https://example.com/data/large.fits \
  --dest-path ./downloads --log-path ./logs \
  --backend aiohttp --streaming-parallel --print-logs --progress-bar

Without an explicit root, absolute file URLs sharing one parent infer that parent as their scope. Positional files default to the current directory and a destination-specific log folder under the system temporary directory. Explicit --url, --dest-path, and --log-path override those defaults.

Several files relative to the remote root

mirror-url --mode download --url https://example.com/data/ \
  a.fits nested/b.fits --dest-path ./downloads --log-path ./logs

This writes ./downloads/a.fits and ./downloads/nested/b.fits. Inputs are literal names, without glob expansion. --dir-suffix narrows selection; file inputs still include that suffix relative to --url.

A UTF-8 list of paths or absolute URLs

# Save as files.txt; blank lines and comments are ignored
a.fits
nested/b.fits
https://example.com/data/c.fits

mirror-url --mode download --url https://example.com/data/ \
  --url-list files.txt --dest-path ./downloads --log-path ./logs \
  --backend httpx --concurrency 6 --parallel-downloads --verify-content

All files must stay within the selected root and map to unambiguous local paths. Existing files require a matching ownership receipt or explicit --overwrite; symlinks remain forbidden. A matching receipt permits replacement and does not skip the download.

Both backends support verified chunks. Explicit chunk modes add range-metadata HEAD probes; ineligible files fall back to whole-file GETs. Known-file downloads restart interrupted files and do not resume partials across runs. Chunk scratch stays inside the destination, so omit --chunk-assembly-dir.

Download mode rejects discovery filters, --missing-files, --check-files, --auto-concurrency, and cleanup policies other than safe. Use mirror mode for incremental synchronization. See the known-URL guide.

🎯 Download Modes & Resume

Mirror mode selects a transfer method automatically unless you choose an explicit flag. Download mode uses whole-file GETs by default; either parallel flag enables eligible chunks. HTTPX (default) and aiohttp support both chunk methods in both modes. Aiohttp uses HTTP/1.1; --http2 applies to HTTPX.

--max-concurrent-downloads (alias --concurrency) limits files, --max-chunks limits chunks per file, and --max-parallel-chunks limits chunk work across files. --auto-concurrency adjusts admitted files for HTTPX mirror mode; it is rejected with aiohttp and in download mode.

Temporary storage

Traditional mode can require roughly two additional file-sized copies for temporary chunks and assembly; streaming mode roughly one staging allocation. Existing destination files remain in place until verified replacement. In mirror mode, --chunk-assembly-dir selects a parent for owned chunk workspaces; otherwise they use reserved destination state. Final assembly and streaming staging stay on the destination filesystem. Download mode requires chunk scratch inside the destination.

Interrupted transfers

HTTPX mirror whole-file partials live in owned .mirror-url-state/ storage and can resume when the recorded URL, size, and strong ETag match, including whole-file fallbacks from parallel modes. Partials without a strong validator restart. Resume uses enable_resume: true in configuration (the default); there is no --enable-resume CLI flag. Known-file downloads and aiohttp whole-file transfers restart interrupted files.

Chunk state does not resume across runs. After acquiring ownership, the next run reclaims recorded abandoned chunk work and staging. Live writers and unrecognized files are preserved. Staging is never published under the final filename until verified, and obsolete-file cleanup excludes reserved state.

⚙️ Configuration Files

Use YAML or JSON for repeatable runs. Explicit CLI flags override file settings, even when their values equal the built-in defaults. Omitted flags preserve the file's settings. Mirror and listing configs require base_url, dest_path, and log_path. CLI download configs with download_files or download_url can use shortcut defaults; standalone model validation still requires those three fields. Unknown keys are rejected.

# Save as mirror.yaml
base_url: https://example.com/data/
dest_path: ./mirror/data
log_path: ./logs
cleanup_policy: safe
mode: mirror
backend: httpx
verify_content: false

# Optional selection and download settings
file_filters: [".fits", ".txt"]
exclude_dirs: ["old_versions"]
max_depth: 50
streaming_parallel: true
max_concurrent_downloads: 6
max_chunks_per_file: 8
min_chunk_size_mb: 10
enable_resume: true

# Networking and diagnostics
workers: 8
timeout: 30
request_delay: 0.05
requests_per_second: 20
http2: true
security_validation: true
metrics_json: ./logs/run.json
health_check_port: 8080

# Preview, then run
mirror-url --config mirror.yaml --cleanup preview --dry-run
mirror-url --config mirror.yaml

# Override a setting explicitly
mirror-url --config mirror.yaml --max-concurrent-downloads 4

Config keys can differ from CLI flags: base_url, file_filters, exclude_dirs, cleanup_policy, and max_chunks_per_file are the model field names. dir_suffix is a single string in a config file; the CLI accepts several suffixes. Pass logging-handler controls such as --print-logs or --quiet on the CLI.

String values support ${VAR} environment placeholders. Unset variables leave placeholders unchanged. See the full configuration reference.

📋 CLI Argument Reference

Expand a group or search for an option. Descriptions and default settings below come from the v3.8.0 parser. Boolean defaults describe the underlying setting before CLI overrides; configuration files can supply different values. The table includes option aliases and positional file inputs. Compatibility settings are accepted but may not affect the current pipeline. Run mirror-url --help for the installed version.

Directory discovery uses up to min(--workers, 4) listing workers with --scan-mode parallel. Other scan modes use sequential discovery; async worker settings control metadata checks separately. Discovery keeps the full remote file list in memory, including with --use-disk-backed-sets.

💡 Practical Examples

These commands use placeholder URLs. Substitute your archive's directory-listing URL and choose paths with adequate free space.

Mirror only FITS files

mirror-url --url https://example.com/data/ \
  --dest-path ./fits-data --log-path ./logs \
  --filter .fits --exclude-dir old_versions

Filename filters are case-insensitive. Extensions match suffixes, plain text matches substrings, and patterns with regex metacharacters use regular expressions. Multiple filters are OR'd.

Mirror several subdirectories

mirror-url --url https://example.com/product/ \
  --dest-path ./mirror --log-path ./logs --dir-suffix L1/v2 L2/v1

Directory exclusions match exact paths relative to --url. To exclude a name at both root level and nested levels, use --exclude-dir old '*/old'.

Parallel transfers with metrics

mirror-url --config mirror.yaml --streaming-parallel \
  --max-concurrent-downloads 6 --max-chunks 8 --metrics-json ./logs/run.json

Conservative settings for a throttled server

mirror-url --config mirror.yaml --sequential-downloads \
  --no-async-metadata --workers 2 --request-delay 0.2

List a channel without downloading

mirror-url --url https://example.com/mission/ \
  --list-files 5 --filter _fe_

🚀 Performance Tuning

Start with automatic selection in mirror mode or whole-file GETs in download mode, then adjust limits after observing your server, disk, and run metrics. Throughput depends on the archive and workload.

• Eligible large files: try --streaming-parallel with a modest file limit. Range support and a strong ETag are still required for chunking.
• Many small files: tune --max-concurrent-downloads; chunk limits usually do not help files below the chunk-size threshold.
• Throttled servers: reduce file and metadata worker counts, increase --request-delay, and consider --no-async-metadata.
• Bandwidth sharing: set --bandwidth-limit in MB/s.
• Mostly static archives: --missing-files skips freshness checks except for paths selected by --check-files. Run occasional full checks for changes elsewhere.
• Large directory trees: limit scope with --dir-suffix, --exclude-dir, or --max-depth. --scan-mode parallel uses up to min(--workers, 4) directory-listing workers. Other scan modes use sequential discovery. Discovery keeps the remote file list in memory.
• Request pacing: the default budget is 20 requests/second with 50 ms spacing. Set both --requests-per-second 0 and --request-delay 0 to remove pacing. --trusted-server relaxes chunk limits; request spacing still follows your explicit settings.

Cache behavior

Keep --log-path stable to reuse the JSON metadata cache. --no-cache bypasses caches but still checks existing files for freshness; it does not force every file to download. Parsed HTML caches are process-local and are not restored on a new CLI launch.

--quick refreshes an existing cache timestamp without scanning or verifying files. It does not establish that a mirror is current.

Refresh selected existing files

mirror-url --config mirror.yaml --missing-files \
  --check-files L1/v2/a.fits L1/v2/nested/b.fits

# Or read root-relative paths from a UTF-8 list
mirror-url --config mirror.yaml --missing-files --check-files @checks.txt

Selected existing files receive normal freshness checks; all missing files in scope are still downloaded. Paths remain relative to --url, including any --dir-suffix. Without --missing-files, the run checks all selected existing files normally.

Verify local content against saved receipts

mirror-url --config mirror.yaml --verify-content

SHA-256 is opt-in. Each downloaded staging file is hashed before publication, and existing local files are hashed before trusting freshness metadata. Missing, expired, malformed, or mismatched receipts require a new download, so the first verified run of an existing mirror can download files again. Keep the log/cache path stable. Hashing adds disk I/O; it does not check a server-provided digest. --verify-content cannot be combined with --missing-files; --no-verify-content overrides configuration.

🔒 Security & Scope

• Transport IP restrictions: private, loopback, link-local, and direct-IP targets are rejected. Resolved addresses are validated before connection.
• URL scope: discovery and redirects are checked against the configured host and path.
• Filesystem protection: traversal rejection, filename sanitization, reserved-name handling, and checks for ambiguous local-path mappings. Distinct case-only names are preserved on confirmed case-sensitive destinations and rejected before downloads on case-insensitive destinations.
• Cooperative ownership: overlapping destination trees and shared log/cache/state paths reject competing runs. OS locks are released on process termination; external writers and network filesystems require separate coordination.
• Transfer consistency: eligible byte ranges are verified before atomic publication; no remote cryptographic digest comparison is performed.
• Resilience: request pacing, retry backoff, and circuit breakers handle failing or overloaded services.

Optional duplicate-subtree detection

--handle-symlinks compares non-empty directory entry names to report possible duplicate subtrees. This heuristic can flag unrelated directories and cannot identify individual file symlinks. Start with detection and review the reported paths:

mirror-url --config mirror.yaml --handle-symlinks --symlink-mode detect --print-logs

detect implies --dry-run. skip omits detected duplicates; follow remains subject to scope and tracker limits; treat-as-file behaves like skip. See the symlink-handling guide before enabling automatic skipping.

📊 Monitoring & Exit Codes

• The normal completed-sync path logs a metrics summary at INFO level. --stats is an inactive compatibility setting; early exits produce shorter summaries.
• --metrics-json FILE exports JSON metrics and enables a localhost health/metrics server for non-dry-run instances.
• --health-check-port selects the port (default 8080); it does not enable the server on its own.
• --progress-bar (alias --progress) works in a standard installation. Both modes show per-file byte/stage progress; chunked transfers include active and completed chunks. --print-logs enables console logs.

Shared transfer reporting

Mirror and download modes use the same startup header, with mode, start time, sanitized command, remote root, destination, log file, backend, and SHA-256 setting. Transfer logs show the actual file/worker plan, explain whole-file fallbacks, and give every published file one Downloaded: line. A completed batch reports File download batch completed: N downloaded, M failed, including sequential transfers and partial failures.

The shared SUMMARY reports this run's published bytes, successful and failed files, timings, throughput, selected and validated chunks, files finalized from chunks, observed peak chunk workers, and SHA-256 setting/hash count. Completion output does not recursively size the destination. With --verify-content, each downloaded file's completion line includes its computed SHA-256. Local receipts do not establish agreement with a remote checksum.

Transfer-stage timings show range metadata probes, data transfer/writes, temporary chunk assembly, payload disk flushing and opt-in SHA-256 hashing. Each is labelled as a worker sum: concurrent work overlaps, so these sums can exceed elapsed download time. Both modes and backends use the same fields; elapsed download time remains the throughput denominator.

In download mode, --print-logs without --quiet prints readable logs and the summary to stderr, leaving stdout empty. Otherwise the CLI prints exactly one JSON result to stdout for scripts. Progress bars work independently of console logs; --quiet suppresses bars and INFO-level console output.

# Capture one JSON result for scripting
mirror-url --mode download https://example.com/data/a.fits >result.json

# During a run with --metrics-json enabled
curl http://localhost:8080/health
curl http://localhost:8080/metrics

Both endpoints return JSON. Metrics export and the health server are disabled during dry runs; the server is cleaned up when the instance closes.

A graceful SIGINT/SIGTERM shutdown currently exits with code 0 even when synchronization was interrupted. Check the completion summary for interrupted scheduled jobs. The Python API returns a boolean from sync().

🐍 Python API

from pathlib import Path
from mirror_url import MirrorConfig, MirrorURL

config = MirrorConfig(
    base_url="https://example.com/data/",
    dest_path=Path("./mirror/data"),
    log_path=Path("./logs"),
    file_filters=[".fits"],
    parallel_downloads=True,
)

with MirrorURL(config) as mirror:
    ok = mirror.sync()

print("sync succeeded" if ok else "sync failed")

Use the context manager to clean up connection pools, workers, async components, and any health server. sync() blocks and returns True or False. Use MirrorConfig.from_yaml(Path("mirror.yaml")) to load a file.

For model validation, use MirrorConfig.model_validate(data). To obtain informational warnings for an existing config, use MirrorConfig.validation_warnings(config). Code that used MirrorConfig.validate(config) for warning strings must use the new warning helper.

🔧 Troubleshooting

Inspect a run

mirror-url --config mirror.yaml --debug --print-logs --dry-run

Uninstall

python -m pip uninstall mirror-url
# For a pipx installation:
pipx uninstall mirror-url

Uninstalling leaves mirrored files, logs, and cache data in place. For more help, read the User Guide or report a reproducible problem on GitHub Issues.

🔎 Choosing a Mirroring Tool

Choose according to the source protocol and the workflow you need. The links below point to each project's own documentation.

For MirrorURL, the current runtime discovers the remote file list in memory and performs no remote cryptographic digest comparison. Check the archive's listing format, metadata, and Range support when choosing a tool.