MirrorURL — User Guide

MirrorURL is a command-line tool and Python library for mirroring and incrementally syncing public HTTP(S) directory trees to local disk. It walks HTML directory listings, decides which files are new or changed, and downloads them with adaptive concurrency, resumable/parallel downloads, integrity checks and incremental caching. It also supports direct file URLs and exact URL lists without directory discovery. URL-scope, private-network and filesystem protections guard discovery and transfers.

• Version: 3.4.6
• Python: 3.10 or newer; CI tests Python 3.10–3.14
• License: MIT

Table of contents

• What it does
• Who it is for
• Requirements
• Installation
• Quick start
• Command-line usage
• Configuration files (YAML/JSON)
• Download modes
• Known-URL downloads
• Filtering and scope
• Filename collisions and download behavior
• Caching and incremental sync
• Cleaning up obsolete files
• Security
• Symlink handling
• Monitoring and metrics
• Using MirrorURL from Python
• Exit codes
• Troubleshooting
• Uninstalling

What it does

Given a base URL that serves an HTML directory index (e.g. an Apache/nginx "Index of /…" page, a data archive, an artifact server), MirrorURL:

1. Discovers the remote tree by recursively parsing directory listings (breadth-first, with depth and exclusion limits, cycle-safe).
2. Validates local names before file transfers. Distinct selected remote URLs must map to safe, unchanged, non-conflicting local paths.
3. Compares each remote file against the local copy using size, timestamp, and ETag. Directory listing validators never stand in for file checks.
4. Downloads the missing/changed files, optionally in parallel (multiple files and/or multiple chunks per file) with resume support.
5. Optionally cleans up local files that no longer exist remotely (preview / move / delete policies).

Highlights: adaptive async metadata checks, per-domain circuit breakers, bandwidth limiting, integrity verification, a persistent cache for fast incremental runs, and strong SSRF/path-traversal protections.

Who it is for

MirrorURL is for people and organizations that regularly need to maintain local copies of files published over public HTTP(S):

• Researchers maintaining local copies of scientific data archives.
• Organizations downloading bulk files from public directory indexes.
• Maintainers of software, package, artifact and documentation mirrors.
• Sysadmins and data engineers maintaining local datasets.
• Public-data preservation communities.
• Users moving from wget --mirror, lftp mirror or custom scripts.

Scientific mission archives used in the examples illustrate workflows that also apply to these other sources. Recursive mirroring requires an HTML directory index; direct file and URL-list downloads do not require a listing.

Requirements

• Python 3.10 or newer.
• Runtime dependencies (installed automatically): httpx (with the http2 extra, which pulls in h2 -- HTTP/2 is on by default, see --no-http2), pydantic (v2), PyYAML, portalocker 3.x (and pywin32 on Windows), and tqdm (progress bars).
• Optional accelerators (install via extras, see below): stringzilla + lxml (faster parsing), psutil (memory/disk monitoring).

Installation

Always install into a virtual environment to keep dependencies isolated.

From PyPI

pip install mirror-url

From a built wheel (recommended for servers)

From a checkout of the repository on a build machine:

pip install build
python -m build          # produces dist/mirror_url-3.4.6-py3-none-any.whl

Copy the wheel to the target server and install it:

python3 -m venv /opt/mirror-url
/opt/mirror-url/bin/pip install /tmp/mirror_url-3.4.6-py3-none-any.whl
/opt/mirror-url/bin/mirror-url --help

To include the optional speed extras:

/opt/mirror-url/bin/pip install "/tmp/mirror_url-3.4.6-py3-none-any.whl[fast]"

Available extras: fast (stringzilla + lxml), aiohttp (whole-file backend), monitor (psutil), all (everything), dev (test/lint toolchain). The progress extra remains compatible; tqdm is installed with the base package.

From a Git repository

pip install "git+https://github.com/bpodlipnik/mirror-url.git@v3.4.6"
# private repo over SSH:
pip install "git+ssh://git@github.com/bpodlipnik/mirror-url.git@v3.4.6"

As an isolated CLI with pipx

pipx install /tmp/mirror_url-3.4.6-py3-none-any.whl
# or:  pipx install "git+https://github.com/bpodlipnik/mirror-url.git@v3.4.6"

With Docker

FROM python:3.12-slim
COPY dist/mirror_url-3.4.6-py3-none-any.whl /tmp/
RUN pip install --no-cache-dir "/tmp/mirror_url-3.4.6-py3-none-any.whl[fast]"
ENTRYPOINT ["mirror-url"]

Verify the install

mirror-url --help
python -c "import mirror_url; print(mirror_url.__version__)"

The package exposes two equivalent entry points: the mirror-url console command and python -m mirror_url.

Quick start

Replace the example URL with your archive's directory-listing URL, then mirror it to a local folder:

mirror-url \
  --url https://example.com/datasets/ \
  --dest-path ./mirror \
  --log-path ./logs

• --url — the base URL to mirror (must serve an HTML directory listing).
• --dest-path — where files are written locally.
• --log-path — where run logs and the incremental cache are stored.

Re-running the same command later performs an incremental sync: only new or changed files are downloaded.

Prefer a config file for anything non-trivial:

mirror-url --config mirror.yaml

Command-line usage

Either supply --url, --dest-path, and --log-path, or point at a config file with --config. You can also use mirror-url --mode download FILE_URL with shortcut defaults, described under known-URL downloads. Run mirror-url --help for the complete, authoritative list of options. The most commonly used options:

--list-dirs and --list-files are exceptions: since they only discover and print the remote tree and never download or delete anything, neither requires --dest-path or --log-path when using --url without --config. Config-file listing and mirror runs still require base_url, dest_path, and log_path. See their entries in "Filtering and scope" below.

Targets

Download method

Performance and networking

Caching

Filtering and scope

Cleanup of obsolete local files

Output and diagnostics

Without --log-file, each --dir-suffix gets its own log file named mirror_url_<suffix>_<timestamp>.log. With --log-file NAME, the filename normally follows NAME_<suffix>_<timestamp>.log — where <suffix> is every --dir-suffix value joined with underscores (or all if none were given), and <timestamp> is YYYYMMDD_HHMMSS. With more than one --dir-suffix, all of them share this single log file rather than each getting a separate one. Unsafe characters in the combined name are replaced with underscores, and long names are shortened with a digest to fit the filename limit. This is handy for wrapper scripts that invoke mirror-url once per date/target and want a recognizable, greppable filename prefix — e.g. --log-file mirror_url_lasco_ql_nrl on a run with --dir-suffix 260727 produces mirror_url_lasco_ql_nrl_260727_20260727_030308.log.

Examples

Mirror only FITS files, large parallel downloads, export metrics:

mirror-url \
  --url https://archive.example.org/mission/ \
  --dest-path /data/mission \
  --log-path /var/log/mirror \
  --filter .fits \
  --streaming-parallel --max-concurrent-downloads 6 \
  --metrics-json /var/log/mirror/run.json

Preview obsolete-file actions without downloading or changing mirrored files:

mirror-url --config mirror.yaml --cleanup preview --dry-run

Dry-run to see what a first sync would download:

mirror-url --url https://example.com/files/ --dest-path ./m --log-path ./l --dry-run

See what's available under a base URL before picking a --dir-suffix (no --dest-path/--log-path needed):

mirror-url --url https://archive.example.org/mission/ --list-dirs

List the last 5 files in each directory (no --dest-path/--log-path needed):

mirror-url --url https://archive.example.org/mission/L3_png/v03/ --list-files 5

Mirror several versioned subdirectories in one run:

mirror-url --url https://example.com/product/ \
  --dest-path ./mirror --log-path ./logs \
  --dir-suffix L1/v2 L2/v1

Conservative settings for a slow/throttled server:

mirror-url --config mirror.yaml \
  --sequential-downloads --no-async-metadata --workers 2 --request-delay 0.2

Configuration files (YAML/JSON)

For repeatable jobs, put settings in a YAML (or JSON) file and run mirror-url --config mirror.yaml. Any CLI flag you actually type on the command line overrides the same setting in the file — including a flag whose value happens to equal its own default (e.g. --workers 8 overrides a file's workers: 4 even though 8 is also the built-in default). A flag you don't type is left alone at whatever the file says. Only base_url, dest_path, and log_path are required for mirror/listing config runs. Direct download configs can use the CLI shortcut defaults described under known-URL downloads. Standalone config validation still requires explicit target fields. Unknown keys are rejected. dir_suffix is one string in a config file; the CLI's --dir-suffix accepts several values. Logging controls such as --print-logs, --quiet, --verbose, --debug, and --log-file should be passed on the CLI to control its shared logging handlers.

Choosing --list-files or --list-dirs switches off the opposing listing mode inherited from the file. A configuration that enables both modes is rejected.

To enable a boolean explicitly over a false config-file value, use its positive flag, for example --async-metadata, --adaptive-async, --cache-html, --http2, --security-validation, --circuit-breaker-enabled, --connection-pool-prewarm, or --fast-parsing-fallback.

# mirror.yaml
base_url: https://archive.example.org/mission/
dest_path: /data/mission
log_path: /var/log/mirror

# Performance
workers: 8
async_metadata: true
async_workers: 50
timeout: 30
max_retries: 3
request_delay: 0.05
http2: true
trusted_server: false

# Download method (pick at most one; omit for auto-select)
parallel_downloads: false
streaming_parallel: false
sequential_downloads: false
max_concurrent_downloads: 10
max_chunks_per_file: 8
min_chunk_size_mb: 10
bandwidth_limit: null          # e.g. 50  (MB/s)
enable_resume: true            # whole-file transfers; no CLI toggle

# Filtering
file_filters: [".fits", ".txt"]
exclude_dirs: ["thumbnails", "old"]
max_depth: 50

# Caching
no_cache: false
refresh_cache: false
cache_max_age: 7               # days
cache_html: true
html_cache_max_age: 24         # hours
no_etag: false

# Cleanup of obsolete local files: safe | preview | move | delete
cleanup_policy: safe
confirm_delete: false

# Integrity / security
hash_algorithm: md5            # md5 | sha256 | blake2b
security_validation: true
circuit_breaker_enabled: true

# Output
progress_bar: false
metrics_json: null             # also enables the local health server when set
health_check_port: 8080

Environment variables in config

String values may contain ${VAR} placeholders, expanded from the environment at load time. An unset variable leaves its placeholder unchanged; export the variables before running and check the resulting paths/URL:

base_url: ${ARCHIVE_BASE}/mission/
dest_path: /srv/${SERVICE_USER}/mirror

Validate a config without running

python -c "from mirror_url.config import validate_config_file; \
from pathlib import Path; print(validate_config_file(Path('mirror.yaml')))"
# -> (True, None)  on success, or (False, '<error message>')

Download modes

Choose one of the three explicit modes, or omit the mode flags for automatic selection. Auto-selection considers file count and sizes, a disk-speed probe, an estimated network speed, and server Range support.

--max-concurrent-downloads caps parallel files; --max-chunks caps chunks per file and --max-parallel-chunks caps chunk work across files. These are separate from metadata-worker limits. --auto-concurrency tunes the admitted file count within --max-concurrent-downloads for either parallel mode, including when auto selection chooses it. If the chunk manager cannot initialize, auto mode falls back to sequential whole-file downloads.

Parallel chunking requires a known size at least --min-chunk-size, byte Range support, and a strong ETag. Without these, the file uses a whole-file transfer, which may still run alongside other files. Each chunk sends If-Range and must return the exact requested Content-Range, length, and ETag. Existing destination files stay intact until verification and atomic replacement succeed. These checks do not compare against a server-provided cryptographic content digest; --hash-algorithm does not add such a comparison.

Traditional chunk files default to owned .mirror-url-state/chunks/ state, or a destination-specific owned child of --chunk-assembly-dir. Final assembly and streaming staging use .mirror-url-state/parallel/ on the destination's filesystem. Traditional mode can need roughly two additional file-sized copies across the temporary and destination storage; streaming needs roughly one additional file-sized staging allocation. Existing destination copies also continue to occupy space until replacement. Assembly reads bounded blocks rather than loading an entire chunk into memory. Both chunk modes restore valid server Last-Modified timestamps before saving cache metadata; an unsupported local timestamp update is logged without failing an otherwise successful transfer.

Interrupted downloads and reserved state

Whole-file transfers use an owned .mirror-url-state/ directory inside each target directory (--dest-path plus any --dir-suffix). A .partial.json sidecar records the source URL, total size, and strong ETag. With enable_resume: true (the default), matching validated partials can resume. Partials without a strong validator restart from zero. A response of 200 to a resume request restarts the transfer; 416 triggers a fresh full request and never certifies the partial as complete.

The .mirror-url-state name is reserved case-insensitively. A conflicting remote path fails the sync before downloading. An existing directory without MirrorURL's valid ownership marker is preserved and rejected when partial state is initialized; obsolete-file cleanup never enters this directory.

Legacy adjacent *.mirror-partial files are not resumed. Review or archive them before enabling obsolete-file cleanup, which may remove them if they are absent from the selected remote listing.

Traditional and streaming chunk state is not resumed across runs; failed chunks can retry within a run. The next owner reclaims recorded abandoned work after acquiring destination/state locks. Per-work OS leases protect writers still finishing after bounded cleanup, including within a multi-suffix CLI run. Manifests are written before transfer bytes; unknown entries, malformed markers, symlinks and hardlinks are preserved for review. Incomplete manifests may leave small unrecognized workspaces requiring manual review. Legacy unmarked chunk directories and adjacent .assembling/.streaming files are not automatically reclaimed; requested obsolete-file cleanup can still archive adjacent legacy files. Legitimate MOVE archive bytes are preserved. Nested mount points must share the reserved staging state's filesystem for atomic publication.

Failed-sync summaries identify download failures, incomplete remote scans and cleanup failures separately. A zero failed-download count does not certify a complete scan or successful cleanup. --no-etag disables freshness comparisons, while range transfers still require ETags to verify that all bytes belong to the same remote representation.

Concurrent runs and destination ownership

MirrorURL acquires cooperative OS locks before starting a mirror. Overlapping destinations (including parent/child trees), shared cache or log files, configured chunk/disk cache directories, metrics files and MOVE archives reject a competing run with DestinationLockError; the CLI exits with status 1. Separate trees and state paths can run concurrently. Dry runs also acquire ownership because cache loading can repair existing cache state. CLI runs reserve their log directory and all selected suffixes before shared logging starts, and retain them until the whole run finishes. Destination keys are conservatively case-folded and Unicode-normalized, so equivalent names conflict even on a case-sensitive filesystem.

Lock files live in ~/.mirror-url/locks-v1/, outside the destination. They remain after cleanup; do not delete them while any mirror is running. Process termination releases the OS locks automatically, including after a hard kill. No stale PID file needs removal. Use with MirrorURL(config) or call cleanup() explicitly. Cleanup rejects new work; a writer still running after a shutdown timeout retains ownership until it stops.

This protects cooperating MirrorURL processes using the same account, home directory and local filesystem. Programs that do not participate in these locks can still modify the same tree. Network filesystems, multiple hosts, mount aliases and external writers need separate coordination.

Known-URL downloads

URL-list mode downloads every selected URL with a whole-file GET; it performs no discovery, HEAD freshness checks, ETag reuse or ranged resume. It never deletes obsolete local files. Use normal mirror mode when you need incremental synchronization.

Known-URL downloads sample destination free space once per batch, before the first payload write. A shared budget reserves each declared file length and charges unknown-length responses as bytes arrive, while keeping write headroom. Published files remain charged; removing a failed attempt's owned staging returns its reservation. This is internal accounting, not a filesystem allocation. If another program consumes the space, write or flush errors still fail the transfer and preserve the existing destination file.

HTTPX downloads with one active worker use a synchronous network reader in a background thread, with small bounded byte batches. Multiple workers keep the asynchronous reader for concurrent files. Both paths use the same progress, retry, integrity, ownership and publication rules; network waits do not block the event loop. Download mode still omits discovery and freshness checks.

One file URL

Use the direct shortcut to download one file:

mirror-url --mode download https://example.org/files/a.fits

Without --url, the file URL's parent directory becomes the allowed remote scope. The file keeps its original name and is downloaded into the current working directory. Logs go to mirror-url-download-<destination-hash> under the system temporary directory, outside the default destination. Use --dest-path and --log-path to choose persistent locations:

mirror-url --mode download https://example.org/files/a.fits \
  --dest-path ./downloads --log-path ./logs --verify-content

An explicit --url sets the allowed scope and retains paths relative to that base. Query strings remain in the HTTP request but are excluded from the local filename. Supply exactly one positional URL or one --url-list file. Selecting one source explicitly on the CLI replaces the other source configured in YAML/JSON. The configuration field for the direct source is download_url. The CLI supplies the same shortcut defaults for a download config containing that field; standalone MirrorConfig/validate_config_file() still require explicit base_url, dest_path and log_path.

Progress and completion output

With --print-logs, known-URL downloads record progress approximately every 10 seconds, including when waiting for a response or flushing/hashing a completed transfer. Progress shows elapsed time, received bytes, average speed, completed/failed file counts and up to three active file paths with their current stage. A file with a known length shows received/expected bytes and an estimated remaining transfer time while receiving bytes; unknown lengths have no ETA. Received bytes include discarded retry attempts, while completed files count only successful publications and their receipt entries.

The completion log uses the same SUMMARY layout as mirror mode. It shows requested files, successful missing/existing-file downloads, failures, downloaded bytes, measured download preparation/download/cleanup times, download throughput and total duration. Throughput uses the download phase; total duration also includes preparation and cleanup. The field order and labels match mirror mode. Direct downloads mark discovery and metadata as not performed; their scan, freshness and obsolete-file counters remain zero. Files requested records the supplied URL count. Existing-file downloads are counted from local file presence; they do not imply the remote file changed.

The summary ends with the same cache and parser counters as mirror mode, followed by the separator, completion message, processed scope (ROOT for the direct shortcut) and connection-pool closure count. Download mode reports zero parser/cache work and counts pools only after their backend closes. The startup banner records the backend, destination, SHA-256 setting and applicable whole-file download settings. JSON results retain the receipt location and summed fsync/hash work. These work sums can exceed elapsed time during concurrent downloads and overlap the download phase. SHA-256 receipts describe local bytes and do not establish agreement with a remote digest.

--progress-bar (also --progress) shows per-file byte bars, independently of --print-logs. The normal installation includes the tqdm renderer. With both options, each file has one updating console bar; periodic progress stays in the log file instead of adding console lines while bars are active. If a bar cannot be displayed, the ten-second console messages remain available. Known lengths show percentage and transfer ETA; unknown lengths show received bytes and speed. Bars also identify response waits, disk flushing and hashing. Retries reset the affected file's byte bar. Reaching 100% means the body has arrived; publication and receipt recording must still succeed.

Without --print-logs, periodic log messages are silent and the summary remains in the log file. --quiet suppresses bars and INFO-level console progress and summary output. With --print-logs and without --quiet, the CLI shows the readable summary and completion lines on stderr and leaves stdout empty. Without --print-logs, or with --quiet, it prints exactly one JSON result to stdout for scripts. For a terminal view, use --print-logs; to capture JSON instead:

mirror-url --mode download https://example.org/files/a.fits >result.json

A URL list

# urls.txt
https://example.org/files/a.bin
https://example.org/files/nested/b.bin

python -m mirror_url --mode download --backend httpx \
  --url https://example.org/files/ --url-list urls.txt \
  --dest-path ./downloads --log-path ./logs --concurrency 20 \
  --requests-per-second 0 --request-delay 0 --verify-content

To use aiohttp, install python -m pip install -e ".[aiohttp]" from the checkout in your development environment, then change --backend httpx to --backend aiohttp. Both clients use the same publication and receipt policy. Do not expect bare-client benchmark timings: filesystem checks, durable staging and optional SHA-256 hashing are included here.

All URLs and redirect hops must stay below the selected base URL, on its exact origin. One optional --dir-suffix narrows that scope and appends the suffix to the destination. Nested file paths are retained. The full list is checked for filename collisions before any GET. Direct IP URLs, private DNS answers, unsafe paths, symlinks, hard links and reserved-state collisions are rejected. TLS certificates remain verified, and proxy environment variables are ignored. --trusted-server and the legacy security toggle do not remove these checks.

An existing file is redownloaded only when its identity matches the receipt from this mode, or --overwrite explicitly permits replacement. With --verify-content, its saved SHA-256 must match too. Failed transfers leave the original bytes intact. Connection and payload interruptions have bounded whole-file retries; HTTP failures are reported without status retries. Only a complete HTTP 200 response with a matching declared length is published. When the server omits the length, the backend must report a clean stream end; SHA-256 records local bytes, not proof of a remote reference digest.

Owned temporary work is removed after success, failure or cooperative cancellation. Unknown state and pre-existing partial downloads are preserved. Receipts for published files are saved at the end of the run, including during cooperative cancellation. An abrupt process kill between publication and this save may leave a file without a receipt: the next run refuses to replace it without --overwrite. This mode does not offer resumable partial transfers.

The CLI writes a run log and, when console logs are disabled or --quiet is set, prints a JSON summary with downloaded/failed file counts, bytes, elapsed time, pacing waits, fsync/hash time and any failures. A failed file makes the command exit nonzero. --dry-run validates the list without network access or download state creation. Discovery/listing/filter flags, chunk modes, auto-concurrency, missing-only mode and cleanup policies other than safe are rejected in download mode. For normal mirroring, --backend aiohttp supports whole-file transfers; chunk modes and auto-concurrency require HTTPX, and existing partials are preserved.

Filtering and scope

• --filter accepts one or more case-insensitive filename patterns. A bare extension (.fits) matches by suffix. Plain text such as _fe_ matches a substring; patterns containing regex metacharacters use re.search. Multiple patterns are OR'd — a file matches if any pattern matches, not all of them:

  --filter .fits .txt                 # any .fits or .txt
  --filter '.*\.fits$'                # regex: files ending in .fits
  --filter '2024.*\.fits$' .png       # mixed regex + extension

  For AND (a file must match multiple independent conditions at once — e.g. a channel and a date range), pass a single pattern combining them with regex lookaheads instead of multiple --filter values — the engine already falls back to full re support (including lookaheads) for any pattern containing regex metacharacters:

  # (fe OR pb channel) AND (18-20 June, 03-05h) -- one --filter value
  --filter '(?=.*(?:fe|pb))(?=.*_202606(?:1[89]|20)T0[3-5]\d{4}_)'

  If you combine --filter with --list-files [N]/--list-dirs [N]'s "last N" ranking, see the callout below --list-files [N] about what happens when a filter matches more than one filename prefix in the same run.

• --exclude-dir skips one or more directories, each matched as an exact path relative to --url — not a suffix match at any depth. --exclude-dir lasco excludes only <root>/lasco/, never <root>/setup/lasco/ or any other directory elsewhere in the tree that happens to share that name. --exclude-dir idl/beta excludes only that specific two-level path. Pass several to exclude several: --exclude-dir lasco idl/beta excludes exactly those two root-relative paths. A pattern containing * is the explicit escape hatch for matching at any depth (--exclude-dir '*/lasco' also catches <root>/setup/lasco/) — use both lasco and */lasco if you want the root-level directory and nested directories with that name. Patterns stay relative to --url when --dir-suffix is used.

• --dir-suffix restricts mirroring to one or more subpaths under the base URL and mirrors each in turn.

• --check-files PATH [PATH ...] selects existing files for normal freshness checks when combined with --missing-files. Missing files anywhere in the mirroring scope still download; other existing files skip freshness checks. A selected file downloads again only when the normal ETag/size/time policy reports it stale. Without --missing-files, normal freshness checking still applies to every existing file in scope.

  Selectors are exact, case-sensitive remote paths relative to --url, written with /. They include the suffix, even with multiple --dir-suffix values. They are not local destination paths or patterns. Use literal decoded filenames (quote arguments containing spaces); URL escaping and local filename mapping do not change the selected remote identity. Absolute paths, URLs, traversal and empty path components are rejected.

  mirror-url --url https://sohoftp.nascom.nasa.gov/sdb/ \
    --dest-path /data/sdb --log-path /data/logs/sdb \
    --dir-suffix soho/gen soho/lasco/monthly \
    --missing-files \
    --check-files soho/gen/file1 soho/gen/file2 soho/lasco/monthly/file3

  soho/gen/file1 refers to /sdb/soho/gen/file1. Removing --dir-suffix expands mirroring to the base URL's tree but leaves each selector's meaning unchanged. Selection never expands the configured suffix, depth, file-filter or directory-exclusion scope. Paths absent from the discovered selection do not trigger a separate download or cleanup action. Existing cleanup policies still determine how obsolete local files are handled.

  For a local input list, use the explicit @ prefix:

  --missing-files --check-files @check-files.txt

  # Paths relative to --url
  soho/gen/file1
  soho/gen/file2
  soho/lasco/monthly/file3

  The list is UTF-8 (an optional BOM is accepted), with one path per line. Blank lines and lines beginning with # after whitespace are ignored; surrounding whitespace is trimmed. Nested @ lists, unreadable lists, invalid paths and lists with no paths are rejected. The list filename is relative to the current working directory, or can be an absolute local path. --check-files check-files.txt instead selects a remote file of that name. Direct paths and @ lists can be mixed, or the flag repeated; duplicates are removed. YAML/JSON configuration uses the same inputs:

  missing_files: true
  check_files:
    - soho/gen/file1
    - soho/lasco/monthly/file3
    # Alternatively: - '@check-files.txt'

  An explicit CLI list replaces a configured list. Both HTTPX and aiohttp mirroring honor the selection; known-URL --mode download rejects it. --verify-content remains incompatible with --missing-files.

• --max-depth counts directory levels below the target root (depth 0); files in its immediate child directories are eligible at depth 1. The crawler stays within the configured host/path. Duplicate file URLs are downloaded once, and conflicting sanitized local names fail before downloads.

• --list-dirs [N] discovers and prints the directory tree under --url/ --dir-suffix, then exits — it reuses the same directory-discovery walk as a real sync, so it respects --exclude-dir and --max-depth, but never scans files, checks freshness, or downloads/deletes anything. --filter doesn't apply, since it only matches filenames, not directories. Handy for seeing what's on a remote server before choosing a --dir-suffix. Unlike download modes, a CLI-only run does not require --dest-path or --log-path; a config-file run still requires both paths.

  Without --config, --list-dirs defaults --max-depth to 1: the current folder's immediate children. With --config, the file's max_depth applies (or 50 when omitted). Pass --max-depth explicitly to override either:

  # Immediate children only (the default)
  mirror-url --url https://archive.example.org/mission/ --list-dirs
  
  # Walk 3 levels deep instead
  mirror-url --url https://archive.example.org/mission/ --list-dirs \
    --max-depth 3

  Each directory is printed to stdout as a bare relative path, one per line (. for the root), independent of --print-logs/--quiet and the usual banner/summary logging — pipe or capture it directly:

  mirror-url --url https://archive.example.org/mission/ --list-dirs \
    | xargs -I{} echo "found: {}"

  The listing is always followed by a # Directories N/total comment line on stdout — including an unrestricted (no-N) run, where N == total — mirroring --list-files' # Files N/total convention. Drop it with grep -v '^#' for a pure one-directory-per-line stream.

  With no N, every directory (within --max-depth) is printed in discovery order, as above. With N, only the last N directories are printed, sorted lexicographically by relative path, with the root (.) excluded from that ranking (it isn't a real --dir-suffix candidate, and would otherwise dilute the "last N real directories" a caller typically wants). Unlike --list-files [N], which ranks per directory (files are naturally grouped by the directory that contains them), --list-dirs [N] ranks across the entire discovered tree for this suffix, since directories have no equivalent natural grouping:

  mirror-url --url https://archive.example.org/mission/L3_png/v03/ \
    --list-dirs 3

  This directly replaces the common pattern of piping --list-dirs through grep -v '^\.$' | sort | tail -n N to pick the most recent N directories to mirror next.

  If you mirror more than one --dir-suffix in the same run, each line gets a tab-separated suffix column prepended instead of a bare path (e.g. L1/v2\tsome/subdir), so you can tell which subtree it came from while keeping the path itself easy to cut -f2/awk -F'\t' out.

• --list-files [N] discovers and prints the files under --url/ --dir-suffix, then exits — it reuses the same directory-discovery walk and per-directory file scan as a real sync, so it respects --exclude-dir, --max-depth, and --filter (unlike --list-dirs, --filter does apply here, since it matches filenames). It never compares freshness or downloads/deletes anything, and — like --list-dirs — a CLI-only run does not require --dest-path or --log-path. Config-file runs still require both paths.

  With no N, every matching file is printed. With N, only the last N files per directory are printed (not N total across the whole run — a directory with 200 files and one with 3 each contribute up to N).

  Each file is printed to stdout as its full path relative to --url/--dir-suffix (e.g. v03/orbit_0042/file_20260722_003.fits), one per line, independent of --print-logs/--quiet. Every directory's block of files is followed by a # Files N/total comment line — always, even on an unrestricted (no-N) run — so scripts can rely on the marker being present unconditionally rather than only when truncated. Comment lines start with # and can be dropped with grep -v '^#' for a pure one-line-per-file stream:

  mirror-url --url https://archive.example.org/mission/L3_png/v03/ \
    --list-files 5 | grep -v '^#'

  When more than one --dir-suffix is mirrored in the same run, each file line gets the same tab-separated suffix column as --list-dirs. Comment lines are never suffix-qualified, since they aren't file paths.

  ⚠️ "Last N" is a filename/path sort, not a timestamp sort. With N, entries are ranked by sorting their relative paths lexicographically (plain string/alphabetical order) and keeping the last N — per directory for --list-files [N], across the whole tree for --list-dirs [N] — it is not based on any server-reported modification time.

  This avoids extra per-file metadata requests. It reflects chronological order only when filenames contain sortable dates or sequence numbers. For arbitrary names, it means alphabetically last, regardless of file age.

  A sharper version of the same tradeoff bites when --filter matches more than one filename prefix/channel in the same run — e.g. --filter fe pb for two instrument channels named ..._fe_l3_... and ..._pb_l3_.... Every fe-file sorts before every pb-file (f < p), regardless of timestamp — so once a directory has at least N pb-files, "last N" is N pb-files, full stop, no matter how recent the newest fe-file is. The fe channel doesn't just rank lower — with more than N pb-files present, it's invisible. Workarounds:

  • Query each channel separately — --filter fe and --filter pb as two separate runs — sidesteps the collision entirely, since each run's "last N" only ever ranks within one prefix.
  • Or request enough files to be sure, then sort/filter by timestamp yourself instead of relying on the filename-prefix order:

    mirror-url --url https://archive.example.org/mission/L3_png/v03/ \
      --list-files 200 --filter fe pb --quiet \
      | grep -v '^#' | sort -t_ -k4 | tail -3

    (sort -t_ -k4 sorts from the 4th underscore-delimited field onward — i.e. from the embedded timestamp, not the channel prefix sitting before it. This assumes that filename layout; adapt the field number for your archive.)

Filename collisions and download behavior

A filename collision fails the entire affected suffix before file downloads. MirrorURL does not choose one of the conflicting files, rename either file, or download the other files in that suffix while silently skipping the pair. Existing mirrored file contents are preserved, and obsolete-file cleanup is not reached. Directory listings, connection/metadata requests and log/cache bookkeeping can still occur before the failure.

For example, NASA SolarSoft has these two different files under stereo:

secchi/idl/daily/Deep_Field_v3.pro
secchi/idl/daily/deep_field_v3.pro

They have different contents. Common case-insensitive macOS volumes cannot store both names as separate files. A case-sensitive Linux filesystem or case-sensitive APFS volume can store both; an existing Linux mirror can therefore contain both originals.

Filesystem-aware behavior: MirrorURL checks the destination when selected names differ only by case. On a confirmed case-sensitive destination, it downloads both files with their original capitalization and keeps their content receipts separate. On a case-insensitive destination, the pair fails preflight before any file payload in that suffix is downloaded. There is no option to choose whichever file arrives first or overwrite one with the other.

The check also covers directory names such as DAILY/one.pro and daily/two.pro. If a requested filename already resolves to a differently capitalized local entry, preflight fails and preserves that entry, even when only one remote file is selected or --missing-files is used. Otherwise a missing-only run could mistake different local data for the requested file.

For an ambiguous pair, MirrorURL creates an exclusively owned, randomly named empty probe in its actual parent directory, or the nearest existing ancestor when the parent does not exist yet. It checks the differently capitalized probe name and removes only its own unchanged regular file. This check also runs during a dry run. If the directory cannot be inspected or probed safely, the suffix fails instead of guessing its filesystem behavior. Ordinary unique names do not require a case probe.

Preserving the complete selected tree: use a destination that can store both original filenames, and the updated filesystem-aware MirrorURL implementation. On macOS, an APFS (Case-sensitive) disk image or volume provides the required storage behavior. Apple's disk-image instructions describe creating a blank image with that format; a sparse image grows as files are added. This avoids reformatting the existing startup volume. Choose a new destination inside the mounted image for the complete mirror; an ordinary case-insensitive destination cannot hold both originals. Size the image for the selected payload and temporary transfer space. With the updated code, retain the same URL and suffix and include daily instead of excluding it. Moving an existing mirror or changing the destination of a running job is a separate operation; do not change its filesystem while it is writing.

Excluding daily does not solve complete mirroring: it discards an entire remote subtree from the selected scope. Automatically renaming one file would also change the archive's original names and may break references between files. Neither behavior is a substitute for preserving both originals.

If a deliberately reduced scope is acceptable, directory exclusions are relative to --url: with the SolarSoft root URL, excluding stereo/secchi/idl/daily omits every file and subdirectory under daily, not just the two conflicting names. Files already present under an excluded subtree are preserved by cleanup. Use this only when that omission is intended.

Other mapping problems also fail preflight: Unicode-normalized name aliases, names that would be truncated or rewritten by sanitization, unsafe local paths, and collisions with reserved .mirror-url-state storage. A repeated identical remote URL is deduplicated; distinct URLs that collide are not deduplicated by content, even if their bytes happen to match. Control characters and Windows reserved names are rejected rather than rewritten. If the configured filename length limit is too small, increase --max-filename-len or narrow the selection. Case-insensitive --filter matching cannot select just one of the two example filenames by capitalization.

For a multi-suffix CLI invocation, failure of one suffix does not roll back earlier successful suffixes. Other suffixes may still run, and the overall CLI exits 1 when a suffix fails. The Python sync() call returns False; inspect the log for Distinct remote URLs map to the same local filename, Remote filename would be changed by local path sanitization, or Remote file has an unsafe or reserved local path.

This describes normal synchronization and its dry run. --list-files is discovery-only and can list both remote URLs; it does not prove that the selected files can be mirrored. --quick also skips remote-file validation and transfers.

Caching and incremental sync

MirrorURL stores file identity metadata and directory signatures in a JSON cache under --log-path. Keep this path stable between runs. The file name contains the directory suffix and a hash of the base URL, so different remote roots do not share one cache accidentally.

Normal runs still discover the remote tree and check existing files. By default, a cached file ETag is trusted only when the recorded local size, modification time, and filesystem change/creation timestamp match the current file. Otherwise the file is checked again. Same-size local edits that leave these timestamps unchanged can be missed, including rapid writes on filesystems with coarse timestamp resolution. This check does not hash local file contents. Directory-listing ETags and signatures do not establish that child file contents are unchanged. Without usable file ETags, checks fall back to size and the server's Last-Modified header when available; changes that preserve those values can be missed. No remote cryptographic digest comparison is performed.

Use --verify-content (or verify_content: true in YAML) to detect local changes even when file sizes and timestamps still match. This mode saves a SHA-256 receipt of each completed staging file before atomic publication, then hashes existing local files before trusting freshness metadata. A missing, expired, malformed or mismatched receipt requires a new download. Enabling it for an existing mirror therefore redownloads files whose cache has no receipt. Keep --log-path stable so those receipts persist between runs.

Hashing reads every verified file in 1 MiB chunks, adding disk IO proportional to the selected data size on every full run. Async metadata checks perform hashing on worker threads. --hash-algorithm does not change these SHA-256 receipts, and the compatibility flag --no-content-hash does not disable them; use --no-verify-content for that. Network request timeouts still apply; local hashing is allowed to finish without the metadata-only async check/batch deadlines. Slow storage can lengthen a run.

Content verification proves agreement with the saved local receipt. Remote freshness still depends on the server's HTTP validators, size and modification time; a remote change that preserves those values can be missed. This mode does not compare against a server-provided cryptographic checksum. Cooperative destination locks prevent overlap between participating MirrorURL processes, but do not isolate the filesystem from other writers. Avoid external edits during a run; see Concurrent runs and destination ownership.

Parsed HTML listings also have bounded in-memory caches. They are not restored from disk on a new process launch. --html-cache-max-age controls their lifetime within the process; it does not mean a fresh CLI invocation will skip fetching the remote listing.

• --cache-max-age DAYS: discard expired JSON metadata (default 7 days).
• --refresh-cache: ignore saved metadata and cached listing results for this run.
• --no-cache: bypass those caches; existing files are still checked. With --verify-content, unavailable receipts require downloading existing files.
• --no-etag: use size/time rather than file ETags for freshness checks.
• --missing-files: download absent files and skip existing files, except paths selected by --check-files. Unselected in-place changes will be missed. Use occasional normal runs when those changes matter. It cannot be combined with --verify-content.
• --quick: refresh an existing JSON cache's expiry timestamp, without scanning or downloading. It does not verify that local or remote files are current and does not create a missing cache. Connection setup can still contact the server.

The cache can be written after a complete scan and updated again after the normal download/cleanup path. It is separate from resumable file state in .mirror-url-state/.

Process-interruption tests cover all three download modes. Killing a process before publication preserves the previous complete destination; killing it after publication leaves the complete replacement. A restart checks receipts again, resumes validated sequential partials where possible, and can redownload files published before their receipt was persisted. Interrupted cache writes leave the previous complete JSON authoritative. MOVE cleanup can be restarted with obsolete bytes preserved in their source or archive. These tests cover process termination, not power loss or failure of the storage device.

--use-disk-backed-sets, --memory-cache-size, and --download-queue-size do not bound the current workflow's remote file list: discovery collects that list in memory. Scope a large archive with --dir-suffix, --exclude-dir, or --max-depth when you need smaller runs.

Cleaning up obsolete files

By default MirrorURL preserves obsolete local files (--cleanup safe). It still replaces files that need updating. Choose preview, move, or delete to handle files absent from the current remote selection. preview reports obsolete-file actions but still allows normal downloads; add --dry-run for an observation-only run. Keep --log-path outside the destination tree so cleanup does not treat your logs and cache as obsolete mirrored files.

Cleanup only acts within the current scan selection. Files excluded by filters, directory exclusions, depth limits, or skipped symlink subtrees are preserved, as are local symlinks and .mirror-url-state/. A complete empty scan can clean the selected local files; an incomplete scan skips cleanup and fails the run. Directory inspection, removal and other cleanup operation failures also fail the run. If the MOVE archive cannot be created, cleanup stops; a failed move leaves the source in place and never falls back to deletion. MOVE also refuses symlinked archive paths and existing timestamp collision names. It preserves the source and previous archives and reports a failed cleanup operation; inspect the archive before retrying.

# Observe cleanup without downloads or changes to mirrored files
mirror-url --config mirror.yaml --cleanup preview --dry-run

# Move obsolete files into sibling <dest>_obsolete/ instead of deleting
mirror-url --config mirror.yaml --cleanup move

# Actually delete, with a confirmation prompt
mirror-url --config mirror.yaml --cleanup delete --confirm-delete

Combine any of these with --dry-run to simulate the entire run (scan + download + cleanup) without downloading or changing mirrored files. The run still requests directory listings and metadata; log/cache bookkeeping may create directories outside the mirrored data tree.

Security

MirrorURL ships with security protections enabled by default:

• SSRF / private-network protection. The HTTP transport resolves and validates target IPs and refuses to connect to loopback or private addresses (127.0.0.1, localhost, RFC-1918 ranges, link-local, etc.), blocks direct-IP URLs, dangerous ports, and IDN/homograph and URL-smuggling tricks.
• URL-scope enforcement keeps the crawler within the configured base host/path and blocks path-traversal (including encoded/double-encoded forms).
• Filesystem safety — filename sanitization, path-traversal rejection, Windows reserved-name handling, and symlink-loop/bomb defenses.

The secure transport rejects direct-IP, private, and loopback targets, including localhost. --no-security-validation disables the extra URL validation layer and per-IP request pacing; it does not disable transport IP validation. There is no production option for mirroring private or local servers.

Discovery, local filename mapping and cleanup use the same validated, once-decoded URL paths. Equivalent percent-encoded roots are accepted; literal percent escapes and semicolons in filenames are preserved. Query parameters do not turn directory links into files. Invalid URL decoding or unsafe expected paths fail the run and suppress obsolete-file cleanup. DNS checks reject a response containing any non-public address, including a mix of public and link-local answers.

Local symlinks in a selected destination root are rejected before managers start; leaf and descendant symlinks block file mapping. The fixed macOS /var, /tmp and /etc system aliases are accepted. Local path checks do not protect against another process changing the destination tree during a run.

Symlink handling is off by default; see Symlink handling below for how it works and how to use it. Remote directory-duplicate handling does not permit writes through local symlinks.

Symlink handling

An HTTP directory listing usually does not identify server-side symlinks. --handle-symlinks enables a heuristic: compare the basenames of each visited directory's immediate files and subdirectories. Two non-empty directories with the same entries are reported as possible duplicates, with the first visited directory treated as the target. For example, two paths exposing the same archive subtree may be detected this way.

The comparison uses listings already fetched by the scan. Directory-listing Last-Modified and ETag headers, when available, add confidence notes to the log; they do not decide whether the directory is flagged and do not verify individual files. Descendants of an already flagged subtree are not repeatedly reported as separate duplicates.

This heuristic can flag distinct directories that merely use the same entry names, cannot detect a target outside the visited tree, and cannot identify individual file symlinks. Empty directories are excluded from matching. Review the reported paths before choosing exclusions or enabling automatic skipping.

Start with a survey:

mirror-url --config mirror.yaml --handle-symlinks --symlink-mode detect --print-logs

Review the Symlink detected log lines. To avoid a known duplicate permanently, use --exclude-dir with its exact path relative to --url, or an explicit * glob for nested matches. Plain exclusions do not match arbitrary path suffixes. --max-symlink-depth, --max-symlinks-per-dir, and --symlink-bomb-threshold limit tracker behavior; they do not make the heuristic a definitive detector.

Monitoring and metrics

• Run summary. A normal mirror sync logs one SUMMARY block at INFO level, including its result and any download, scan or cleanup failure. File outcomes distinguish existing files checked and current, freshness checks skipped, missing files downloaded, changed files downloaded, and downloads made because freshness could not be established. Other skips include HTTP resources that became unavailable; they are not described as current. Download counts record successful publication, rather than queued or failed transfers.

  With --check-files, the summary reports how many configured paths were discovered within scope and how many selected existing files were checked. A selected missing file downloads normally. Unmatched selectors can be absent or outside the configured discovery scope; they do not trigger extra discovery. Selected current files and files needing replacement have a short decision message. A metadata failure is reported as uncertain freshness, not as proof of a remote change. DEBUG logs include returned freshness headers.

  Discovery, metadata, download preparation, downloads and cleanup have separate elapsed times. Download throughput divides newly transferred bytes by the download phase's wall-clock duration, including concurrent transfers, retries, pacing and publication. It is n/a when nothing was published. Total duration covers sync() after manager setup; initial construction and final shutdown are outside that interval. The summary reports the actual Discovery mode and worker count, Directories examined, Discovery elapsed time and Discovery throughput together. Throughput divides examined directories by the discovery phase's elapsed time, including cache loading and file-list collection. It is n/a when discovery was not started or no positive duration was measured. The progress banner times the directory walk alone. Listing fetch/parse work (sum) includes network wait and sums individual operations; concurrent operations overlap, so this value can exceed elapsed time. It is neither CPU parsing time nor the throughput denominator.

  Discovery mode: parallel (4 workers)
  Directories examined: 42
  Discovery elapsed time: 6.276s
  Discovery throughput: 6.7 dirs/s
  Listing fetch/parse work (sum): 11.947s

  Cache reporting separates directory signatures loaded, in-memory parsed listing reuse, HTML-listing hits/misses and bypass reasons. Directory listing GET calls and directory signature HEAD calls have separate counts and summed request times, including failed calls. A successfully parsed listing fetched during this discovery supplies its ETag or Last-Modified directly. If neither is supplied, MirrorURL hashes the fetched listing's complete decoded HTML with --hash-algorithm. Both paths avoid a separate signature HEAD. Listings served entirely from an earlier discovery's cache still use HEAD. The summary counts these fresh GET signature reuses separately, including content signatures. Directory content signatures do not verify child file contents. File freshness checks follow the configured policy regardless of directory signatures. Counts refer to logical calls; transport redirects and retries can make additional HTTP requests. --stats is accepted for backward compatibility and has no effect. Quick mode, connection failures and discovery-only surveys have shorter summaries.

• --metrics-json FILE writes the full metrics summary to JSON (skipped in --dry-run). Its metrics.run object contains the per-sync outcomes, result, phase timings, discovery_mode, discovery_workers, directories_examined, discovery_throughput and download_throughput (throughputs are null when unavailable). Examined directories include depth-limit entries and failed listing attempts; throughput does not establish a successful scan. Existing top-level metric keys remain available; download_speed is the older average over the collector's lifetime, not the new download-phase throughput.

• --progress-bar / --progress shows live tqdm bars (included in the normal installation). Both mirror and known-URL downloads show one bar per active file, updated as bytes arrive, with the current transfer stage. Mirror metadata checks use a separate files-examined bar that clears before transfers start. Completed bars close before the summary. Resumed bytes advance the file position without counting as new traffic; parallel ranges share one file bar. Directory discovery reports examined-directory counts and elapsed time while fetching listings; its total tree size is unknown, so it does not show a percentage or ETA. The final discovery rate includes listing request time.

• Health/metrics HTTP endpoints. Setting --metrics-json FILE also starts a local HTTP server for a non-dry-run instance, until it is cleaned up. --health-check-port alone does not enable it. The server binds to localhost (default port 8080); both endpoints are rate-limited and serve:

  • GET /health → JSON health status. Returns HTTP 200 when connected with fewer than 10 failed files, otherwise HTTP 503 with degraded status.
  • GET /metrics → JSON counters (files downloaded/failed/skipped, bytes, elapsed).

  curl http://localhost:8080/health
  curl http://localhost:8080/metrics

• Logging. Each run writes a timestamped log under --log-path. Use --verbose/--debug for more detail, --quiet for less, and --print-logs to also echo to the console.

Using MirrorURL from Python

MirrorURL is a library as well as a CLI. The public API:

from pathlib import Path
from mirror_url import MirrorURL, MirrorConfig

config = MirrorConfig(
    base_url="https://archive.example.org/mission/",
    dest_path=Path("/data/mission"),
    log_path=Path("/var/log/mirror"),
    file_filters=[".fits"],
    parallel_downloads=True,
    cache_max_age=7,
)

with MirrorURL(config) as mirror:
    ok = mirror.sync()  # returns True on success, False on failure

print("sync succeeded" if ok else "sync failed")

Use the context manager (with) to clean up connection pools, background workers, async components, and any health server. sync() is blocking; library construction leaves signal handlers unchanged. The CLI separately opts in to SIGINT/SIGTERM handling. Freshness checks, resume rules, and cleanup policies are the same for CLI and library runs.

Load configuration from a YAML/JSON file (from_yaml accepts JSON as YAML-compatible syntax):

from pathlib import Path
from mirror_url import MirrorConfig, MirrorURL

config = MirrorConfig.from_yaml(Path("mirror.yaml"))
with MirrorURL(config) as mirror:
    mirror.sync()

Handle configuration errors:

from pathlib import Path
from pydantic import ValidationError
from mirror_url import MirrorConfig
from mirror_url.exceptions import ConfigError

try:
    cfg = MirrorConfig(base_url="ftp://nope", dest_path=Path("d"), log_path=Path("l"))
except (ConfigError, ValidationError) as e:
    print("bad config:", e)

Useful exported names: MirrorURL, MirrorConfig, load_config_from_args, main, and the exception types (MirrorError, ConfigError, MirrorConnectionError, SecurityError, DownloadError, DestinationLockError, PathTraversalError, URLScopeError).

For a configuration dictionary, use MirrorConfig.model_validate(data) to validate it and obtain a model. To obtain informational warnings about an existing configuration, use MirrorConfig.validation_warnings(config). Code that previously used MirrorConfig.validate(config) for warnings must switch to validation_warnings; validate now follows Pydantic's model validation API, where model_validate is preferred.

Exit codes

The CLI currently exits 0 after graceful SIGINT/SIGTERM cleanup, even if the sync was interrupted. For scheduled jobs, review the completion summary when an interruption occurred rather than treating that exit code as proof that all files were mirrored. The Python API returns a boolean from sync() instead of setting a process exit code.

mirror-url --config mirror.yaml && echo "OK" || echo "FAILED ($?)"

Troubleshooting

"Hostname resolves to private IP" / connection refused to localhost. The secure transport blocks private/loopback targets even with --no-security-validation. Use a public hostname resolving to a public address. The loopback bypass used by integration tests is confined to those tests.

Server returns 403/429 or downloads are slow/failing. The remote may be throttling you. Omit --trusted-server (or set trusted_server: false in your config), increase --request-delay, reduce --workers/--max-concurrent-downloads, add --no-async-metadata, or switch to --sequential-downloads.

Nothing is downloaded / "0 directories". Confirm --url actually serves an HTML directory listing (not a single file or a JS-rendered page). Check your --filter isn't excluding everything, and try --debug to see the parsed links and scope decisions.

"Distinct remote URLs map to the same local filename" / case-only names. The selected suffix stopped before file downloads. Neither conflicting file is chosen; other files in that suffix are also not transferred. See Filename collisions and download behavior for the NASA example, the filesystem-aware behavior, and how a case-sensitive destination preserves both original files. A probe/inspection failure also requires resolving destination access before retrying.

Parallel mode isn't kicking in. Chunking needs a known file size of at least --min-chunk-size, byte Range support, and a strong ETag. Otherwise MirrorURL uses whole-file transfers. Check the log with --print-logs --debug for selection and fallback messages.

Progress bar / memory stats missing. Progress bars are included in the normal installation. Install monitoring with pip install "mirror-url[monitor]" (psutil).

Re-runs re-download everything. Make sure --log-path is stable between runs (that's where the cache lives) and you're not passing --no-cache / --refresh-cache.

Uninstalling

pip uninstall mirror-url
# or, if installed with pipx:
pipx uninstall mirror-url

Generated logs, the JSON cache under --log-path, mirrored data and partial state under --dest-path, and per-user domain-health metadata are left in place. Domain-health metadata lives under $XDG_CACHE_HOME/mirror-url/ (or ~/.cache/mirror-url/) on POSIX and under the local application-data mirror-url/ directory on Windows. Remove these manually if desired.