MirrorURL — Developer Guide

This guide is for contributors working on the MirrorURL codebase itself. It is a self-contained architecture deep-dive: how the package is structured, why it is layered the way it is, how the MirrorURL orchestrator is composed, how data flows through a run, and how to extend the system without breaking its invariants.

If you only want to use MirrorURL (install, CLI, config, Python API), read USER_GUIDE.md instead. For day-to-day contribution mechanics (branching, PR etiquette) see CONTRIBUTING.md; this guide repeats the essentials so you can work from it alone.

Table of contents

Background: the monolith and the refactor

MirrorURL began as a single mirror_url.py of ~15,000 lines containing ~70 classes and ~25 module-level functions. It was split into the modular src/mirror_url/ package (46 Python files, organized by responsibilities) by a behavior-preserving migration: code was relocated verbatim and class/function method sets were verified identical to the original via AST comparison. Logic changes were kept out of the migration and made only in separate, reviewable commits.

Two consequences shape how you should work in this codebase:

  1. Verbatim heritage. Much of the code is a faithful port of working, audited, sometimes idiosyncratic logic (including hard-won bug fixes recorded in the changelog). When touching migrated code, prefer surgical changes over "cleanups" — the original behavior is the contract.
  2. Python 3.10 baseline. Python 3.10 is the minimum supported runtime. Existing code retains classic typing (Dict, Optional, List) and from __future__ import annotations. Lint rules that would modernize syntax (pyupgrade, most of SIM) remain disabled to keep unrelated rewrites separate — see Coding conventions.

The legacy mirror_url.py was retained as a frozen reference, excluded from lint and packaging, until the package's test suite passed with real runtime dependencies installed — then deleted before the v3.1.20 release.

Design principles

The package is built on four rules. Internalize them before making structural changes — most review feedback traces back to one of these.

  1. Behavior-preserving by default. Relocation and refactoring must not change runtime behavior. Functional fixes are separate commits with their own tests and changelog entries.
  2. One concern per module. Each module is independently readable and testable. If a change makes a module "about two things," that is a signal to split it.
  3. Acyclic dependencies. Keep imports acyclic. Same-layer imports already exist (for example, connection.py imports metrics.py and concurrency.py), so the diagram below is a responsibility map rather than an enforced strictly downward layering invariant.
  4. Small, verifiable steps. Keep the fast test lane green after every change.

Repository layout

mirror-url/
├── src/mirror_url/          # the package (48 Python files including private helpers)
│   ├── __init__.py          # public API re-exports
│   ├── __main__.py          # `python -m mirror_url`
│   ├── _version.py          # __version__, __author__  (one of two version sources)
│   ├── cli.py               # argument parsing, shared logging, main()
│   ├── core.py              # MirrorURL — composed from the _core mixins
│   ├── _core/               # the MirrorURL class, split into mixins (private)
│   │   ├── _base.py         # __init__, shared state, lifecycle, logging
│   │   ├── urls.py          # URL scheme/scope validation, path helpers
│   │   ├── scan.py          # remote discovery (BFS), filtering
│   │   ├── compare.py       # is-up-to-date checks (sync + async)
│   │   ├── downloads.py     # per-file download orchestration
│   │   ├── cleanup.py       # obsolete-file cleanup policies
│   │   └── report.py        # sync() entry, summaries, benchmark
│   ├── config.py            # ConfigSchema, MirrorConfig, load_config_from_args
│   ├── … (subsystem modules, see the module reference)
├── tests/                   # pytest suite (smoke, utils, security, subsystems, integration)
├── docs/                    # USER_GUIDE, DEVELOPER_GUIDE (this file), HTML renders
├── pyproject.toml           # packaging, deps, tool config  (the other version source)
├── REFACTORING_PLAN.md      # the migration plan + module/line map
├── CHANGELOG.md             # Keep a Changelog format
└── .github/workflows/       # ci.yml (lint+test matrix), release.yml (tag → build → GitHub Release, optional PyPI)

The dependency-layer architecture

The historical layers summarize responsibilities; runtime imports can cross those groupings. The important invariant is an acyclic runtime import graph.

Foundations: _version · compat · constants · exceptions · enums · utils
Data and primitives: models · async_primitives · primitives · parsing · security · destination_lock
Managers: transport · storage · circuit_breaker · rate_limiter · queue · cache
Runtime support: metrics · progress · monitoring · connection · async_connection · concurrency
Engines: download · download_integrity · scanner · health · domain_health · config · tuner
Composition: core · _core/{_base,urls,scan,compare,downloads,cleanup,report}
Static contract: _core/_typing (TYPE_CHECKING only)
Entry points: cli · __main__

Why it matters for you:

Compile and import the package after an import change to catch syntax errors and common missing/circular-import failures. These checks do not prove that every runtime path is acyclic: inspect function-local and dynamically invoked imports separately. Both connection implementations share utils.url_within_scope for their scheme, authority and decoded-path boundary; pool warm-up checks each redirect before contacting its target.

Module reference (by layer)

Layer 0 — foundations (no intra-package imports).

Layer 1 — pure helpers.

Layer 2 — stateless/low-state building blocks.

Layer 3 — transport & resilience primitives.

Layer 4 — managers & observability.

Layer 5 — feature engines & config.

Layer 6 — orchestration.

Layer 7 — entry point.

The MirrorURL orchestrator and its mixins

MirrorURL was ~3,600 lines as a single class. It is now composed from focused mixins under the private _core/ subpackage. core.py is a thin composer:

class MirrorURL(
    UrlMixin,  # _core/urls.py
    ScanMixin,  # _core/scan.py
    CompareMixin,  # _core/compare.py
    DownloadMixin,  # _core/downloads.py
    CleanupMixin,  # _core/cleanup.py
    ReportMixin,  # _core/report.py
    _MirrorBase,  # _core/_base.py — __init__ + shared state, listed LAST
): ...

Why this is safe and unambiguous: every method is defined in exactly one mixin, so the MRO never has to disambiguate. _MirrorBase is listed last so that it sits at the base of the MRO and owns __init__ plus all shared instance state; the feature mixins are "above" it and call into the state it sets up. Behavior is identical to the pre-split class — from mirror_url.core import MirrorURL is unchanged for callers.

Responsibilities and key methods per mixin:

Mixin (_core/…) Responsibility Representative methods
_MirrorBase (_base.py) Construction, shared state, lifecycle, logging, connection bring-up, the on-disk caches, disk-space checks __init__, __enter__/__exit__, cleanup, setup_logging, test_connection, _warm_up_connections, check_disk_space, install_signal_handlers
UrlMixin (urls.py) URL scheme/scope validation, path extraction _validate_url_scheme, _is_url_within_scope, _is_within_target_scope, _is_dir_excluded, _get_target_base_url, _parse_url_cached, _get_filename_fast
ScanMixin (scan.py) Remote discovery, filtering, symlink tracking get_remote_files, _discover_directories_bfs, matches_filter, get_directory_signature, is_symlink/record_symlink, _get_local_path_from_url
CompareMixin (compare.py) "Is the local copy up to date?" — local identity plus remote size/timestamp/ETag, sync and async file_exists_and_up_to_date, _check_files_sync, _check_files_async, _comparison_metadata, _response_is_current, get_remote_timestamp, get_directory_size
DownloadMixin (downloads.py) Per-file download orchestration (delegates to the download.py engines) download_file_with_resume, _download_file_single
CleanupMixin (cleanup.py) Removing/moving local files no longer present remotely clean_obsolete, _scan_local_tree, _cleanup_path_selected
ReportMixin (report.py) The top-level sync() driver, summaries, benchmarking sync, _print_early_exit_summary, benchmark

Working rule: when you add a method to MirrorURL, put it in the mixin whose responsibility it matches, and keep shared attributes initialized in _MirrorBase.__init__. Don't add a second __init__ to a feature mixin.

_core/_typing.py declares the shared state and method signatures as the MirrorHost protocol. Each mixin and _MirrorBase inherits this contract only while type checking; its runtime base remains object. Update the contract alongside changes to shared fields or cross-mixin signatures. Optional managers and paths must be narrowed before use. The contract has no runtime methods, state, or imports, and the composed class keeps the same MRO.

Runtime data flow: anatomy of a sync()

A full mirror run is driven by ReportMixin.sync(). The high-level path:

  1. Construction (_MirrorBase.__init__). Parse the base URL, compute the destination/target paths, build the cache file path (mirror_url_<suffix>_<hash>.json, where the hash is the first 16 hex chars of sha256(base_url) — this disambiguates different base URLs that share a directory suffix), set up logging, and instantiate the subsystem managers (connection, async, concurrency, metrics, circuit breakers, rate limiter, caches).
  2. Connect (test_connection, _warm_up_connections). Validate reachability and resolve the target scope. Until the target is resolved, scope checks fall back to the base URL (so the scanner doesn't drop every subdirectory).
  3. Scan (ScanMixin.get_remote_files). Breadth-first discovery of the remote tree, honoring max_depth, exclude_dirs, scope enforcement, and a visited set (cycle-safe). Directory listings are parsed by parsing.py. Results feed in-memory parsed-listing caches. The resulting remote file list is deduplicated and preflighted for unsafe, reserved, or colliding local paths. ReportMixin.sync() calls _validate_remote_paths() before per-file comparison and downloads. filename_mapping.FilenameMap checks every file and directory prefix. Unicode-normalized aliases and exact file/directory conflicts remain rejected. Case-folded candidate pairs trigger an actual filesystem probe in their parent (or nearest existing ancestor). A confirmed case-sensitive parent permits distinct original spellings; an insensitive parent fails. Parent caches use spelling-preserving tuples because Windows Path equality folds case even for case-sensitive directories. Existing directory entries are checked against requested names so an insensitive lookup cannot overwrite or skip a differently capitalized local file. The random exclusive probe is removed only after confirming its regular file type, single link and unchanged device, inode, size and timestamps; unexpected entries are preserved and inspection/probe failures fail closed. A preflight exception is caught by sync(), which logs a fatal error and returns False: no file in the affected suffix is downloaded, and obsolete-file cleanup is not reached. Discovery-only listing modes bypass this check. Lossy-name, reserved-state and unsafe-path rejection remain in place. Probe checks also run during dry-run preflight. The User Guide documents the NASA pair and the storage and preflight requirements for preserving both original files. Directory exclusion reduces the mirrored scope and does not resolve complete mirroring.
  4. Compare (CompareMixin). For each remote file, decide whether the local copy is current using a shared sync/async size, timestamp, and ETag policy. Cached ETags require matching local size/mtime/ctime metadata, and directory signatures never validate child file contents. When async_metadata is enabled and there are more than 80 remote files, HEAD checks can use the async manager; smaller batches, dry runs, and fallback paths use sync checks. With missing_files, absent files remain download candidates. Existing files skip checking unless MirrorConfig.check_file_selected() matches their once-decoded path relative to base_url. Sync/async comparisons, adaptive profiling and async warm-up use the same selection policy. Suffixes and local filename sanitization never redefine the selection root; discovery, filtering and cleanup retain their existing scope.
  5. Download (DownloadMixin → download.py). Missing/changed files are fetched. ParallelDownloadManager.auto_select_method (or an explicit DownloadMethod) picks sequential vs. streaming-parallel vs. traditional-parallel chunking; download_integrity.py validates strong ETags, exact ranges, and persistent whole-file resume metadata. Streaming chunks write to a staging path and publish atomically after verification. Worker pools and the coordinator impose separate limits. Per-domain CircuitBreakerManager handles request failures; the bandwidth limiter throttles bytes read, and chunk work also uses ChunkCircuitBreaker.
  6. Cleanup (CleanupMixin.clean_obsolete). Optionally preview/move/delete local files no longer present remotely, per CleanupPolicy. Preserve paths omitted by the scan selection and never traverse local symlinks. MOVE failures leave source paths intact.
  7. Report (ReportMixin). Save cache metadata again after download and cleanup work when the scan is complete, render the normal summary, and optionally export metrics JSON. A failed download, incomplete scan, or recorded cleanup-operation failure makes sync() return False.

Run reporting

MetricsCollector.run_report starts a fresh reporting interval for each sync(). It records file decisions by remote URL, replacing intermediate decisions after async-to-sync fallback instead of counting the same file twice. CompareMixin._freshness_result() supplies the existing policy result with a reason and a reporting category: current, changed, or uncertain. Missing and unchecked local files are recorded before making any remote freshness request. Reporting must not change the freshness policy or add requests.

Successful publication hooks in sequential, chunked and aiohttp transfers record the published URL. A truthy transfer return can also mean an HTTP skip, so it is not sufficient evidence for a downloaded-file count. Missing, changed and uncertain-freshness download counts include only published files. Failed or queued transfers never receive publication credit. An unavailable/invalid HEAD result is uncertain, even if a later GET succeeds.

RunReport.phase() and begin_phase()/end_phase() measure coordinator wall-clock intervals with perf_counter(), including exception unwinding. Download preparation is separate from actual transfers. Aggregate download throughput uses transferred bytes divided by the download phase's elapsed time, rather than summed worker times. Completion freezes elapsed time; beginning another sync clears its decisions, publication set and phases. Listing work is measured per call, includes fetch wait, and is reported as summed work rather than CPU parsing time. begin_discovery() records the actual BFS strategy and worker bound; record_directory_examined() counts each distinct directory admitted by the coordinator, including entries at the depth limit and failed listing attempts. Both fields and the count reset at each sync. discovery_throughput divides that count by phase_seconds["discovery"], which includes cache loading and file-list collection. The progress banner measures the BFS walk alone. Throughput is nullable when discovery did not start or its elapsed time is not positive. Failed scans retain their partial work and failed status; a rate is not evidence of successful discovery. The summary groups mode, worker count, examined count, elapsed time and throughput, and omits the former rate derived from summed listing work. Summed worker time can exceed elapsed discovery time.

The text summary reports this sync's cache/subsystem activity using a baseline of lifetime counters. JSON retains existing metric keys and adds metrics.run with explicit outcomes, selection coverage, status, reasons, phase_seconds, elapsed_seconds, discovery_mode, discovery_workers, directories_examined and nullable discovery_throughput/download_throughput. Directory signatures, in-memory listing reuse, HTML hits/misses and cache bypasses remain distinct. listing_fetches remains the logical listing GET count. listing_get_seconds, directory_signature_head_calls, directory_signature_head_seconds and directory_signatures_from_get expose directory request work separately. Request times use perf_counter() around the connection-manager call, including failures, retries and pacing within that call. Counts exclude additional transport redirect/retry attempts. Text uses per-sync deltas; JSON retains collector totals for these keys.

Normal sync finalization goes directly from the transfer phase to obsolete-file cleanup and cache persistence. Do not walk the whole destination to produce a completion statistic: that includes unrelated files, delays unchanged runs and does not measure allocated filesystem space. get_directory_size() remains an explicit library helper, but sync does not call it. Keep the initial filesystem free-space check and the planned-batch preflight before any payload GET. Transfer budgets still reserve staging and assembly space; published-byte counters supply the summary's downloaded-byte total.

At the start of get_remote_files(), clear DirectoryScanner.fresh_listing_headers and fresh_listing_content_signatures. Populate these maps only after a current GET returns HTTP 200 and the listing parses successfully; discard both of the URL's previous entries before a new fetch. Signature collection prefers a nonempty ETag, then Last-Modified. Without either validator, store and reuse a content: digest of the complete decoded listing encoded as UTF-8, using hash_algorithm, consistent with get_directory_signature(html_content=...). Retain the digest rather than the HTML. The existing HEAD path remains the fallback when no current successful GET proof exists, including parsed/HTML cache hits from an earlier discovery. directory_signatures_from_get counts both header and content reuse. Directory signatures never substitute for a child file's freshness check. Listing failures still mark the scan incomplete and prevent obsolete-file cleanup and signature-cache publication.

Test repeated syncs, async fallback, HTTP skips, failed publication and both backends when changing reporting. No release-version labels belong in usage descriptions.

MirrorURL is a context manager — use with MirrorURL(cfg) as mirror: so __exit__/cleanup tears down pools, async loops, and the health server.

Runtime guarantees and compatibility

These invariants are part of the current implementation:

The sync pipeline materializes the remote file list in memory. Explicit scan_mode: parallel uses a discovery pool of min(workers, 4) threads; adaptive (default), sequential and async scan modes retain sequential discovery. The pool submits at most one bounded batch of listings at a time. The coordinator consumes results in breadth-first order and owns scope/depth/exclusion decisions, symlink state and incomplete-scan flags. It checks scope, depth and duplicate identity before submitting requests. Workers retain both directory per-IP pacing and normal connection-manager/transport rate limits. Pool shutdown cancels queued work and joins running requests on normal exit, exceptions and generator close. Scanner counters and shared XPath evaluation use a short state lock; HTTP work does not hold it. Fresh GET proofs retain their current-pass lifetime, and any listing failure still blocks cache publication and obsolete-file cleanup. Discovery logs counts and monotonic elapsed time before and during network work, then reports the final wall-clock rate; unknown tree size has no percentage or ETA. The later file-collection pass no longer emits a cached-work discovery rate.

parallel_threshold, download_queue_size, the batch-size settings, and disk-backed remote tracking do not select another pipeline or impose a list-memory bound. FileSystemCache is available as a component, but freshness checks use direct filesystem stats. The scanner's batch_processor supplies parser statistics; it is not a bounded download batch scheduler.

Other accepted compatibility settings include RGET-LIST options, http2_pipelining, circuit_breaker_downloads, chunk_timeout_multiplier, stats, and content_hash_small_files. Check actual readers before documenting an effect. Seven reserved model fields also emit warnings when explicitly changed: see _UNUSED_CONFIG_FIELDS and warn_unused_fields() in config.py.

Cached file ETags are bound to size, st_mtime_ns, and st_ctime_ns by CompareMixin._comparison_metadata. These stat fields do not prove content identity: same-size local edits with unchanged timestamps can be missed. Metadata-change regressions must explicitly advance the file timestamp rather than rely on native clock resolution. With verify_content=True, the size and SHA-256 receipt instead validate local identity; timestamp-only changes do not invalidate an otherwise matching receipt. Missing or invalid receipts fail closed even when a HEAD response reports matching size/time. Async checks run the hashing step in _meta_check_executor to avoid blocking the event loop. Content-verification runs disable the enclosing 30-second check and 120-second batch deadlines while retaining the 15-second HEAD timeout. Otherwise a hash can outlive its cancelled task and be repeated by fallback workers, causing unnecessary downloads of large unchanged files.

download_integrity.file_sha256 reads regular files in 1 MiB chunks and rejects observable stat changes during hashing. The single, assembled and streaming download paths hash the completed staging file before os.replace, then pass the digest to CacheManager.save_file_metadata(..., sha256=digest). Cache entries can contain a receipt without an ETag. Cache saves use os.replace so existing JSON can be replaced on Windows. Legacy receipts are not invented from an unverified destination: missing receipts require downloading the file again. This mode adds local verification; remote freshness retains the HTTP policy.

The regression contracts live in test_release_audit_regressions.py, the download failure/integrity/storage-fault tests, HTTP workflow tests, and config precedence tests. Keep those observable contracts intact during refactoring.

The configuration system

Transfer configuration includes mode (mirror or download), backend (httpx or aiohttp), url_list, overwrite and requests_per_second. The default rate remains 20 requests/second with 50 ms minimum spacing. effective_request_interval is the maximum of both limits; zero for both explicitly requests unpaced traffic. Neither option changes security policy. CLI overrides use the existing explicit-argument precedence rules for YAML/JSON too.

The single-URL shortcut uses download_url as an alternative to url_list. --mode download FILE_URL supplies the parent URL, current working directory and a destination-specific system temporary log folder only for omitted target fields. Explicit scope/paths win. CLI source selection clears the other configured source; the model and transfer entry point reject two sources or no source. Standalone models still require all three target fields.

transfers.py owns the shared async whole-file pipeline and the known-URL chunk adapter. Its backend adapters provide raw bytes and response headers; the controller owns redirects, a monotonic request budget, aggregate bandwidth, fixed workers, full filename preflight, disk headroom, retry bounds, scratch leases and atomic publication. For whole-file mode, the HTTPX adapter selects its reader using the actual worker count. One worker uses SecureTransport and a synchronous HTTPX client, running header/body/close operations through asyncio.to_thread. Raw bytes are grouped into small bounded batches with a short batching window, reducing thread handoffs and file writes without requiring slow streams to fill a complete block. Multiple workers use SecureAsyncTransport. Both readers only supply response headers and raw bytes; AsyncTransfers._one provides whole-file policy and _publish provides the shared identity, hash, timestamp and atomic-publication policy. Cancellation drains an in-flight thread operation before response closure and staging cleanup, even when shutdown cancels a worker again. The optional aiohttp resolver returns the validated public address directly to its connector, preserving the original hostname for Host and TLS. aiohttp requests retain encoded URL paths, use HTTP/1.1 and disable implicit redirects, decompression, cookies and proxies.

The controller samples shutil.disk_usage(target).free once per batch, at the first accepted response before payload writes, and subtracts worker write headroom. The shared in-memory budget reserves a known Content-Length before streaming or charges each unknown-length block before writing. Reservations stay charged after publication. Retries return the old reservation only after unlinking owned staging; failed transfers return it only after proven scratch cleanup. Failed cleanup retains the reservation conservatively. No per-MiB filesystem queries occur. This budget does not allocate disk extents or exclude other writers; write/fsync failures continue through the existing preservation and receipt rules. The same budget serves known-URL downloads and the aiohttp mirror transfer batch.

Explicit HTTPX chunk modes in known-URL downloads also use this budget: traditional chunks reserve twice the file length before workspace allocation, while streaming chunks reserve one length. A streaming pre-allocation fallback reserves the additional assembly length before chunk writes. The probe, retries and redirects share the batch's monotonic request pacer; chunks and whole-file fallbacks share its bandwidth budget. Published lengths stay charged after temporary chunks and assembly work are removed.

download_url_list(MirrorConfig(mode="download", ...)) takes destination/log ownership before creating state. It consumes either url_list or one download_url through the same plan, backend, scratch and publication path. Receipts are namespaced by the scoped base URL under .mirror-url-state, and existing payloads require a matching receipt or explicit overwrite. It never calls the scanner or metadata freshness layer. ReportMixin.sync() uses the same pipeline for backend="aiohttp" after its normal discovery and checks, then saves results through CacheManager. HTTPX remains the default mirror implementation, including ranged resume and chunks. The whole-file pipeline preserves those existing partials.

AsyncTransfers._one_chunked adapts the same ParallelDownloadManager used by mirror mode, rather than implementing another range engine. _ChunkConnection uses the known-URL HTTPX client's pinned transport, validates every redirect against the selected scope, preserves Range/If-Range and paces each request. A HEAD probe supplies size, range support and a strong representation validator to create_chunks; no scanner, freshness or cache lookup runs. Ineligible files use _one. ParallelFileDownload.publish delegates completed staging to _publish and immediately records ownership for completed files, including when cancellation arrives before the event loop receives the worker result. Chunk workers drain before scratch cleanup, client closure and lock release; cancellation prevents pending publication. Known-URL chunk retries respect max_retries without the mirror engine's additional failed-chunk recovery pass. The default whole-file and aiohttp paths remain available.

The CLI creates one MetricsCollector and passes it through download_url_list, AsyncTransfers, ParallelDownloadManager, and format_download_summary. Its RunReport observes the same publications and chunk-worker concurrency. The summary uses MetricsCollector.report in both modes; JSON-only callers can feed chunk_metrics to the same renderer. Chunk plans are logged by create_chunks after representation validation and staging selection. Constructor capacity/policy messages stay at DEBUG. The semaphore wrapper brackets each admitted worker with shared metrics and progress observations, releasing admission even if an observer raises. get_stats distinguishes planned_chunks from actual active_chunks. Run peaks reset on RunReport.start; cumulative counters retain API compatibility while console and download-result counters use each run's own activity.

progress.DownloadProgress observes both known-URL and mirror transfers. ReportMixin.sync() owns its context for the download phase and passes the same observer to aiohttp workers and HTTPX whole-file/ranged workers. Sequential body writes and concurrent chunk writes update received bytes before publication; chunk retries replace only their own range in the file position, while discarded traffic remains in aggregate received bytes. Retained resume bytes advance the file position without adding to this run's traffic or transfer rate. Chunk observations use the published destination name, independent of staging paths. The single metadata ProgressTracker clears before transfers start, and all transfer bars and the timer close before the summary. Both trackers use the same tqdm constructor; the base package requires tqdm, while the progress extra remains a compatibility alias. Renderer failures still degrade safely. Byte formatting uses the shared utils.format_bytes(); completion summaries follow mirror units and live byte progress uses IEC labels. One timer thread waits ten seconds between progress messages, independently of the async loop so synchronous fsync and optional SHA-256 work remain visible. Workers update bounded active-file observations under a short lock; logging runs outside that lock. Retry attempts reset per-file progress, while aggregate received bytes include discarded attempts. Completion/failure counts update only after the worker result and publication receipt hook. The context stops and joins the timer before releasing destination ownership or closing logs, including exception unwinding. No timer is started for silent or quiet runs.

--progress-bar and its explicit --progress alias enable per-file byte bars without requiring --print-logs. Positions are reused after workers finish; only active bars are retained. Body updates use tqdm's refresh throttling, while stage changes and timer ticks refresh waiting/flushing/hashing displays. Unknown lengths have no percentage or ETA. Retry bars reset to byte zero, and all bars close on success, failure or exception. Renderer errors do not fail transfers. Periodic observations mark whether every active file has a visible bar. The shared console ProgressConsoleHandler writes ordinary messages through public tqdm.write() so logs do not overwrite active bars. Its ConsoleProgressFilter suppresses periodic observations on stderr while the file handler retains them. Missing, failed or disabled bars and receipt-saving without an active bar retain console updates; warnings and completion summaries always pass through the filter.

The CLI uses a download-specific startup banner and metrics.format_download_summary() for readable completion logs, using the destination already validated by the CLI. Mirror and known-URL completion logs share MetricsCollector.report(), _format_run_summary() and the RunReport coordinator clock. Known-URL summaries retain mirror's field order and subsystem counters, marking discovery/metadata as not performed and reporting zero scan, cache, freshness and obsolete-file work. Known-URL preparation, downloads and cleanup are measured separately; cleanup includes receipt saving, progress timer shutdown and transfer guard cleanup. The outer CLI owner releases the run's kernel locks after completion output and log closure. Missing versus existing publication counts use the preflight local file identity; existing downloads are not classified as changed without a freshness check. The adapter prepares metrics and delegates formatting to MetricsCollector.report(); it appends no separate download-details footer. Shared completion helpers in metrics.py supply the sync outcome, processed scope and pool-closure messages for both CLI paths. RunReport.closed_connection_pools records successful known-URL backend closures outside the JSON snapshot and resets for each run; dry runs and preflight failures create no pools. The JSON result schema remains unchanged; its elapsed clock includes final timer and transfer guard cleanup. With --print-logs and without --quiet, the CLI sends human logs to stderr and leaves stdout empty. Otherwise it emits one JSON result on stdout. CLI logging flags control progress visibility, matching the shared handler configuration.

The new candidate needs its own CI and full qualification runs. Frozen validation of an older source snapshot does not cover this implementation.

MirrorConfig is the validated pydantic runtime model, including Paths, enums, flags and numeric bounds. ConfigSchema is a compatibility alias for this same model, so standalone file validation and runtime construction agree.

CLI args ──┐
           ├─► cli.main(): merge explicit overrides ─► MirrorConfig ─► MirrorURL
config file┘   (expand environment variables, then validate merged values)

validate_config_file() validates a standalone complete YAML/JSON file. load_config_from_args() remains a separate public helper. The CLI allows required URL/path values to come from explicit arguments before validating its final merged model. Benchmark mode uses the same merge precedence. load_config_from_args() maps an already populated argparse namespace; it does not read args.config or implement the CLI's explicit-override detection.

check_files is a list of literal remote paths relative to base_url. Its field validator resolves explicit @ UTF-8 list files from the current working directory, validates paths, rejects nested lists and preserves ordered unique entries. Models carry resolved paths, so per-suffix reconstruction does not reread list files. CLI inputs replace YAML/JSON inputs only when explicit. The option is mirror-only and does not change the verify_content/missing_files conflict or the normal freshness policy.

Model bounds can raise pydantic ValidationError; cross-field/URL checks can raise ConfigError. extra="forbid" rejects unknown fields. Environment expansion leaves unset ${VAR} placeholders intact. Logging handler setup is controlled by CLI logging flags, rather than config-file verbosity fields. For listing modes, CLI-only runs supply scratch paths and a shallow directory depth; config-file runs still require URL/destination/log fields and use their model/file depth unless explicitly overridden. Library/config-file listing modes are validated, and an explicit CLI listing choice disables the opposing file mode.

Subsystem deep-dives

SSRF-hardened transport (transport.py). SecureTransport / SecureAsyncTransport wrap httpx and reject requests whose resolved address is loopback or private. This is a security boundary: do not weaken it for convenience. Both accept a test_mode flag that relaxes the guard; this is how integration tests hit a local server (see Testing). Note the flag is not wired from MirrorConfig and should remain test-only. Existing HTTP tests install scoped monkeypatch transport bypasses or replace the fixture's pooled client; they need no new production configuration flag. DNS validation classifies every returned answer, including IPv6 link-local addresses, and rejects mixed public/non-public results. URL decode exceptions reject requests.

Directory parsing (parsing.py). Lightweight-parser selection when lxml is unavailable is independent of the lxml-failure fallback flag. Disabling fast_parsing_fallback does not require lxml to be installed. Scanner HTML statistics read the live CacheManager cache.

Circuit breakers (circuit_breaker.py). CircuitBreakerManager keeps one breaker per domain, created lazily via get_breaker(domain). State transitions are CLOSED → OPEN → HALF_OPEN → CLOSED. A historical bug (fixed in 3.1.13) was that the manager's record_*/can_execute methods didn't lazily create the breaker, so production domains never tripped — when changing this code, keep the lazy-creation path intact and covered by tests.

Concurrency (concurrency.py). UnifiedConcurrencyManager coordinates selected sync and chunk work. Metadata and file executors also have independent limits; there is no single cap covering every thread and async task. Streamed requests keep their coordinator lease until their response is closed. Always close streamed responses in a finally.

The standalone submit_to_shared_pool() helper rechecks its condition after every wakeup and rejects submission during shutdown. Completion includes failed and cancelled jobs; each contributes once to the failure count. Its queue_size argument is accepted for compatibility and does not bound pending submissions. use_shared_thread_pool enables the coordinator's chunk pool; file transfers and metadata comparisons retain separate executors. The ordinary chunk path reuses the raw executor and acquires its own coordinator leases.

Async path (async_connection.py). AdaptiveAsyncManager tunes its concurrency from measured RTT, throughput, and error rate; AsyncTaskManager runs the metadata HEAD checks. The non-adaptive AsyncConnectionManager has its own fixed admission and client lifecycle; do not assume adaptive-only attributes exist on it. Both request paths validate redirect scope and apply retry/backoff handling to 429 and retryable 5xx responses.

Cache (cache.py). CacheManager owns the JSON cache lifecycle: load + validate (_validate_and_sanitize_cache), schema-version gating (CACHE_SCHEMA_VERSION), atomic save via a temp file, and corrupted-file backup. The cache filename is built in _MirrorBase.__init__, not here. Directory metadata is saved after a completed scan and again at sync completion; file identity metadata is updated after successful publication. Parsed-listing LRU caches are in memory, not persisted/restored HTML caches. refresh_timestamp() changes expiry metadata only and is used by quick mode.

Extension recipes

These are the common changes and the exact touch-points.

Add a configuration option

  1. Add the field once to MirrorConfig in config.py, with a sensible default and pydantic validation bounds. ConfigSchema is the same class, so there is no second schema to edit.
  2. If it should be settable from the CLI, follow the flag recipe below: update parser/mapping and direct constructors as well as the public namespace helper.
  3. Read self.config.<field> where the behavior lives (a mixin or a subsystem).
  4. Add a test (a config round-trip test for the field; a behavior test for the effect). Document it in USER_GUIDE.md if user-facing.

Add a CLI flag

Flags are defined directly on parser/the argument groups in cli.main(). Add the argparse argument there, matching its dest to the MirrorConfig field name whenever possible — main()'s --config branch picks up any flag whose dest equals a MirrorConfig field automatically (via _cli_overrides); only a dest that differs from the field name (or that needs special handling, e.g. the three mutually-exclusive download-mode flags) needs an entry in _CLI_DEST_TO_CONFIG_KEY or a branch in _cli_overrides. The non---config branch (MirrorConfig(...) call further down main()), its benchmark constructor, and load_config_from_args() (the separate public namespace helper) also need the field passed explicitly. Test no-config, config-file override, and benchmark paths. Keep --help text consistent with the User Guide's option tables, and add the new option's row to the TYPED/boolean-flag tables in tests/test_cli_config_precedence.py (a missing row fails test_typed_table_covers_every_valued_option). Review the complete python -m mirror_url --help output too: both the option descriptions and the parser's example epilog must agree with current behavior and the User Guide. Flag-existence checks alone do not verify those claims.

Add a download mode

  1. Add a value to DownloadMethod in enums.py.
  2. Implement the mechanism in download.py (ParallelDownloadManager), and make auto_select_method able to return it when appropriate.
  3. Handle the new method where methods are dispatched in _core/report.py/_core/downloads.py (the if method == DownloadMethod.… branches).
  4. Expose a CLI flag (see above) if users should be able to force it.
  5. Add tests covering selection and the download path.

Add a new exception type

Add it to exceptions.py, subclassing the nearest existing base in the MirrorError hierarchy. If it is part of the public surface, re-export it from __init__.py and add it to __all__.

Add a method to MirrorURL

Pick the mixin matching the method's responsibility (scan/compare/download/ cleanup/report/urls) and add it there. Use shared state initialized in _MirrorBase.__init__; don't introduce a second __init__. If the method is public API, consider whether it belongs on the documented surface.

Add a new subsystem module

Place it with related responsibilities and keep module-level runtime imports acyclic; the historical layer numbers do not prohibit all same-layer imports. Wire it into MirrorURL from _MirrorBase.__init__ (layer 6 is where composition happens). Add unit tests at its own layer with no higher-layer setup.

Coding conventions

Testing

The suite lives in tests/ and runs under pytest. Test lanes:

End-to-end tests: tests/test_integration.py runs a real local HTTP mirror with a transport bypass installed only by monkeypatch in that test. The server fixture serves a dedicated served/ directory. No production configuration flag bypasses the transport's private-IP guard.

Where to add tests:

Markers are declared in pyproject.toml under [tool.pytest.ini_options] (--strict-markers is on, so register new markers there).

Unused code review

Use static findings as candidates, then verify calls and attribute receivers:

python -m pip install vulture
python -m vulture src/mirror_url --min-confidence 80
python -m vulture src/mirror_url tests --min-confidence 80

For each candidate, inspect repository call sites, __all__ exports, inheritance, dynamic lookup, and framework dispatch. Pydantic validators, HTTP/HTML handler hooks, signals, context managers, and pytest fixtures may have no ordinary call expression. Required callback parameters remain even when unused. An export or test-only call does not prove participation in the production sync workflow.

Record whether a finding is internal dead state, an intentionally dormant public helper, a compatibility field, or a framework hook. Do not remove a public API solely because the repository has no production caller. Examples retained for library users include get_remote_timestamp, retry_with_backoff, compute_file_hash, AdaptiveBatchProcessor.record_batch, and FileSystemCache.get_stat.

Match receivers precisely: DirectoryScanner.batch_processor has parser-stat readers; that does not establish a reader for an attribute with the same name on a MirrorURL instance. Verify actual per-IP semaphores and cache locks before removing synchronization. Vulture is a review aid, not a zero-findings CI gate.

Build, lint, and type-check

# editable install with the dev toolchain
pip install -e ".[all,dev]"
pre-commit install        # optional but recommended

ruff check .              # lint
ruff format --check .     # canonical formatter
mypy                      # required type-check of src/mirror_url (zero errors)
pytest -m "not integration"   # fast lane
pytest                        # full suite (includes integration)
pytest --cov=mirror_url --cov-branch --cov-fail-under=80 \
  --cov-report=term-missing --cov-report=xml:coverage.xml \
  --cov-report=json:coverage.json --cov-report=html
python scripts/check_download_coverage.py coverage.json  # each download module: 100%
python scripts/check_safety_coverage.py coverage.json
python scripts/check_safety_mutations.py

# build distributions
pip install build twine
python -m build           # wheel + sdist into dist/
python -m twine check dist/*

# regenerate both HTML guides after Markdown edits (requires pandoc)
bash scripts/render_guides.sh

CI (.github/workflows/ci.yml) runs lint/format checks on Python 3.12 and the fast test lane across Python 3.10–3.14. A separate Python 3.12 coverage job installs [all,dev], runs every test, including real local HTTP mirroring, and requires at least 80% combined statement/branch coverage overall. It additionally requires 100% statement and branch coverage separately for download.py and download_integrity.py, using the exact missing counts in the JSON report. It uploads HTML, JSON, and XML reports. The safety gate separately requires 100% statements and branches for scanner.py, _core/scan.py, _core/urls.py, _core/cleanup.py, security.py, filename_mapping.py, transport.py, destination_lock.py and scratch.py, plus utils.url_within_scope and utils._relative_url_path. It uses exact missing counts and checks helper exclusions; unrelated uncovered utility functions do not get hidden or counted as covered.

Hypothesis generates once-decoded path identities, origin boundaries and nested traversal encodings. Filesystem tests inject permission, resolution, listing and move failures, and assert preserved source and archive bytes. Local HTTP tests assert that malformed listings and lossy filenames fail before destructive cleanup or publication. test_filename_mapping.py checks the native filesystem and can use a separately mounted case-sensitive APFS test image through MIRROR_URL_CASE_TEST_VOLUME. Real HTTP tests preserve both NASA-style names, different contents and separate receipts through all modes and repeat syncs; insensitive filesystem tests require failure before any selected file payload. Additional tests preserve preexisting aliases and replaced/shared probe entries. The mutation check temporarily weakens twenty-one guards: mixed DNS rejection, archive collision rejection, incomplete-scan cleanup, same-origin scope, address pinning, lossy filename preflight and content receipt verification, the long-hash deadline policy, destination locking and retention of abandoned-worker ownership, scratch manifest ownership, live work leases and late chunk writer cleanup, case-insensitive filename collision rejection, probe inode/metadata identity, timestamp file-handle identity and preservation of existing filename aliases. Each selected test must first pass against the original source and then fail an assertion against the mutation. Collection or environment errors do not count as success. These checks improve evidence of correctness; neither 100% coverage nor this small mutation set guarantees that all bugs have been found.

test_process_recovery.py starts real subprocesses against the local HTTP fixture, kills them at deterministic IO checkpoints, then restarts the same destination. Checkpoints cover partial/chunk writes, both sides of publication, partial cache JSON serialization and a completed MOVE before cleanup finishes. All three download modes assert complete destination bytes, preserved obsolete bytes, persisted receipts and a subsequent sync without another file download. process_recovery_worker.py supplies test-only checkpoints; production has no crash hooks or loopback security bypass. Process kill tests do not simulate power loss or arbitrary external writers. A competing process must fail at every checkpoint before the owner is killed. test_process_destination_lock.py also checks simultaneous starts, parent/child overlap, independent trees, shared state, normal release, hard-kill recovery and cleanup while an HTTP writer is blocked. Native CI runs these tests on Windows and macOS too.

For a 24-hour synthetic soak, use the dev environment and a new output directory:

PYTHONPATH=src python scripts/run_local_soak.py \
  --output /path/to/new-soak-directory --duration-hours 24 --interval 60

The harness cycles through all download modes, repairs same-size corruption, preserves files on failed listings, checks MOVE collisions and recovers dropped connections and process kills. It writes atomic status.json, source hashes, resource samples and bounded logs. Freeze src/, tests/ and scripts/ for a long run: source changes invalidate the result. With optional psutil, resource checks include RSS and descriptors/handles; thread counts are always checked. Each completed cycle also asserts zero remaining owned temporary workspaces and bytes, and checks MOVE archive payload against the fixture's known legitimate files. Filesystem free space is sampled separately because other applications can change it. Crash worker output is appended with cycle/checkpoint markers and bounded rotation, allowing a full review of retained worker and main logs. An elapsed short smoke run is not long-duration evidence. Local synthetic tests also do not replace a representative archive/destination soak.

Native macOS and Windows CI jobs also run the full suite on Python 3.12 with core dependencies and with all optional dependencies. The coverage gate runs on Linux with optional dependencies; its results do not claim native Windows branches were executed on a different OS.

scripts/render_guides.sh renders both Markdown guides with Pandoc and the committed docs/guide-style.html. Its Lua filter points cross-guide links to HTML copies. It keeps the Markdown's table of contents and emits one title per guide; run it after editing either guide and review the HTML too. Keep Ruff's version in pyproject.toml, .pre-commit-config.yaml, and documentation synchronized when upgrading the formatter.

Release process

  1. Bump the version with bash scripts/bump_version.sh X.Y.Z. It updates pyproject.toml, src/mirror_url/_version.py, and the version references in both Markdown/HTML guides. The CLI imports the shared version; no separate banner edit is needed. A test checks that the two version sources agree. The script replaces every occurrence of the current version in those files, including historical mentions of that exact version; review the diff and preserve history manually when needed. Regenerate HTML after content edits.
  2. Update CHANGELOG.md. Replace the release's [Unreleased] heading with ## [X.Y.Z] - YYYY-MM-DD, retain its notes, and remove any empty duplicate [Unreleased] section. For a release with no pending changes, leave no [Unreleased] heading. Follow Keep a Changelog (### Added/Changed/Fixed).
  3. Validate and commit the release. Run mypy, ruff check ., ruff format --check ., the full pytest suite, python -m build, twine check dist/*, and git diff --check. All checks must pass. Include the full branch-coverage run, both coverage gates and the safety mutation check shown above, and require release PR CI to pass before tagging. Commit the source metadata, changelog, and guides; push the release branch and merge its reviewed pull request.
  4. Tag the merged release on main and push the tag: 
    git switch main
git pull --ff-only origin main
git tag -a vX.Y.Z -m "mirror-url X.Y.Z"
git push origin vX.Y.Z
  5. The tag triggers .github/workflows/release.yml, which builds the wheel/sdist, creates a GitHub Release, and (when configured) publishes to PyPI via Trusted Publishing (OIDC — no API token). The PyPI step requires a one-time setup: a pending publisher on PyPI (owner = repo owner, workflow release.yml, environment pypi) and a matching pypi Environment in the repo settings. The PyPI job also requires the repository variable PUBLISH_TO_PYPI=true. Without that opt-in, it is skipped; the tag still builds the wheel and creates the GitHub Release. Verify the version and changelog before creating the tag.

Known hazards and gotchas

Preserve these constraints when extending or refactoring the current code.

Quick "where do I find…" map

I want to change… Go to
A tuning default or limit constants.py
An error type exceptions.py (+ __init__.py if public)
A run-mode / state enum enums.py
A config field config.py (MirrorConfig; ConfigSchema is its alias) + cli.py
URL scope/validation logic _core/urls.py
How the remote tree is discovered _core/scan.py (+ parsing.py)
"Is the file up to date?" logic _core/compare.py
How a file is actually downloaded _core/downloads.py → download.py
Range validation / resume metadata download_integrity.py
Loop binding / live async admission async_primitives.py
Persistent throttled-domain knowledge domain_health.py
Obsolete-file cleanup behavior _core/cleanup.py
The top-level run / summary _core/report.py (sync())
Construction / shared state / logging _core/_base.py
The on-disk cache format/lifecycle cache.py (filename in _core/_base.py)
SSRF / network security boundary transport.py, security.py
Throttling / retries / breakers rate_limiter.py, connection.py, circuit_breaker.py
CLI flags / entry point cli.py, __main__.py
The version number _version.py and pyproject.toml

This guide describes the architecture as of version 3.5.2. When you change the structure, update this document in the same PR.