Metadata-Version: 2.4
Name: mirrorer
Version: 0.1.0
Summary: dist-mirror shard mirroring CLI
Author: Postgres Professional
License: Proprietary
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: docker>=7.0
Requires-Dist: pydantic>=2.7
Requires-Dist: python-json-logger>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# mirrorer

`mirrorer` is the shard-host CLI for dist-mirror. It runs repository sync
containers, KESL antivirus scan, ZFS snapshots, snapshot replication and
publication commands and repository catalog generation.

Repository catalog generation is bundled as reusable library code in
`mirrorer.services.catalog` and is exposed through `mirrorer catalog`. There is
no separate `repo-catalog` console script.

## Install

Install on a shard host into the dist-mirror virtualenv:

```bash
python3.11 -m venv /opt/dist-mirror/venv
/opt/dist-mirror/venv/bin/pip install \
  --index-url "https://gitlab.example/api/v4/projects/<project-id>/packages/pypi/simple" \
  mirrorer
```

The installed CLI entrypoint is:

```bash
/opt/dist-mirror/venv/bin/mirrorer --help
```

## Configuration

Runtime config is external to the package. The default path is:

```text
/etc/dist-mirror/dist-mirror.yaml
```

Example:

```yaml
log:
  log_level: INFO

mirror:
  workdir: /opt/dist-mirror
  syncer_dir: /etc/dist-mirror
  zfs_mirror_dataset: distpool/mirror
  zfs_mirror_dir: /distpool/mirror
  kesl_max_workers: 4

replication:
  replica_host: shard-replica01.example.org
  ssh_user: tech_dist-mirror
  skip_remote: false

antivirus:
  fail_on_threats: false

repo:
  paths:
    apt/mirror:
      deb.debian.org: debian
    dnf/repos:
      centos: centos

syncer:
  - type: apt
    name: debian
    prefix: debian
    config: syncer.yaml

  - type: rsync
    name: epel
    prefix: epel
    repos: syncer_repos.yaml

  - type: dnf
    name: rpm
    prefix: rpm
    config: syncer.yaml
    repos: syncer_repos.yaml
    dest: dnf/repos

  - type: rmt
    name: sles-x86_64
    prefix: sles-x86_64
    config: syncer.yaml
    repos: syncer_repos.yaml
    dest: rmt_x86_64/storage/public/repo
```

Syncer config files are expected under:

```text
/etc/dist-mirror/
```

Do not commit secrets into config. Deploy `/etc/dist-mirror/dist-mirror.yaml`,
`/etc/dist-mirror/syncer.yaml` and `/etc/dist-mirror/syncer_repos.yaml` with the
infrastructure configuration system.

## Commands

Common options:

```bash
mirrorer --config /etc/dist-mirror/dist-mirror.yaml --log-level INFO --no-json-logs <command>
```

Run all syncers:

```bash
mirrorer sync
```

Run one configured prefix:

```bash
mirrorer sync --syncer debian
```

Stage commands:

```bash
mirrorer sync
mirrorer av-scan
mirrorer snapshot
mirrorer replicate
mirrorer publish-snapshot
mirrorer publish-inventory
mirrorer publish-catalog
mirrorer catalog --root-dir /opt/dist-mirror/repo/snapshots/20260603-120000
mirrorer cleanup-state
```

Snapshot selection:

```text
publish-snapshot                 # publishes state/current_snapshot.txt
publish-snapshot --snapshot NAME # publishes NAME
publish-snapshot --all           # rebuilds publication for all ZFS snapshots
publish-catalog                  # catalogs state/current_snapshot.txt
publish-catalog --snapshot NAME  # catalogs NAME
publish-catalog --rebuild        # catalogs all published snapshots
```

`publish-snapshot` builds the public repository tree from symlinks to ZFS
snapshot contents and does not write metadata into the repository tree.

Metadata is stored outside the published symlink repository:

```text
/opt/dist-mirror/metadata/
├── inventory.json
└── snapshots/
    └── <snapshot>/
        └── repo_catalog.json
```

The stable HTTP metadata contract is:

```text
/metadata/inventory.json
/metadata/snapshots/<snapshot>/repo_catalog.json
```

The nginx mapping for `/metadata/` is managed outside `mirrorer`, but the layout
above is part of this package contract.

`catalog` is the direct standalone catalog builder and does not require a
mirrorer config file or shard runtime binaries:

```bash
mirrorer catalog \
  --root-dir /opt/dist-mirror/repo/snapshots/20260603-120000 \
  --output /tmp/repo_catalog.json \
  --format apt \
  --format rpm \
  --summary-json
```

## Logging

Plain text logs are the default and are intended for GitLab job logs. Use
`--json-logs` when machine-readable logs are needed. Python tracebacks are
visually marked in plain log mode.

Secrets must not be written into event fields or command arguments that are
logged.

## Build

Local build with uv:

```bash
uv sync --extra dev
uv run ruff check .
uv run pytest
uv build
```

`mypy` is included in the development dependencies, but the extracted codebase
should be tightened incrementally before making type checks a required CI gate.

## Publish

GitLab CI publishes tagged builds to the GitLab PyPI Package Registry:

```bash
uv publish \
  --publish-url "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi" \
  --username gitlab-ci-token \
  --password "${CI_JOB_TOKEN}" \
  dist/*
```
