Metadata-Version: 2.4
Name: conda-oci-mirror
Version: 0.3.0
Summary: Tools to mirror conda packages to an OCI registry.
Author-email: conda-oci-mirror team <conda_oci_mirror@issues.github.com>
License: BSD-3-Clause
Keywords: mamba,conda,oci
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: requests
Requires-Dist: oras==0.1.14
Requires-Dist: conda-package-handling
Requires-Dist: zstandard
Requires-Dist: py-rattler>=0.15
Requires-Dist: msgpack
Dynamic: license-file

# Conda OCI Mirror

Welcome to the Conda Forge Mirror python module! We use this module
to retrieve and update conda mirror packages. You can see it being used
in the wild [here](https://github.com/channel-mirrors/mirrormirror/blob/main/.github/workflows/main.yml)
and brief usage is shown below.

## Concepts

### Mirror

A **mirror** means you will copy (mirror) all packages from a regular conda-channel to an OCI registry.
In practice, this typically means **all** packages, however for the purposes of testing we allow selection of a subset.
You must have control of the registry you intend to mirror to, meaning you can push and pull from it.
When you do a mirror, the repodata.json is always pulled fresh, and any local changes you've made are
over-written. We do this so the local cache is in sync with the remote.
The downloaded repodata snapshot is published only after all scheduled package uploads succeed;
an upload failure prevents repodata publication. Metadata is not downloaded again at publication time.
Filtered mirrors still publish the full upstream repodata, not a filtered index.

Packages whose encoded version/build cannot form a valid OCI tag (for example, build strings
containing spaces or `*`, or tags longer than 128 characters) are skipped before package downloads
or registry lookups for that archive. Each skip is logged as a warning with the source archive and tag.
Mirroring and cache operations continue with valid packages, and the CLI prints a summary warning
but still exits with status 0. Python callers can inspect `mirror.errors` after each
operation; `PackageRepo.find_packages()` also exposes its scan errors via `repo.errors`.
This does not rename packages or remove entries from the upstream repodata snapshot: a run with
skipped packages is incomplete, not a successful full mirror. No GitHub issues are created automatically.

### OCI channel layout

Package archives are stored by package name and version/build tag. Repository metadata is stored at:

```text
<registry>/<channel>/<subdir>/repodata.json:latest
```

That manifest contains `repodata.json`, `repodata.json.zst`, and, when the source channel provides it, `repodata_shards.msgpack.zst` as separate media-type layers. Content-addressed repodata shards are stored as:

```text
<registry>/<channel>/<subdir>/repodata.json/shards:<sha256>
```

The shards repository is nested under `repodata.json` so it can never collide with a package repository, since conda package names cannot contain `/`. Clients map the index's relative `./shards/<sha256>.msgpack.zst` URLs to this repository.

Shards are pushed before the index that references them. Each run compares the upstream index with the one published at `repodata.json:latest` and only downloads and pushes shards the published index does not already reference, so the registry's tags are not listed. Until the first index is published, the shard tags are listed once so that an interrupted first push resumes where it stopped. Shards are uploaded by `--shard-workers` threads (default 2); keep this low to avoid overloading the registry.

`--repush-shards` pushes every shard in the index again, not only new ones. Use it once after a change to the shard manifests, such as the switch of the shard media type from `+zstd` to `+zst`: shards that are already published are otherwise never pushed again and keep their old manifests. The blobs are unchanged, so this rewrites one manifest per shard. A run with this flag does not resume, so if it is cut off, run it again with the flag.

Shards are mirrored only for complete channel mirrors. A package-filtered mirror cannot publish the upstream shard index because it would reference packages that were not mirrored.

### Pull Cache

A **pull-cache** can pull from a registry that you may not be able to write to, to your local cache.
It selects the newest conda version and build number separately for each package and archive format;
when builds tie, the first record in repodata wins. Versions are not ordered by upload date.
Downloaded OCI layers are checked against their SHA-256 digests before replacing local files.
Interrupted or invalid downloads leave an existing destination unchanged and remove temporary files.
Missing metadata, missing requested package layers, and download failures cause the command to fail rather than silently succeed.
Upstream package downloads also replace files atomically and verify checksums when repodata supplies them.

### Push Cache

A **push-cache** can push your local cache to a registry you control. This means that we compare packages you've
built against what are known in the repodata.json, and we push the ones that are not known to the repodata.json.
A push cache with `--push-all` (also available as `--all`) will push all selected local packages to your registry,
regardless of status. Cache pushes leave local archives and repodata unchanged, including during dry runs and failed uploads.
They compare against the existing local repodata without running `conda index`.

### Archive formats and concurrent publishers

A package tag can contain both `.conda` and `.tar.bz2` archive layers. Uploading one format preserves the other;
shared `info` layers describe the most recently uploaded archive. No new tag naming scheme is required.
Uploads to the same tag are serialized within a run; independent mirror processes must not write the same tag concurrently.

Normal mirroring keeps the tag-only check for builds with a single upstream format. When upstream lists both formats,
the mirror checks the manifest and repairs a missing layer. Known-new tags need no preservation lookup on their first upload;
cache pushes and retries inspect existing manifests. None of these checks download package blobs.
For a full layer-level audit, `PackageRepo.get_existing_packages()` verifies manifests by default; this is intentionally
more expensive than normal mirror planning. Registry authorization and server errors are not treated as absent packages.

## Usage

### Install

Python 3.8 or newer is required. Conda version ordering uses `py-rattler`, not PEP 440 ordering.

Create a new environment:

```bash
$ python -m venv env
$ source env/bin/activate
```

And install:

```bash
$ pip install -e .
```

### Authentication

For registries that require authentication, set `ORAS_USER` and `ORAS_PASS` in the environment.
Use `--dry-run` to test without pushing. Missing credentials do not enable dry-run automatically;
anonymous writes are attempted and succeed only if the registry permits them.

```bash
export ORAS_USER=myuser
export ORAS_PASS=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

Note that when you make the token, ensure the packages box (for read and write)
is checked.

Each `Mirror` owns an independent `client`, including its transport and credentials.
Worker processes retain that configuration but create their own HTTP sessions.
For standalone reads, pass the original registry URL (including `http://` for local registries),
or pass `client=mirror.client` when constructing a `PackageRepo` from `mirror.registry`.
The legacy global `oras` client remains available for direct callers but is not used by mirror operations.

### Shared options

All commands accept repeatable `--package` names or glob patterns, such as `--package 'numpy*'`.
Quote patterns to prevent shell expansion. `--push-all` includes already-indexed packages but still honors the package filter.
`--quiet`, `--debug`, and `--push-all` are switches; do not append `true` or `false`.
Workers must be positive. `--upload-delay` sets a nonnegative delay in milliseconds; `--timeout` remains an alias for it,
not a network timeout. `--quiet` suppresses informational logs, not all progress output.

### Mirror

The main functionality of the mirror is to create a copy of a channel and packages
and push them to a mirror (or just stage for a dry run). Let's say that we want to
mirror `conda-forge` and the package `zlib` We might first do a dry run:

```bash
$ conda-oci mirror --channel conda-forge --package zlib --dry-run
```

And then for realsies, here we want to push to our organization `myorg`

```bash
$ conda-oci mirror --channel conda-forge --package zlib --registry ghcr.io/myorg
```

For this command, we are pulling packages from a **channel** and mirroring to the
registry defined under **user**.

### Pull Cache

You can use `pull-cache` to pull the latest packages to a local cache.
The same authentication is needed only if the registry packages are
private or internal. Akin to the others, you can start with a `--dry-run`

```bash
$ conda-oci pull-cache --registry ghcr.io/researchapps --dry-run
```

Target a specific arch (subdir of the cache) and one package we know exist

```bash
$ conda-oci pull-cache --registry ghcr.io/researchapps --dry-run --subdir linux-64
```

If you want to preview what would be pulled, set `--dry-run`:

```bash
$ conda-oci pull-cache --registry ghcr.io/researchapps --subdir linux-64 --package zlib --dry-run
```

```console
Downloading conda-forge/linux-64/repodata.json to /home/vanessa/Desktop/Code/conda_oci_mirror/cache/conda-forge/linux-64/conda-forge/linux-64/repodata.json
Would be pulling zlib, but dry-run is set.
```

For this command, we are pulling packages from our registry defined as **user** and mirroring
to a local filesystem cache.

### Push Cache

You can use `push-cache` to push the packages in your cache to your remote.
This command will require authentication, and you are also encouraged to use
`--dry-run` first.

```bash
$ conda-oci push-cache --registry ghcr.io/researchapps --dry-run --package zlib --subdir linux-64
```

### Python API

#### Mirror

Let's say we want to do a simple mirror of a conda forge package to our own registry at `http://127.0.0.1:5000`.
Note that cache_dir will default to "cache" in your present working directory.

```python
from conda_oci_mirror.mirror import Mirror

mirror = Mirror(
    channel="conda-forge",
    packages=["redo"],
    # Push repodata and packages to this registry
    registry="http://127.0.0.1:5000/dinosaur",
    subdirs=["noarch"],
)

updates = mirror.update()
```

#### Pull Cache

A pull cache means that we start with a mirror, but then pull down packages from our registry
that aren't in our local cache. The creation of the Mirror is the same, except we call a different
function.

```python
from conda_oci_mirror.mirror import Mirror

mirror = Mirror(
    channel="conda-forge",
    packages=["redo"],
    # Push repodata and packages to this registry
    registry="http://127.0.0.1:5000/dinosaur",
    subdirs=["noarch"],
)

latest_packages = mirror.pull_latest()
```

### Push Cache

A push cache checks your local repodata.json and finds packages that exist that aren't yet added,
and then updates and pushes them to your registry. You can also use `push_all` to push all
local packages regardless of presence in the repodata.json. First, here is pushing new:

```python
from conda_oci_mirror.mirror import Mirror

mirror = Mirror(
    channel="conda-forge",
    packages=["redo"],
    # Push repodata and packages to this registry
    registry="http://127.0.0.1:5000/dinosaur",
    subdirs=["noarch"],
)

pushed_packages = mirror.push_new()
```

And pushing all:

```python
all_packages = mirror.push_all()
```

If you want to run in serial (for either of the above):

```python
all_packages = mirror.push_all(serial=True)
```

Right now this is checking against the local repodata.json, and I'm not sure if this should
be checking against the registry (intuitively it should if we expect a push-cache to push local cache
entries that aren't in the remote to the remote).

#### PackageRepo

You can use the `PackageRepo` class to interact directly with a package in a registry.
As an example, let's say we want to interact with a local registry `http://127.0.0.1/dinosaur`
to look for the `conda-forge` channel, `linux-64` subdirectory and package `zlib`.
You might first mirror the package there as follows:

```bash
$ conda-oci mirror --registry http://127.0.0.1:5000/dinosaur --subdir linux-64 --package zlib
```

You could also do this first with the Python API to get explicitly back the list of tags mirrored.
We would create a package repo as follows:

```python
from conda_oci_mirror.repo import PackageRepo
import os

cache_dir = os.path.join(os.getcwd(), "cache")
repo = PackageRepo(
    "conda-forge", "linux-64", cache_dir, registry="http://127.0.0.1:5000/dinosaur"
)
```

Now let's retrieve the index.json. You need the exact tag you are interested in - there
is no "latest."

```python
# Should retrieve from
# http://127.0.0.1:5000/dinosaur/conda-forge/linux-64/zlib:1.2.11-0'
index_json = repo.get_index_json("zlib:1.2.11-0")
```

```
{'arch': 'x86_64',
 'build': '0',
 'build_number': 0,
 'depends': [],
 'license': 'zlib',
 'license_family': 'Other',
 'name': 'zlib',
 'platform': 'linux',
 'subdir': 'linux-64',
 'version': '1.2.11'}
```

Now we might want to get the package info:

```python
info = repo.get_info("zlib:1.2.11-0")
# <tarfile.TarFile at 0x7f7b78a00e80>
```

This is technically a tarfile, so to iterate over members:

```python
for member in info:
    print(member.name)
```

And finally, to get the full package archive:

```python
pkg = repo.get_package("zlib:1.2.11-0")
```

Note that we first try to get the new format (.conda) and fall back to .tar.bz2.

### Development

If you want to test pushing to packages, make sure to export your credentials first,
as discussed above. Then ensure that `--user` is targeting your GitHub user or organizational
account to push to:

```bash
$ conda-oci mirror --channel conda-forge --package zlib --registry ghcr.io/researchapps
```

For a package that includes the new format:

```bash
# Mirror zope.event from conda-forge to our local registry
$ conda-oci mirror --channel conda-forge --subdir noarch --package zope.event --registry http://127.0.0.0:5000/dinosaur

# Pull the package from our registry to out local cache
$ conda-oci pull-cache --channel conda-forge --subdir noarch --package zope.event --registry http://127.0.0.0:5000/dinosaur
```

You can also develop with a local registry (instead of ghcr.io):

```bash
$ docker run -it --rm -p 5000:5000 ghcr.io/oras-project/registry:latest
```

And then specify your local registry - oras
will fall back to insecure mode given that you've provided http://.

```bash
$ conda-oci mirror --channel conda-forge --package testinfra --registry http://127.0.0.0:5000/dinosaur --subdir noarch
```

And run tests:

```bash
$ pixi run test
```

To use an already-running test registry instead of starting Docker containers, set
`CONDA_OCI_TEST_REGISTRY=http://127.0.0.1:5000`. Tests use isolated namespaces in that registry.

See [TODO.md](TODO.md) for some questions and items to do.

### Linting

We use [ruff](https://docs.astral.sh/ruff/) for linting and formatting, run through [pixi](https://pixi.sh):

```bash
$ pixi run lint  # check
$ pixi run fmt   # fix and format
```
