Metadata-Version: 2.4
Name: myst_libre
Version: 0.4.1
Summary: A Python library for managing source code repositories, interacting with Docker registries, handling MyST markdown operations, and spawning JupyterHub instances locally.
Author-email: agahkarakuzu <agahkarakuzu@gmail.com>
License: MIT License
        
        Copyright (c) 2024 Agah Karakuzu
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/neurolibre/myst_libre
Keywords: myst,docker,jupyterhub,markdown,repository
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: docker
Requires-Dist: python-dotenv
Requires-Dist: PyGithub
Requires-Dist: rich>=13.0.0
Requires-Dist: mystmd
Requires-Dist: repo2data
Requires-Dist: pyyaml
Requires-Dist: gitpython
Requires-Dist: plotext>=5.2.0
Dynamic: license-file

# MyST Libre

![PyPI - Version](https://img.shields.io/pypi/v/myst-libre?style=flat&logo=python&logoColor=white&logoSize=8&labelColor=rgb(255%2C0%2C0)&color=white)

Following the [REES](https://repo2docker.readthedocs.io/en/latest/specification.html), `myst-libre` streamlines building [✨MyST articles✨](https://mystmd.org/) in containers.

* A repository containing MyST sources
* A Docker image (built by [`binderhub`](https://github.com/jupyterhub/binderhub)) in a public (or private) registry, including:
  * Dependencies to execute notebooks/markdown files in the MyST repository
  * JupyterHub (typically part of images built by `binderhub`)
* Input data required by the executable content (optional)

Given these resources, myst-libre starts a Docker container, mounts the MyST repository and data (if available), and builds a MyST publication.

> [!NOTE]
> This project was started to support publishing MyST articles as living preprints on [`NeuroLibre`](https://neurolibre.org).

## Installation

### External dependencies 

> [!IMPORTANT]
> Ensure the following prerequisites are installed:

- Node.js (For MyST)  [installation guide](https://mystmd.org/guide/installing-prerequisites)
- Docker              [installation guide](https://docs.docker.com/get-docker/)

### Install myst-libre

```
pip install myst-libre
```

**Set up environment variables:**

If you are using a private image registry and/or Curvenote CLI features, create a `.env` file in the project root and add the following:

```env
DOCKER_PRIVATE_REGISTRY_USERNAME=your_username
DOCKER_PRIVATE_REGISTRY_PASSWORD=your_password
CURVENOTE_TOKEN=your_curvenote_api_token
```

The `CURVENOTE_TOKEN` is required for operations like `curvenote submit`, `curvenote deploy`, `curvenote pull`, etc. You can generate an API token from your [Curvenote profile settings](https://curvenote.com/profile?settings=true&tab=profile-api&subtab=general).

## Quick Start

**Import libraries and define REES resources**

Minimal example to create a rees object:

```python
from myst_libre.tools import JupyterHubLocalSpawner, MystMD
from myst_libre.rees import REES
from myst_libre.builders import MystBuilder

rees = REES(dict(
                  registry_url="https://your-registry.io",
                  gh_user_repo_name = "owner/repository"
                  ))
```

Other optional parameters that can be passed to the REES constructor:


- `gh_repo_commit_hash`: Full SHA commit hash of the `gh_user_repo_name` repository (optional, default: latest commit)
- `binder_image_tag`: Full SHA commit hash at which a binder tag is available for the "found image name" (optional, default: latest)
- `binder_image_name_override`: Override the "found image name" whose container will be used to build the MyST article (optional, default: None)
- `dotenv`: Path to a directory containing the .env file for authentication credentials to pull images from `registry_url` (optional, default: None)
- `bh_image_prefix`: Binderhub names the images with a prefix, e.g., `<prefix>agahkarakuzu-2dmriscope-7a73fb`, typically set as `binder-`. This will be used in the regex pattern to find the "binderhub built image name" in the `registry_url`. See [reference docs](https://binderhub.readthedocs.io/en/latest/zero-to-binderhub/setup-binderhub.html) for more details. 
- `bh_project_name`: See [this issue ](https://github.com/jupyterhub/binderhub/issues/800) (optional, default: [`registry_url` without `http://` or `https://`])


Note that in this context what is meant by "prefix" is not the same as in the reference docs. (optional, default: `binder-`)

**Image Selection Order**

1. If the `myst.yml` file in the `gh_user_repo_name` repository contains `project/thebe/binder/repo`, this image is prioritized.
2. If `project/thebe/binder/repo` is not specified, the `gh_user_repo_name` is used as the image name.

Note that if (2) is the case, your build command probably should not be `myst build`, but you can still use other builders, e.g., `jupyter-book build`.

If you specify `binder_image_name_override`, it will be used as the repository name to locate the image.

This allows you to build the MyST article using a runtime from a different repository than the one specified in `gh_user_repo_name`, as defined in `myst.yml` or overridden by `binder_image_name_override`.

The `binder_image_tag` set to `latest` refers to the most recent successful build of an image that meets the specified conditions. The repository content might be more recent than the `binder_image_tag` (e.g., `gh_repo_commit_hash`), but the same binder image can be reused.

**Fetch resources and spawn JupyterHub in the respective container**

```python
hub = JupyterHubLocalSpawner(rees_resources,
                             host_build_source_parent_dir = '/tmp/myst_repos',
                             container_build_source_mount_dir = '/home/jovyan', #default
                             host_data_parent_dir = "/tmp/myst_data", #optional
                             container_data_mount_dir = '/home/jovyan/data', #optional
                             )
hub.spawn_jupyter_hub()
```

* MyST repository will be cloned at:

```
tmp/
└── myst_repos/
    └── owner/
        └── repository/
            └── full_commit_SHA_A/
                ├── myst.yml
                ├── _toc.yml
                ├── binder/
                │   ├── requirements.txt (or other REES dependencies)
                │   └── data_requirement.json (optional)
                ├── content/
                │   ├── my_notebook.ipynb
                │   └── my_myst_markdown.md
                ├── paper.md
                └── paper.bib
```

Repository will be mounted to the container as `/tmp/myst_repos/owner/repository/full_commit_SHA_A:/home/jovyan`.

* If a [`repo2data`](https://github.com/SIMEXP/Repo2Data) manifest is found in the repository, the dataset it names is expected to be **already staged** at:

```
tmp/
└── myst_data/
    └── my-dataset
```

**Data is never downloaded automatically.** A repository's `binder/data_requirement.json` can point at arbitrary sources, so running `repo2data` on any submitted repository would let it pull unreviewed content onto the build host. A build uses whatever is already present: if the directory is missing or empty, the build proceeds *without* the data mount and warns, so content that reads the dataset will fail to execute.

To fetch data once you have reviewed its source and destination, opt a single run in:

```python
hub = JupyterHubLocalSpawner(rees_resources,
                             host_build_source_parent_dir = '/tmp/myst_repos',
                             allow_repo2data_download = True,  # off by default
                             )
```

The dataset directory can also be named explicitly for data you staged yourself:

```
rees_resources.dataset_name = "my-dataset"
```

Dataset names are validated: `projectName` becomes a path component on both the host and in the container, so absolute paths and names containing `..` are rejected, and a name that resolves outside `host_data_parent_dir` (via a symlink, say) is refused.

When the data is present it is mounted read-only as `/tmp/myst_data/my-dataset:/home/jovyan/data/my-dataset`. If no data is declared, this step is skipped.

**Build your MyST article**

```python
MystBuilder(hub).build()
```

**Check out the built document**

In your terminal:

```
npx serve /tmp/myst_repos/owner/repository/full_commit_SHA_A/_build/html
```

Visit ✨`http://localhost:3000`✨.

## Table of Contents

- [Myst Libre](#myst-libre)
  - [Table of Contents](#table-of-contents)
  - [Installation](#installation)
  - [Usage](#usage)
    - [Authentication](#authentication)
    - [Docker Registry Client](#docker-registry-client)
    - [Build Source Manager](#build-source-manager)
    - [JupyterHub Local Spawner](#jupyterhub-local-spawner)
    - [MyST Markdown Client](#myst-markdown-client)
  - [Module and Class Descriptions](#module-and-class-descriptions)
  - [Contributing](#contributing)
  - [License](#license)

## Usage

### Authentication

The `Authenticator` class handles loading authentication credentials from environment variables.

```python
from myst_libre.tools.authenticator import Authenticator

auth = Authenticator()
print(auth._auth)
```


### Docker Registry Client

The DockerRegistryClient class provides methods to interact with a Docker registry.

```python
from myst_libre.tools.docker_registry_client import DockerRegistryClient

client = DockerRegistryClient(registry_url='https://my-registry.example.com', gh_user_repo_name='user/repo')
token = client.get_token()
print(token)
```

### Build Source Manager

The BuildSourceManager class manages source code repositories.

```python
from myst_libre.tools.build_source_manager import BuildSourceManager

manager = BuildSourceManager(gh_user_repo_name='user/repo', gh_repo_commit_hash='commit_hash')
manager.git_clone_repo('/path/to/clone')
project_name = manager.get_project_name()
print(project_name)
```

## Module and Class Descriptions

### AbstractClass
**Description**: Provides basic logging functionality and colored printing capabilities.

### Authenticator
**Description**: Handles authentication by loading credentials from environment variables.  
**Inherited from**: AbstractClass  
**Inputs**: Environment variables `DOCKER_PRIVATE_REGISTRY_USERNAME` and `DOCKER_PRIVATE_REGISTRY_PASSWORD`

### RestClient
**Description**: Provides a client for making REST API calls.  
**Inherited from**: Authenticator

### DockerRegistryClient
**Description**: Manages interactions with a Docker registry.  
**Inherited from**: Authenticator  
**Inputs**:
- `registry_url`: URL of the Docker registry
- `gh_user_repo_name`: GitHub user/repository name
- `auth`: Authentication credentials

### BuildSourceManager
**Description**: Manages source code repositories.  
**Inherited from**: AbstractClass  
**Inputs**:
- `gh_user_repo_name`: GitHub user/repository name
- `gh_repo_commit_hash`: Commit hash of the repository

### JupyterHubLocalSpawner
**Description**: Manages JupyterHub instances locally.  
**Inherited from**: AbstractClass  
**Inputs**:
- `rees`: Instance of the REES class
- `registry_url`: URL of the Docker registry
- `gh_user_repo_name`: GitHub user/repository name
- `auth`: Authentication credentials
- `binder_image_tag`: Docker image tag
- `build_src_commit_hash`: Commit hash of the repository
- `container_data_mount_dir`: Directory to mount data in the container
- `container_build_source_mount_dir`: Directory to mount build source in the container
- `host_data_parent_dir`: Host directory for data
- `host_build_source_parent_dir`: Host directory for build source
- `allow_repo2data_download`: Permit this run to fetch the dataset declared in the repository's `binder/data_requirement.json` (default `False`; see the data section above)
- `container_network`: Name of an existing Docker network to attach the spawned container to (default: Docker's default bridge; see "Restricting network access" below)
- `verify_metadata_blocked`: Refuse to spawn if the instance metadata service is reachable from `container_network` (default: `True` when `container_network` is set)
- `metadata_probe_image`: Minimal image used for that probe (default: `busybox:latest`)

### Restricting network access from build containers

> For a full server preparation runbook, see [docs/server-setup.md](docs/server-setup.md).

Notebook code from a submitted repository executes inside the spawned container with whatever network access that container has. On the default bridge that includes the cloud instance metadata service — on OpenStack, `169.254.169.254`, which serves user-data and config-drive contents.

Docker has no per-container egress ACL, so the block is applied with host firewall rules. Put build containers on their own network so the rules can target only that traffic:

```bash
docker network create --driver bridge \
  --opt com.docker.network.bridge.name=br-mystbuild \
  mystbuild
```

```python
hub = JupyterHubLocalSpawner(rees_resources,
                             container_network = 'mystbuild',
                             )
```

Then block the metadata service for that interface. Use `DOCKER-USER`, which Docker evaluates before its own rules and does not overwrite:

```bash
# Instance metadata (OpenStack, and the same address on EC2/GCP)
iptables -I DOCKER-USER -i br-mystbuild -d 169.254.0.0/16 -j REJECT
# If IPv6 is enabled on the instance
ip6tables -I DOCKER-USER -i br-mystbuild -d fe80::a9fe:a9fe -j REJECT
```

These rules match traffic *forwarded from* the container, so the host's own access to metadata (cloud-init, etc.) is unaffected. They do not survive a reboot on their own — persist them with `netfilter-persistent save` or the equivalent for your distribution.

To also keep build containers off internal networks, add rules for the ranges you use, for example:

```bash
iptables -I DOCKER-USER -i br-mystbuild -d 10.0.0.0/8 -j REJECT
```

Verify from inside a build container before relying on any of this:

```bash
docker run --rm --network mystbuild curlimages/curl \
  -s -m 3 http://169.254.169.254/openstack/ ; echo "exit=$?"
```

A non-zero exit (timeout or refused) means the rule is working.

myst-libre also checks this itself: with `container_network` set, the spawner probes the metadata service from that network before building and refuses to start if it answers. The result is cached per network per process. This exists because the Docker network survives a reboot while `DOCKER-USER` rules do not, so "the network exists" is not evidence that it is protected.

> **Note:** be careful before blocking `172.16.0.0/12` or the range holding your other Docker networks. In Docker-in-Docker mode myst-libre reaches the spawned Jupyter container by container IP, so blanket-blocking inter-container traffic will break the build.

### MystMD
**Description**: Manages MyST markdown operations such as building and converting files.  
**Inherited from**: AbstractClass  
**Inputs**:
- `build_dir`: Directory where the build will take place
- `env_vars`: Environment variables needed for the build process
- `executable`: Name of the MyST executable (default is 'myst')
- `state_file`: Path to the process state file used for orphan recovery (default: `<tmpdir>/myst_libre_processes.json`)

#### Reaping orphaned build processes

A `myst build` launches a tree — `myst` → `npm run start` → `node ./server.js` — that holds ports until it is torn down. Normal teardown kills the whole process group, but a worker crash or restart leaves no PID to signal and the tree survives as an orphan.

myst-libre records each launched process group at spawn time, along with the process that owns it. Under Celery, hook it to `celeryd_after_setup` — once per worker node, after setup, before children fork or any task is consumed:

```python
from celery.signals import celeryd_after_setup
from myst_libre.tools import MystMD

@celeryd_after_setup.connect
def reap_myst_orphans(sender, instance, **kwargs):
    reaped = MystMD.reap_orphans()
    if reaped:
        logging.warning(f"Reaped {len(reaped)} orphaned myst process group(s)")
```

Use `celeryd_after_setup`, not `worker_process_init` — the latter runs in every prefork child, so N reaps would race.

**Concurrency safety.** A record is only reaped when the process that launched it is gone. A build running right now in a sibling worker has a live owner and is left untouched — without that rule, a worker restart during a long build would kill healthy work, since a live build and an orphan are indistinguishable from the myst process alone. Because of this, `reap_orphans()` is safe to call at any time, including from a periodic task; a long-lived worker accumulates orphans from crashed children that no startup hook will see.

Orphan records are additionally re-checked against the recorded start-time key, so a recycled PID is dropped rather than signalled. Stale and dead entries are pruned on every call.

> Ports are deliberately not used as the key here. With `myst build --execute`, mystmd finishes the entire execution phase before starting either server, so on a long build no port exists for most of its duration and both appear only near the end. The pgid is known at launch and stays valid throughout.
