Metadata-Version: 2.5
Name: nb-wrangler
Version: 0.9.0
Summary: A notebook curation tool for managing Jupyter notebook environments
Project-URL: Homepage, https://github.com/spacetelescope/nb-wrangler
Author-email: Todd Miller <jmiller@stsci.edu>
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.11
Requires-Dist: ddtrace
Requires-Dist: httpx
Requires-Dist: ipykernel>=6.0.0
Requires-Dist: jupyter
Requires-Dist: papermill>=2.4.0
Requires-Dist: ruamel-yaml>=0.17.0
Requires-Dist: setuptools
Requires-Dist: uv
Provides-Extra: dev
Requires-Dist: bandit; extra == 'dev'
Requires-Dist: black>=22.0.0; extra == 'dev'
Requires-Dist: flake8>=5.0.0; extra == 'dev'
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest-profiling; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: radon; extra == 'dev'
Description-Content-Type: text/markdown

# nb-wrangler

![nb-wrangler logo](docs/nb-wrangler-logo.png)

## Overview

`nb-wrangler` is a command-line tool that streamlines the curation of JupyterLab notebooks and their runtime environments. It automates the process of building and testing container images from notebook requirements, ensuring that notebooks have the correct dependencies to run successfully.

Key features include:

- **Environment Management:** Bootstraps its own dedicated Conda environments, isolated from your system's Python.
- **Dependency Resolution:** Compiles `requirements.txt` files from multiple notebooks into a single, consistent set of versioned dependencies.
- **Automated Testing:** Tests notebooks and their package imports to verify the environment.
- **Data Management:** Manages the data required to run notebooks.
- **Image Building:** Integrates with a build system to automatically create container images.
- **Local Installs:** Works equally well doing local installs with no Docker overhead or learning curve.

The project uses `micromamba` for environment management and `uv` for fast pip
package installation.

Note that while `nb-wrangler` was conceived as a way to streamline notebook
Docker image creation for JupyterHub, at it's core nb-wrangler is merely defining:

1. A set of notebooks particularly relevant to a science platform.
2. Any supporting data required to run those notebooks.
3. A Python environment capable of running the entire set of notebooks.
4. Tests to help verify the system is built correctly and correctly runs those
   notebooks.
5. Standard methods to locally install the notebooks, data, and Python
   environment and run the tests.

All of the above don't even build Docker images directly,  but nb-wrangler does
provide two ways to hand off the information to STScI's science-platform-images GitHub
repository which can autonomously or manually build an image from a wrangler
spec.

Two other points are worthy of note:

1. nb-wrangler can support easy installation of custom environments directly
   on the science platform that were not first pre-installed in the platform
   Docker image.  This can be exploited to set up shared global or team installations
   areas as well as personalized environments.
2. The network distribution and installation protocols used equally enable
   off-platform laptop users to set up the same environment locally in an easy
   manner.

## Installation

### Locally / On your laptop

#### Pip Installing (experimental)

nb-wrangler now supports pip installs into existing mamba environments:

```bash
pip install nb-wrangler
```

This is a relatively new feature and because the wrangler itself works by creating multiple
environments it is not always as reliable as the bootstrap method described below. If you do
use this method it's advisable to create a dedicated nbwrangler environment to install to.

#### Bootstrapping an Isolated Environment

To get started, bootstrap `nb-wrangler` to create the necessary environments and directories (by default in `$HOME/.nbw-live`):

```bash
curl https://raw.githubusercontent.com/spacetelescope/nb-wrangler/refs/heads/main/nb-wrangler >nb-wrangler
chmod +x nb-wrangler
./nb-wrangler bootstrap [<wrangler-spec>.yaml]
```

**Advanced**: If you specify the optional wrangler spec then the bootstrap will
use the wrangler source code from the repository and/or ref specified in the
spec, whether it is the production version or a dev version specified under
*dev_overrides*.  This is important when doing wrangler development to enable
you to work off your own fork or a branch.

After bootstrapping, you can activate and/or reactivate the `nbwrangler` environment with:

```bash
source ./nb-wrangler environment
```

This command sets up the shell environment and activates the `nbwrangler` Python environment so
that it (temporarily) replaces any other Python you had activated previously and is ready to
start executing wrangler commands.  It should be mentioned that when the wrangler creates target
environments from the spec it will install them indepdently to this nbwrangler environment which
is intended only to support the tool itself.

#### General Notes on Installing

##### `wget` is needed for Data Handling

The data download mechanisms in nb-wrangler utilize `wget` under the hood to fetch data URL's. If
your system does not already have it installed,  you need to install it before you can do much.  On Linux (including the science platforms) `wget` is almost certainly there but
if not it should be an easy package install.   On OS-X `wget` is an easy install via `brew` and also
via `mamba`.  Before working on data tasks, it's worth verifying `wget` is available on your PATH.

### nb-wrangler On the STScI Science Platform

This is viable but the exact environment settings and required workflows are still being formalized.

If you're curious contact octarine@stsci.edu and we will work out platform and image appropriate
instructions for doing in-situ development of platform environments using nb-wrangler.

## How It Works

The `nb-wrangler` workflow is divided into two main phases: **curation** and **reinstallation**.   Throughout the examples below you'll see the command **`nbw`** which is the modern pre-installed method of running `nb-wrangler` vs. manually installing the command `nb-wrangler`.

### Phase 1: Curation

Curation is the process of defining the notebooks, Python packages, and data
required for a specific environment. This is done by creating a `nbw-spec.yaml`
file that describes the desired environment. Typically notebook repository
maintainers perform these steps in addition their fundamental roles of producing
correct notebooks, pip requirements, and installable data.

The main curation workflows are:

- **`--spec-init FILENAME`:** Generate a basic default `spec.yaml` template to get started.
- **`--curate`:** Compiles notebook requirements, creates the mamba environment, and
  installs pip dependencies.
- **`--data-curate`:** Gathers data requirements from notebook repositories and
  adds them to the spec.
- **`--test, --test-imports, --test-notebooks`:** Tests the notebook imports and
  notebooks themselves in the context of the environment and data installation.

Example:
```bash
# Curate the software environment
nbw spec.yaml --curate

# Test environment basics rapidly
nbw spec.yaml --test-imports

# Curate the data dependencies
nbw spec.yaml --data-curate

# Run each notebook headless using papermill
nbw spec.yaml --test-notebooks

# Activate your new "target" environment!  Here kernel-name == mamba environment you are curating
source ./nb-wrangler activate <your-kernel-name>

# Deactivate your current environment
source ./nb-wrangler deactivate
```

The curation process involves:
1.  **Choosing notebooks:** Selecting the notebooks to be included in the environment.
2.  **Resolving dependencies:** Identifying and resolving any conflicts between Python packages.
3.  **Defining data sources:** Specifying the data required by the notebooks.
4.  **Testing:** Building the environment and testing the notebooks to ensure they run correctly.

### Phase 2: Reinstallation

Reinstallation is the process of creating a new environment from a completed `spec.yaml` file. This is useful for reproducing an environment on a different machine or for a different user.

The main reinstallation workflows are:

- **`--reinstall`:** Recreates the software environment from a spec.
- **`--data-reinstall`:** Installs the data required by the notebooks.
- **`--test, --test-imports, --test-notebooks`:** Tests the notebook imports and notebooks themselves within the defined environment and data.


Example:
```bash
# Reinstall the software environment
nbw spec.yaml --reinstall

# Reinstall the data
nbw spec.yaml --data-reinstall

# Run both import and notebook tests
nbw spec.yaml --test-all

# Activate your new "target" environment!  Here kernel-name == mamba environment you are curating
source ./nb-wrangler activate <your-kernel-name>

# Deactivate your current environment
source ./nb-wrangler deactivate
```

For both curation and reinstallation there is the assumption that tests may fail and it may be necessary
to circle back to earlier steps, make fixes, and iterate.

For more information on notebooks and environment curation see [Managing Notebook Selection and Environment](docs/notebooks_and_environment.md)
For more information on supporting data see [Managing Notebook Reference Data](docs/data.md)

## Advanced Usage

### Development Overrides

To streamline development with custom branches without altering your core `spec.yaml`, `nb-wrangler` supports `dev_overrides`.

- The `dev_overrides` section in `spec.yaml` allows you to temporarily specify development branches for repositories.
- Use the `--dev` flag (or rely on implicit activation for curation workflows) to apply these overrides.
- Use the `--prod` flag to explicitly disable these overrides and force production settings.
- Use `--spec-disable-dev-overrides` to deactivate the `dev_overrides` section when preparing for production.

By default, nb-wrangler uses `--prod` mode (production, no dev overrides). Set the `NBW_OVERRIDES_MODE` environment variable to change the default behavior:

```bash
# Default to development mode for all workflows
export NBW_OVERRIDES_MODE="--dev"

# Explicitly set production mode (same as default)
export NBW_OVERRIDES_MODE="--prod"
```

CLI flags (`--dev`, `--prod`) always override the environment variable. For example, even if `NBW_OVERRIDES_MODE=--dev` is set, running with `--prod` will disable dev overrides.

For more details, see the [Spec Format documentation](docs/spec-format.md).

### SPI Injection

`nb-wrangler` can also inject the package and test requirements from a spec into the classic Science Platform Images (SPI) repository layout. This is a transitional feature to support older build processes.

See the [SPI Injection documentation](docs/inject-spi.md) for more details.

## Configuration Options

`nbw` provides a wide range of command-line options to customize its behavior. Here are some of the most common ones, grouped by function:

### Workflows

Workflows are commands that execute an ordered sequence of steps to accomplish some end-to-end task:

- `--curate`: Run the full curation workflow to define notebooks and Python environment.
- `--reinstall`: Reinstall an environment from a spec.
- `--reset-curation`: Delete installation artifacts like the environment, install caches, and spec updates.
- `--data-curate`: Curate data dependencies.
- `--data-reinstall`: Reinstall data dependencies.
- `--submit-for-build`: Submit a spec for an automated image build.
- `--inject-spi`: Inject a spec into the SPI repository.

### Environment Management

- `--env-init`: Create and kernelize the target environment.
- `--env-delete`: Delete the target environment.
- `--env-archive-delete`: Delete the "code" portions of the pantry for the current spec (environment archives).
- `--env-pack`: Pack the target environment into an archive file.
- `--env-unpack`: Unpack an environment from an archive.
- `--env-register`: Register the environment as a Jupyter kernel.
- `--env-unregister`: Unregister the environment from Jupyter.
- `--env-compact`: Compact the wrangler installation by deleting package caches.
- `--env-archive-format`: Override format for environment pack/unpack.
- `--env-print-name`: Print the environment name for the spec.
- `--env-kernel-cleanup`: Scans the user's kernel registry for 'dead' kernels (kernels pointing to non-existent environments) and removes them.

### Package Management

- `--packages-compile`: Compile package requirements.
- `--packages-install`: Install packages into the environment.
- `--packages-uninstall`: Uninstall packages from the environment.
- `--packages-omit-spi`: Don't include 'common' SPI packages.
- `--packages-ignore-versions`: Remove version constraints from notebook requirements.txt files before environment compilation.

### Testing

- `-t`, `--test-all`: Run all tests (`--test-imports` and `--test-notebooks`).
- `--test-imports`: Test package imports.
- `--test-notebooks [REGEX]`: Test notebook execution. Can optionally take a comma-separated list of regex patterns to select specific notebooks.
- `--test-notebooks-exclude [REGEX]`: Exclude notebooks from testing using a comma-separated list of regex patterns.
- `--test-copy-shared [GLOB]`: Copy Python modules matching a glob pattern to the runtime directory where an individual import test is running with respect to a particular notebook.
- `--jobs INT`: Number of parallel jobs for notebook testing.
- `--timeout INT`: Timeout in seconds for notebook tests.

### Data Management

- `--data-collect`: Collect data archive and installation info and add to spec.
- `--data-list`: List data archives.
- `--data-download`: Download data archives to the pantry.
- `--data-update`: Update metadata for data archives (e.g., length and hash).
- `--data-validate`: Validate pantry archives against the spec.
- `--data-unpack`: Unpack data archives.
- `--data-pack`: Pack live data directories into archive files.
- `--data-reset-spec`: Clear the 'data' sub-section of the 'out' section of the spec.
- `--data-delete [archived|unpacked|both]`: Delete data archives and/or unpacked files.
- `--data-env-vars-mode [pantry|spec]`: Define where to locate unpacked data.
- `--data-print-exports`: Print shell exports for data environment variables.
- `--data-env-vars-no-auto-add`: Do not automatically add data environment variables to the runtime environment.
- `--data-select [REGEX]`: Regex to select specific data archives.
- `--data-no-validation`: Skip data validation.
- `--data-no-unpack-existing`: Skip unpack if the target directory exists.
- `--data-symlinks`: Create symlinks from install locations to the pantry data directory.
- `--data-no-symlinks`: Do not create symlinks during --data-unpack.

### Spec Management

- `--spec-init FILENAME`: Generate a basic default `spec.yaml` template file.
- `--spec-reset`: Reset the spec file to its original state (preserves `out.data`).
- `--spec-add`: Add the spec to the pantry (a local collection of specs).
- `--spec-list`: List available specs in the pantry.
- `--spec-select [REGEX]`: Select a spec from the pantry by regex.
- `--spec-validate`: Validate the spec file.
- `--spec-update-hash`: Update spec SHA256 hash.
- `--spec-ignore-hash`: Do not add or verify the spec hash.
- `--spec-add-pip-hashes`: Record PyPI hashes for packages during compilation.
- `--print-wrangler-repo`: Print the nb-wrangler repository URL associated with this spec.
- `--print-wrangler-ref`: Print the nb-wrangler repository ref associated with this spec.
- `--print-repo-tags`: Print out the repositories sections of the spec with one URL and reference per line.
- `--spec-name`: Generate a name suitable for referring to this image/spec/build and print to stdout.

### Miscellaneous
- `--verbose`: Enable DEBUG log output.
- `-q`, `--quiet`: Suppress all log output to stderr; only stdout will be visible.
- `--debug`: Drop into debugger on exceptions.
- `--profile`: Run with cProfile and print stats.
- `--reset-log`: Delete the log file.
- `--log-times [MODE]`: Include timestamps in log messages (valid modes: absolute, elapsed, both, none).
- `--color [MODE]`: Colorize log output.

### SPI Automation
These flags automate Docker builds, testing, security scanning, and git operations during an `--inject-spi` workflow. See the [SPI Injection documentation](docs/inject-spi.md) for examples.

- `--spi-branch NAME`: Create a new branch in the SPI repo with this name (auto-generated if omitted).
- `--spi-commit-message MESSAGE...`: Commit message for the new branch. If not provided, a default message is used.
- `--spi-image-name`: Print the image name corresponding to the current spec to stdout and exit.
- `--spi-inject-reqs`: Copy the appropriate requirements fields from the wrangler spec into locations in the SPI repo clone.
- `--spi-push-branch`: Push the new branch to the remote SPI repo.
- `--spi-pr`: Create a pull request for the new branch in the SPI repo.
- `--spi-prune-docker`: Prune old Docker images before a build to free space.  Only removes SPI images.
- `--spi-image-build`: Trigger a Docker build in the SPI repo.  REQUIRED prior to the switches below.
- `--spi-run-lab`: Run Jupyter Lab in a Docker container for the SPI deployment based on the current spec. Choose "Shutdown Jupyter Lab" from the lab File menu to exit cleanly when you're done.
- `--spi-image-test [PARAMS]`: Run the image-test script for the SPI deployment. `PARAMS` is an optional single string of parameters passed to the nested image-test invocation; it is split on whitespace into individual arguments (quote each parameter and any attached value, e.g. `"\"--test-notebooks --dev\""`). If present without a value, an empty parameter list is used. You must specify either no parameters (`--test-imports`) or one of `--test-imports`, `--test-notebooks`, `--test-all` as your first parameter in the string. Use the form `--spi-image-test='parameters...'` for it to parse correctly with parameters.
- `--spi-image-scan`: Run the image-scan script for the SPI deployment, scanning the built Docker image for security vulnerabilities.

### Docker Registry

These parameters are used to interact with the public image repo, currently spacetelescope GHCR nb-wrangler-images.

- `--docker-pull IMAGE`: Pull a Docker image from a registry.
- `--docker-cat IMAGE`: Extract and print /spec.yaml from a Docker image to stdout.
- `--docker-list SPEC_GLOB`: List all spec names matching the spec-glob from the registry.

### Notebook Clones
- `--clone-repos`: Clone notebook repos to the directory indicated by --repos-dir.
- `--repos-dir PATH`: Directory where notebook and other repos will be cloned.
- `--delete-repos`: Delete --repo-dir and clones after processing.
- `--repos-clean [PATTERN]`: Clean up specified patterns in cloned repos (defaults to __pycache__).
- `--overwrite-local-changes`: In any cloned repo, overwrite local uncommitted changes to match the requested ref.
- `--stash-local-changes`: In any cloned repo, stash local uncommitted changes before matching the requested ref.
- `--use-dirty-repos`: In any cloned repo, use the current state as-is even if it has local changes.

For a full list of options, run `nbw --help`.


## Input Formats

`nb-wrangler` uses several input formats to define the environment:

- **Notebook (`.ipynb`):** Jupyter notebooks.
- **Wrangler Spec (`spec.yaml`):** The main YAML file that defines the notebook repositories and Python environment. See the [spec format documentation](docs/spec-format.md) for details on the current spec format (version 2.3), which uses a `repositories` dictionary, named `selected_notebooks` blocks, top-level package lists, optional `assets`, `environment_vars`, `test_environment_vars`, and nested `system.commands` overrides.
- **Notebook Repo:** A Git repository containing Jupyter notebooks.  e.g., [TIKE Content](https://github.com/spacetelescope/tike_content), [Roman Notebooks](https://github.com/spacetelescope/roman_notebooks)
- **Science Platform Images (`SPI`):**  The GitHub repository where code for the docker images for the Science Platforms is kept.  [Science Platform Images](https://github.com/spacetelescope/science-platform-images)
- **Refdata Spec (`refdata_dependencies.yaml`):** A YAML file in a notebook repository that specifies data dependencies. See the [refdata dependencies documentation](docs/refdata_dependencies.md).
- **Requirements (`requirements.txt`):** A file specifying Python package dependencies for a notebook in its directory.
- **Supporting Python (`.py`):** Any supporting Python files (`.py`) included in a notebook's directory.

The goal of nb-wrangler is to combine these inputs, resolve any conflicts, and create a unified environment capable of running all specified notebooks.

Secondary goals,  include but are not limited to:

- Collecting, freezing, distributing, and re-installing **data** associated with notebook repos.
- Initializing notebook and terminal environment variables as spec'ed, partcularly regarding spec'ed/installed data which may be installed in a shared location.
- Building Docker images for curators or science platform admins or pipelines.
- Testiing environments (importing all requested package) and notebooks.
- Automating any/all of these tasks for notebook repos / curators and the science platforms.
.
