Metadata-Version: 2.4
Name: lispyn
Version: 0.19.0a1
Summary: Lisp-powered PyTorch experiments and Jupyter notebooks, built on Hy.
Author: Eduardo J. Barrios
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/edujbarrios/lispyn
Project-URL: Repository, https://github.com/edujbarrios/lispyn
Project-URL: Issues, https://github.com/edujbarrios/lispyn/issues
Project-URL: Documentation, https://github.com/edujbarrios/lispyn/blob/main/docs/guide.md
Keywords: lisp,hy,jupyter,pytorch,notebooks,machine-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: hy<2.0,>=1.0
Requires-Dist: nbformat<6,>=5.8
Requires-Dist: packaging>=23
Provides-Extra: notebook
Requires-Dist: nbclient<1,>=0.9; extra == "notebook"
Requires-Dist: ipykernel>=6; extra == "notebook"
Provides-Extra: data
Requires-Dist: pandas>=2; extra == "data"
Provides-Extra: vision
Requires-Dist: torch; extra == "vision"
Requires-Dist: torchvision; extra == "vision"
Requires-Dist: pillow; extra == "vision"
Provides-Extra: audio
Requires-Dist: torch; extra == "audio"
Requires-Dist: torchaudio; extra == "audio"
Requires-Dist: torchcodec; extra == "audio"
Provides-Extra: text
Requires-Dist: torch; extra == "text"
Provides-Extra: peft
Requires-Dist: peft>=0.15; extra == "peft"
Requires-Dist: transformers<5,>=4.45; extra == "peft"
Provides-Extra: plots
Requires-Dist: matplotlib>=3.8; extra == "plots"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: nbclient<1,>=0.9; extra == "dev"
Requires-Dist: ipykernel>=6; extra == "dev"
Dynamic: license-file

# Lispyn

[![PyPI](https://img.shields.io/pypi/v/lispyn?label=PyPI&color=40e0d0)](https://pypi.org/project/lispyn/) [![Python](https://img.shields.io/pypi/pyversions/lispyn?color=008080)](https://pypi.org/project/lispyn/) [![Tests](https://github.com/edujbarrios/lispyn/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/edujbarrios/lispyn/actions/workflows/tests.yml) [![Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-006666)](https://github.com/edujbarrios/lispyn/blob/main/LICENSE)

**Write complex experiments. Make them easy to understand, reuse, and share.**

Great experiments deserve to be understood—not just executed. Whether you're exploring new AI models, fine-tuning neural networks, or running advanced training pipelines, you often need to **show others what you built, how it works, and what you learned**.

**Lispyn** uses Lisp-based [Hy](https://hylang.org/) macros to turn repetitive steps into **concise, reusable building blocks**. Prepare datasets, train with PyTorch, document your choices in Markdown, and generate plots—all from a readable Lisp source that becomes a standard **Python Jupyter notebook**.

Spend less time rewriting boilerplate and more time experimenting. When your dataset, model, or training strategy changes, adapt the macro and reuse the workflow. **Lisp for expressive experiments. Python for the ecosystem. Jupyter for sharing results.**

## Quick installation

Install Lispyn directly from [PyPI](https://pypi.org/project/lispyn/):

```bash
python -m pip install lispyn

# Recommended for notebooks and Matplotlib plots
python -m pip install "lispyn[notebook,plots]"

# Check your current Python and available dependencies
lispyn doctor
```

Lispyn needs **Python 3.10+**. Basic installation includes Hy, so you can
import and use standard Python modules immediately. PyTorch, TorchVision,
and GPU-specific libraries are optional; install compatible versions for
your hardware before running AI training examples.

## Your first Lispyn notebook

Create a file named `hello.hy`:

```hy
(require lispyn.macros [md])
(md "# My first Lispyn experiment"
    "Synthetic training curves, generated with reusable Lisp macros.")

(require lispyn.macros [defcurve])
(setv history {"train_loss" [1.4 1.0 0.7 0.5]
               "val_loss" [1.5 1.2 0.9 0.75]})
(defcurve my-style :theme "docusaurus-dark"
  :palette "turquoise-spectrum")
(my-style history :metric "loss"
  :title "Training curves" :save "training.png")
```

Then run:

```bash
# Translate Lispyn/Hy into ordinary Python Jupyter cells
lispyn build hello.hy

# Execute the notebook, keeping its real outputs
lispyn execute hello.hy

# Or run a Hy file directly (without creating a notebook)
lispyn run hello.hy
```

`hello.ipynb` is a standard Jupyter notebook with real Markdown and Python
cells. The synthetic graph is written to `training.png`. All plots are
generated with Matplotlib; no new language interpreter or special kernel
is needed. After a Markdown cell, repeat `(require lispyn.macros [...])`
for any macros used in a later cell because Hy code cells compile independently.

**Developing Lispyn itself?** Clone the repository and use
`python -m pip install -e ".[dev,notebook,plots]"` instead.
See the [technical guide](https://github.com/edujbarrios/lispyn/blob/main/docs/guide.md)
and [release instructions](https://github.com/edujbarrios/lispyn/blob/main/docs/releasing.md).

## Matplotlib with reusable Lisp macros

Define a design once and apply it to any recorded history. This short example
uses a dark gray background inspired by Docusaurus and the seven-color
**turquoise-spectrum** palette:

```hy
(require lispyn.macros [defcurve compare-curves])
(setv history {"train_loss" [1.5 1.1 0.8 0.55]
               "val_loss" [1.6 1.2 0.95 0.76]})
(defcurve turquoise-dark
  :theme "docusaurus-dark"
  :palette "turquoise-spectrum"
  :train-color "#40E0D0" :val-color "#8CF0EB"
  :title-size 18 :line-width 2.8 :fill-alpha 0.08)
(turquoise-dark history
  :metric "loss" :title "Training dynamics"
  :subtitle "Synthetic example · 4 epochs"
  :save "training.png")
```

**Generated from Lispyn + Matplotlib using synthetic values, not a trained model.**

![Turquoise training and validation lines on a Docusaurus-like dark gray background](https://raw.githubusercontent.com/edujbarrios/lispyn/main/docs/images/synthetic_training_curves.png)

The same Lisp file can compare several named runs, automatically assigning
distinct colors from the palette:

![Three synthetic validation curves in turquoise on a dark gray background](https://raw.githubusercontent.com/edujbarrios/lispyn/main/docs/images/synthetic_comparison_curves.png)

Full reproducible source: [synthetic_curves.hy](https://github.com/edujbarrios/lispyn/blob/main/examples/synthetic_curves.hy)
· [generated Jupyter notebook](https://github.com/edujbarrios/lispyn/blob/main/example_generated_notebooks/synthetic_curves.ipynb).
Regenerate the figures with `lispyn run examples/synthetic_curves.hy`.

Classification reports are equally concise: the new `defconfusion` macro
creates a reusable dark-themed confusion matrix from actual and predicted labels.
See the [Lisp example](https://github.com/edujbarrios/lispyn/blob/main/examples/confusion_matrix.hy)
and its [generated figure](https://github.com/edujbarrios/lispyn/blob/main/docs/images/synthetic_confusion_matrix.png).
The example uses **synthetic class labels**, not real model predictions.

## Compare experiments and share classification reports

Compare two recorded training runs without rewriting plotting code:

```bash
lispyn compare runs/baseline runs/finetuned --metric loss --split val \
  --output comparison.md --plot comparison.svg
```

`lispyn compare` also exports CSV or JSON and reports the best metric,
its epoch, and the final value for each run. **No model execution is needed.**

Create per-class precision, recall, F1 and support metrics from predictions:

```hy
(require lispyn.macros [classification-report])
(setv scores (classification-report ["cat" "dog" "cat"]
                                    ["cat" "dog" "dog"]
                                    :save "classification.md"))
(print (get scores "accuracy"))
```

These calculations need neither scikit-learn nor Matplotlib. The
[synthetic classification example](https://github.com/edujbarrios/lispyn/blob/main/examples/confusion_matrix.hy)
generates a [Markdown metrics table](https://github.com/edujbarrios/lispyn/blob/main/docs/reports/synthetic_classification_report.md)
alongside its turquoise confusion matrix.

## Structure Jupyter cells from Lisp

Use literal macros for native Jupyter metadata and raw cells:

```hy
(require lispyn.macros [md raw-cell cell-tags])
(md "## Hyperparameters" "Keep experiment settings in a tagged code cell.")
(raw-cell "<!-- Exporter note: retained as a raw Jupyter cell -->")
(require lispyn.macros [cell-tags])
(cell-tags "parameters")
(setv learning-rate 0.001)
```

`cell-tags` tags the **next code cell** (use `parameters` or `hide-input`).
`raw-cell` creates a raw Jupyter cell without executing its text. Both
remain harmless no-ops when you run trusted Hy directly. The
[example notebook](https://github.com/edujbarrios/lispyn/blob/main/example_generated_notebooks/notebook_metadata.ipynb)
shows the resulting cell types and tags.
## Train smarter with one Lisp macro

Long PyTorch experiments can stop when a selected validation metric stops
improving. The new `train-until` macro handles this policy while keeping
checkpoints, histories, and the exact number of completed epochs:

```hy
(require lispyn.macros [train-until])
(setv run
  (train-until model training-data 8
    :validation validation-data
    :loss-fn loss-function
    :epochs 300
    :min-delta 0.0005
    :experiment-dir "runs/experiment"))
```

`300` is the maximum: if the monitored loss does not improve by more
than `min-delta` for eight consecutive epochs, training stops early.
With an experiment directory, the run records `best.pt`, `last.pt`,
`history.json`, and a summary of the completed epochs in `manifest.json`.
The model returned by the run remains at the **last** epoch; the best
weights can be restored explicitly with `load_checkpoint`.

Try the [offline early-stopping notebook](https://github.com/edujbarrios/lispyn/blob/main/example_generated_notebooks/early_stopping.ipynb)
or its [Lisp source](https://github.com/edujbarrios/lispyn/blob/main/examples/early_stopping.hy).

## Advanced training: accumulate gradients, schedule learning rates

Lispyn now supports native PyTorch gradient accumulation, optional global
gradient clipping and per-epoch learning-rate schedulers. Express the policy
once in Lisp and use the same Matplotlib styles to present real results:

```hy
(require lispyn.macros [train-accumulating learning-rate-curve])
(setv run
  (train-accumulating model training-data 4
    :validation validation-data
    :loss-fn criterion
    :epochs 100
    :lr 0.001
    :gradient-clip 1.0
    :scheduler "cosine"
    :experiment-dir "runs/optimized"))

(learning-rate-curve run
  :theme "docusaurus-dark"
  :palette "turquoise-spectrum"
  :save "runs/optimized/learning_rate.svg")
```

`accumulation-steps` changes the **optimizer update frequency**, not the
number of training examples; the last partial group is handled correctly.
`gradient-clip` applies after accumulating gradients. Schedulers include
`cosine`, `step`, and `plateau`, plus native PyTorch scheduler factories.
Learning rates are recorded in `history.json` and the returned Run.

An [offline working example](https://github.com/edujbarrios/lispyn/blob/main/examples/advanced_training.hy)
and its [generated notebook](https://github.com/edujbarrios/lispyn/blob/main/example_generated_notebooks/advanced_training.ipynb)
show the full process with a tiny CPU PyTorch classifier and dark turquoise
training/learning-rate plots.
## Full experiment — ResNet18, two-stage fine-tuning

The following is the complete source of [`examples/resnet_finetune_300.hy`](https://github.com/edujbarrios/lispyn/blob/main/examples/resnet_finetune_300.hy), including its Markdown cells. Put your images in `examples/data/resnet_images/<class_name>/*.jpg` (or `.png`); each class needs multiple images. The first stage trains the classifier for **10 epochs**, then the second unfreezes all layers for **300 epochs**.

```clojure
; ResNet18 experiment: all results are Python/Jupyter notebook cells.
; Required actual images: examples/data/resnet_images/class_name/*.jpg or *.png
; This source is NOT executed in CI (300 epochs need a real dataset/GPU).
(require lispyn.macros [md])

(md "# ResNet18 — transfer learning + full fine-tuning"
    "An end-to-end **PyTorch** experiment written in Lispyn. It loads labeled images, splits by class, runs head-only tuning, then unfreezes the backbone for **300 epochs**, and produces publication-ready curves."
    "**Author:** Eduardo J. Barrios · **Source:** Lispyn.")

(md "## Install required libraries"
    "The **pip-install** macro generates a dedicated Python notebook setup cell. Already available packages report *Requirement already installed in this environment*; only missing packages are installed in the current kernel."
    "Python standard-library modules such as `math` need no installation. For CUDA/accelerator setups, install matching PyTorch/TorchVision builds beforehand. Installing native packages may require a kernel restart.")

(require lispyn.macros [pip-install])
(pip-install "torch" "torchvision" "matplotlib>=3.8")

(require lispyn.macros [use-package])
(setv math (use-package "math" :install False))
(print "math.sqrt(16):" (math.sqrt 16))

(md "## Load and split the image dataset"
    "Use the original TorchVision preprocessing for the pretrained ResNet18 weights. Classes come from your own image folders; samples are split reproducibly into training and validation.")
(import torch)
(import torchvision)
(import lispyn.media [image-folder])
(import lispyn.data [split-dataset])
(import lispyn.reproducibility [seed-all])

(seed-all 42)
(setv weights torchvision.models.ResNet18_Weights.DEFAULT)
(setv dataset (image-folder "data/resnet_images" :transform (weights.transforms)))
(setv groups (split-dataset dataset :stratify True :validation-fraction 0.2 :seed 42))
(setv training (get groups 0))
(setv validation (get groups 1))
(setv model (torchvision.models.resnet18 :weights weights))
(setv model.fc (torch.nn.Linear model.fc.in_features (len dataset.classes)))
(print "Classes:" dataset.classes)
(print "Train/validation:" (len training) (len validation))

(md "## Phase 1 — classifier fine-tuning"
    "Freeze the pretrained ResNet18 backbone and optimize only `model.fc` for **10 epochs**. The class split is reproducible and separate from the training set."
    "**Note:** This example needs at least two labeled images per class. For correlated images from the same subject, use a *group-aware split* instead.")

(require lispyn.macros [head-tune])
(setv head-run
  (head-tune model training "fc"
    :validation validation
    :loss-fn (torch.nn.CrossEntropyLoss)
    :epochs 10
    :batch-size 16
    :lr 0.001
    :seed 42
    :metrics "accuracy"
    :experiment-dir "runs/resnet/head"))
(print "Head validation accuracy:" (get head-run.history "val_accuracy"))

(md "## Phase 2 — deep/full fine-tuning: 300 epochs"
    "Unfreeze **every ResNet18 layer**, create a new optimizer with a smaller learning rate, and train for **300 epochs**. This is full fine-tuning of pretrained weights, **not** training from scratch."
    "A real 300-epoch run may take hours and require a GPU. Run it only with your own dataset and appropriate resources.")

(require lispyn.macros [full-tune])
(setv deep-run
  (full-tune head-run.model training
    :validation validation
    :loss-fn (torch.nn.CrossEntropyLoss)
    :epochs 300
    :batch-size 16
    :lr 0.00001
    :seed 42
    :metrics "accuracy"
    :experiment-dir "runs/resnet/deep"))
(print "Full tuning best accuracy:" (max (get deep-run.history "val_accuracy")))

(md "## Curves and export"
    "Use one Lisp macro to define a consistent Matplotlib style for **both** training curves. The best checkpoints and JSON histories are saved under `runs/resnet/`.")

(require lispyn.macros [defcurve])
(defcurve resnet-curve
  :theme "paper"
  :train-color "#2563EB"
  :val-color "#D97706"
  :xlabel "Epoch"
  :train-label "Train"
  :val-label "Validation"
  :figsize [9 4])

(resnet-curve deep-run :metric "loss"
  :title "ResNet18 — 300-epoch loss"
  :ylabel "Cross-entropy loss"
  :save "runs/resnet/deep/loss.png")

(resnet-curve deep-run :metric "accuracy"
  :title "ResNet18 — 300-epoch accuracy"
  :ylabel "Top-1 accuracy"
  :save "runs/resnet/deep/accuracy.png")
(print "Saved ResNet18 plots, histories, and best checkpoints.")
```

## Generated notebook

**[Open the generated ResNet18 notebook](https://github.com/edujbarrios/lispyn/blob/main/example_generated_notebooks/resnet_finetune_300.ipynb)** in [`example_generated_notebooks/`](https://github.com/edujbarrios/lispyn/tree/main/example_generated_notebooks/). It contains real Markdown and translated Python code cells, but **no simulated losses or training outputs**. Use `lispyn gallery` to regenerate the examples without running training.

To install Lispyn from this repository, use `python -m pip install -e ".[notebook,plots]"`. Run `lispyn build examples/resnet_finetune_300.hy` to regenerate the notebook, or `lispyn execute examples/resnet_finetune_300.hy -o example_generated_notebooks/resnet_finetune_300.ipynb` to **actually train** and save the outputs.

The `pip-install` macro creates a dedicated dependency cell that checks availability when the notebook **runs**, not when it compiles. Available packages print *Requirement already installed in this environment*. Missing named packages can be installed through the notebook kernel’s Python interpreter. Python's standard-library `math` module requires no pip install. GPU-compatible PyTorch builds should normally be installed separately in advance; installing packages into a running kernel may require a restart.

**This experiment has not been trained for 300 epochs in CI.** It needs an image dataset and may require substantial compute. Stratified sample-level splitting is not group-aware; use a grouped split when images come from the same subjects, patients or sessions.

More capabilities, examples and package options are in the [technical guide](https://github.com/edujbarrios/lispyn/blob/main/docs/guide.md). Created by **Eduardo J. Barrios**. Apache-2.0 licensed; see [LICENSE](https://github.com/edujbarrios/lispyn/blob/main/LICENSE) and [NOTICE](https://github.com/edujbarrios/lispyn/blob/main/NOTICE).
