Metadata-Version: 2.4
Name: scriptweave
Version: 2026.0.0
Summary: Parsers that turn MATLAB publish XML and jupytext percent-format Python sources into a uniform cell-based model for downstream document generation.
Keywords: matlab,jupyter,jupytext,publish,parser
Author: Gustaf Hendeby
Author-email: Gustaf Hendeby <gustaf.hendeby@liu.se>
License-Expression: MIT
License-File: LICENSE.md
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Requires-Dist: defusedxml>=0.7.1
Requires-Dist: nbformat>=5.10
Requires-Dist: nbclient>=0.10
Requires-Dist: ipykernel>=6.29
Requires-Dist: markdown>=3.6
Requires-Python: >=3.13
Description-Content-Type: text/markdown

This package is used to parse the xml output of MATLAB publish, and
this way facilitate deriving more advanced documentation that contain
MATLAB code with matching output and figures.

A sibling submodule, `scriptweave.jupyter`, provides the same
extraction model for Python notebooks expressed in the
[jupytext](https://jupytext.readthedocs.io/) **percent** format. Unlike the
MATLAB flow, the Python parser **executes** the source as part of `parse()`
using a Jupyter kernel and captures stdout, image outputs, and rich results.

Both backends now map into a shared normalized representation defined in
`scriptweave.model`:

- `ParsedDocument` for file-level metadata and ordered cells
- `Cell` for per-cell content, text rendering, outputs, and images

Backend-specific values are surfaced through shared metadata fields on the
document (`matlab_release`, `matlab_version`, `kernel_name`,
`python_version`) and through `backend` / `execution_mode`.

## Migration notes (old -> new assumptions)

If you consumed parser results before the shared model change, these are the
relevant adjustments:

- `scriptweave.matlab.parse(...)` and `scriptweave.jupyter.parse(...)`
  still exist and remain the primary entry points.
- Returned objects now share the same base shape (`ParsedDocument` and
  `Cell`) even though backend-specific names (`MatlabFile`, `PythonFile`) are
  still exported for compatibility.
- Prefer capability-style checks over backend class checks:
  - use `doc.backend` and `doc.execution_mode`
  - instead of `isinstance(doc, MatlabFile)` / `isinstance(doc, PythonFile)`
- Backend metadata moved to common accessors:
  - MATLAB: `doc.matlab_release`, `doc.matlab_version`
  - Jupyter: `doc.kernel_name`, `doc.python_version`
  - all are also available in `doc.metadata`
- Cell payload access is aligned:
  - use `cell.source` for canonical cell input text
  - `cell.code` is still available for code cells and remains a compatibility
    alias
  - use `cell.text_as_str()` and `cell.text_as_latex()` for rendered text
    across both backends

In short: keep your existing parse entry points, but treat outputs as one
normalized document format with backend metadata attached.

# Usage

## MATLAB processing

Write the code that you want to create output from as a MATLAB publish
document.  The package will parse the code, and separate the different
cells.  The package will separate documentation, executed code, console
output and figures.

In MATLAB process the created file (for example ``cubic_plot_publish.m``) using publish:
```
publish('cubic_plot_publish.m',...
        'format', 'xml',...
        'imageFormat', 'epsc',...
        'outputDir', 'gen/',...
        'catchError', false);  % Fail on thrown exception

```
This will produce the file ``gen/cubic_plot_publish.xml``

## Python Processing
Usage:

```
from pathlib import Path

from scriptweave.matlab import parse

mf = parse(Path('cubic_plot_publish.xml'))

print(mf.filename)  # The name of the processed file
print(mf.output_dir)  # Directory for generated files
for c in mf.cells:
  print(c.title)  # Title of cell
  print(c.code)  # Executed code in cell
  print(c.output)  # Console output from cell
  ...

print(mf.backend)  # "matlab"
print(mf.execution_mode)  # "artifact"

```

Explore the object to figure out what has been extracted.

Current fixture examples in `tests/data/` are:
`cubic_plot_publish.m`, `formatted_text_publish.m`, `text_only_publish.m`,
`single_figure_publish.m`, and `multi_figure_publish.m`.

## HTML → LaTeX conversion

`Cell.text_as_latex()` converts the small HTML fragment that MATLAB
`publish` emits inside each `<text>` element into LaTeX.  The conversion is
performed by a tiny in-tree module,
`scriptweave.html_to_latex`, rather than a third-party
library.

**Rationale.**  Earlier versions depended on the unmaintained
`html2latex` package, installed straight from a personal GitHub fork.
The package is not on PyPI, has not tracked changes in its own
dependencies (notably `justhtml`), and pulls in a heavy HTML-parsing
stack to handle constructs MATLAB publish never produces.  General
HTML-to-LaTeX libraries (e.g. routing through Pandoc via `pypandoc`)
were considered, but they add either a non-Python runtime dependency
or substantially more surface area than this project needs.

The MATLAB-publish HTML subset is narrow and stable — paragraphs, basic
inline emphasis (`<b>`, `<i>`, `<tt>`), simple lists, links, line
breaks, preformatted/verbatim blocks, inline and display LaTeX
equations, external images, raw LaTeX blocks, and (dropped) raw HTML
blocks — so a small purpose-built converter is both easier to test and
easier to keep working.  The set of tags handled was derived directly
from real MATLAB `publish('...','format','xml')` output (see
`tests/data/all_markup_publish.m`).

**MATLAB-specific notes.**

* `<equation>` is emitted by `publish` with the original LaTeX source
  preserved on either the `<equation text="$$...$$">` attribute (display
  equations) or on the inner `<img alt="$...$">` (inline equations).
  The converter prefers those over re-deriving math from rendered
  pixels.
* `<latex>` blocks contain raw LaTeX (e.g. `\begin{tabular}...`) and are
  emitted **verbatim**, not wrapped in math delimiters.
* `<html>` blocks have no faithful LaTeX equivalent and are dropped
  with a `UserWarning`.
* `<mcode-xmlized>` (sample code with a three-space comment indent) is
  rendered as a `\begin{verbatim}...\end{verbatim}` block, stripping
  the namespaced `mwsh:*` syntax-highlight tags.
* MATLAB emits the trademark symbols `®` / `™` as literal Unicode, which
  passes through the converter unchanged.

**Extending the converter.**  The converter dispatches each HTML tag
through a registry of handlers.  To support a new tag without forking
the module, register a handler on the default renderer:

```python
from scriptweave.html_to_latex import HtmlToLatex, register_tag

def _underline(element, renderer: HtmlToLatex) -> str:
    return f"\\underline{{{renderer.render_element(element)}}}"

register_tag("u", _underline)
```

A handler receives the `xml.etree.ElementTree.Element` it is rendering
and the active renderer (so it can recurse via
`renderer.render_element` / `renderer.render_children`).  For
private rendering pipelines that should not mutate the module-level
default, instantiate `HtmlToLatex()` directly and call its
`register` / `convert` methods.

Unknown tags do **not** raise: they render their children transparently
and emit a `UserWarning`.  This keeps existing pipelines working when
MATLAB introduces new constructs, while still surfacing them so a
handler can be added.
## Jupyter / jupytext processing

For Python sources, write your script in jupytext percent format. A cell
begins with `# %%`; markdown cells use `# %% [markdown]` and prefix each
content line with `# `. Optional cell titles follow the `# %%` marker:

```python
# %% [markdown]
# # Cubic plot example
#
# A short description of the experiment.

# %% cubic_plot
%matplotlib inline
import matplotlib.pyplot as plt

xs = list(range(11))
ys = [x ** 3 for x in xs]
print(f"x_max={xs[-1]} y_max={ys[-1]}")

fig, ax = plt.subplots()
ax.plot(xs, ys)
plt.show()
```

Parse and execute the file:

```python
from pathlib import Path

from scriptweave.jupyter import parse

pf = parse(Path("cubic_plot_publish.py"))

print(pf.filename)        # Source file name
print(pf.output_dir)      # Where extracted images were written (defaults to ./gen)
print(pf.kernel_name)     # "python3"
print(pf.python_version)  # Reported by the kernel
print(pf.backend)         # "jupyter"
print(pf.execution_mode)  # "executed"

for cell in pf.cells:
    print(cell.id, cell.cell_type, cell.title)
    if cell.cell_type == "markdown":
        print(cell.text)              # Raw markdown source
        print(cell.text_as_str())     # Rendered HTML
        print(cell.text_as_latex())   # Rendered LaTeX (via scriptweave.html_to_latex)
    else:
        print(cell.code)              # Source code
        print(cell.output)            # Captured stdout / stream text
        print(cell.results)           # text/plain values from execute_result
        print(cell.images)            # Saved PNG / SVG file names in output_dir
```

`parse()` accepts an explicit `output_dir`, a `kernel_name` (defaults to
`python3`), a per-cell `timeout`, and an `execution_policy`:

- `execution_policy="execute"` (default): execute code cells and raise
  `ParseError` on execution failure.
- `execution_policy="skip"`: parse structure only; code cells keep source but
  have no runtime outputs/results/images.
- `execution_policy="best_effort"`: execute all code cells, continuing past
  failing cells. Per-cell failures are recorded on each cell's `error` field;
  kernel-level failures (e.g. startup or timeout) are stored in
  `pf.metadata["execution_error"]` while preserving any partial outputs. If
  anything failed, `execution_mode` is `"best_effort"`, otherwise
  `"executed"`.

For non-skip execution policies, parser metadata also includes
`pf.metadata["execution_diagnostics"]` with summary fields: total code cells,
code cells with outputs, code cells with errors, and the first failed cell
index (or `None`).

The parser suppresses kernel-process stderr during startup so benign
`IPKernelApp` warnings from the local Jupyter runtime do not leak into caller
output.

## Logging

The library uses Python's standard `logging` module and emits structured
messages from both parser backends:

- `scriptweave.matlab.parser`: XML parse lifecycle and parse failures.
- `scriptweave.jupyter.parser`: source parsing, execution-policy decisions,
  notebook execution outcomes, and parse summaries.

By default, the package installs a `NullHandler`, so importing it does not
configure global logging or emit messages unless your application enables them.

Example:

```python
import logging

from scriptweave.jupyter import parse

logging.basicConfig(level=logging.INFO)

document = parse("example_publish.py")
```

Set the level to `DEBUG` to see detailed execution diagnostics.

Recommended logger names and levels:

- `scriptweave`: set to `INFO` for high-level lifecycle messages.
- `scriptweave.matlab.parser`: set to `DEBUG` to troubleshoot MATLAB XML parsing.
- `scriptweave.jupyter.parser`: set to `DEBUG` to troubleshoot notebook execution and policy handling.

Example with per-logger levels:

```python
import logging

logging.basicConfig(level=logging.WARNING)

logging.getLogger("scriptweave").setLevel(logging.INFO)
logging.getLogger("scriptweave.jupyter.parser").setLevel(logging.DEBUG)

# logging.getLogger("scriptweave.matlab.parser").setLevel(logging.DEBUG)
```

Current Jupyter fixture examples in `tests/data_jupyter/` are:
`cubic_plot_publish.py`, `formatted_text_publish.py`, and
`text_only_publish.py`.

## Testing

Run the test suite with:

```bash
pytest
```

Optionally run tests with coverage reporting:

```bash
pytest --cov=scriptweave --cov-branch --cov-report=term-missing
```

When you change a MATLAB source fixture in `tests/data/*.m`, regenerate the
matching published XML fixtures in `tests/data/gen/*.xml` by running
`tests/data/runme.m` in MATLAB before running the tests again. The suite checks
both that every source fixture has a matching XML file and that the XML
`originalCode` still matches the current MATLAB source.

Output rendering expectations are stored as snapshots in `tests/snapshots/`.
This includes per-cell rendering snapshots and full parsed-file snapshots for
all published examples. When output formatting intentionally changes, refresh
snapshots with:

```bash
pytest --update-output-snapshots
```

When adding a new example fixture:

```text
1. Add the `.m` file to `tests/data/`
2. Add it to `tests/data/runme.m`
3. Generate the matching XML in `tests/data/gen/`
4. Run `pytest --update-output-snapshots` if the parsed output changed intentionally
```

When adding a new Jupyter fixture:

```text
1. Add the `.py` percent-format file to `tests/data_jupyter/`
2. Run `pytest --update-output-snapshots` to capture the executed snapshot
```

The Jupyter tests require a working `python3` Jupyter kernel
(`ipykernel`) and the runtime dependencies `nbclient`, `nbformat`, and
`markdown`. `matplotlib` is pulled in as a dev dependency so that the
figure-producing fixtures can execute end-to-end.

## License

This project is licensed under the MIT License. See [LICENSE.md](LICENSE.md)
for the full text.
