Metadata-Version: 2.4
Name: zrun-py
Version: 0.1.0
Summary: Execution for languages built with zgram and zrules: semantics written as Python functions, compiled to native code.
Author: Daniele Linguaglossa
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Software Development :: Compilers
Classifier: Topic :: Software Development :: Interpreters
Requires-Python: >=3.10
Requires-Dist: zgram-py>=0.4.2
Requires-Dist: zrules-py>=0.1.6
Description-Content-Type: text/markdown

<div align="center">

<img src="https://raw.githubusercontent.com/dzonerzy/zrun/main/docs/assets/logo.svg" alt="zrun Logo" width="150">

# zrun

**Execution for languages built with [zgram](https://github.com/dzonerzy/zgram) and [zrules](https://github.com/dzonerzy/zrules): write an interpreter in Python, run programs as native code.**

You say what each kind of node *does*, as ordinary Python functions: an interpreter, the most natural way to define a language. zrun runs those functions, and it also compiles them: because a program's tree is known before it runs, zrun specializes your interpreter for that program and hands the result to LLVM. `node.op == "+"` is decided while compiling, evaluating a child becomes the child's own code, and with zrules' types an `a + b` of two ints becomes one machine add.

[![GitHub Stars](https://img.shields.io/github/stars/dzonerzy/zrun?style=flat)](https://github.com/dzonerzy/zrun)
[![Python](https://img.shields.io/badge/python-3.10+-blue)](https://www.python.org/)
[![Zig](https://img.shields.io/badge/zig-0.16+-orange)](https://ziglang.org/)
[![License](https://img.shields.io/badge/license-MIT-green)](https://github.com/dzonerzy/zrun/blob/main/LICENSE)

Built with [PyOZ](https://github.com/pyozig/PyOZ)

</div>

---

- **Semantics in Python, programs native.** The functions you write are compiled with each program (partial evaluation); a semantic outside the compilable subset runs as Python, and zrun tells you which and why.
- **The same behavior in every mode.** A program runs with its semantics as Python (the reference) or compiled; output, errors, error locations and call stacks are the same.
- **Scripts and engines.** Run a program from its start, or load it once and call its functions millions of times: from several Python threads, or on native threads with `map()`, over data passed without copies.
- **Ahead of time.** Save a program's compiled code as a module and load it in another process without compiling.
- **One LLVM.** zrun builds its code in memory through the LLVM zgram already ships, through zgram's capsule: no second copy, nothing to install.

Part of zsuite: zgram (syntax), zrules (semantics), zrun (execution), zlsp (editor support).

## Performance

### Typed code against C

[examples/typed](https://github.com/dzonerzy/zrun/tree/main/examples/typed), a statically typed language whose types zrules works out, against the same code in C (`gcc -O2`, with the same overflow checks, since zrun's ints raise on overflow):

| | zrun | C |
|---|---|---|
| `fib(30)`, recursive | 0.0043 s | 0.0022 s |
| summing a 1000-item list, `len()` in the loop's condition | 0.27 ns an item | 0.22 ns an item |

### A real language

[examples/lua](https://github.com/dzonerzy/zrun/tree/main/examples/lua) is Lua 5.4: a grammar in zgram, its static rules in zrules, and about 2,300 lines of semantics in Python (values, metatables, closures, varargs, the string, table and math libraries, `pcall`). It passes its test programs in both modes. Compiled against the same semantics run as Python, and against Lua's own interpreter (PUC-Rio's, written in C):

| Program | zrun, compiled | zrun, semantics as Python | Lua 5.4 |
|---|---|---|---|
| `fib(30)` | 0.068 s | 14.1 s | 0.026 s |
| a numeric loop, 10 million times | 0.125 s | 20.7 s | 0.021 s |
| binary trees, depth 14, 8 times | 0.099 s | 3.3 s | 0.021 s |
| building a string of 200,000 numbers | 0.105 s | 1.1 s | 0.031 s |

Lua is dynamically typed and its semantics here are generic Python; zrun compiles them to 11-209x the speed of running them, within 2.6-6x of a hand-written C interpreter.

### An engine

[examples/scan](https://github.com/dzonerzy/zrun/tree/main/examples/scan) is a small YARA-like rule language: conditions over a file's bytes, two native host functions (`count`, `entropy`) called without Python. Loaded once, called for 6,000 memory-mapped files (467 MB, page cache warm, files already mapped) with `program.map()`:

| Threads | Time |
|---|---|
| 1 | 0.159 s |
| 8 | 0.023 s |

No call takes the GIL. Every way of running it (threads, `map()`, the reference mode) gives the same results.

### Compiling

Compiled code is kept between runs (on disk, by what it compiles). A program run compiled whose optimized code isn't there yet is compiled fast first and starts at once, while the optimized code is made on other threads; later runs use it. `examples/lua/tests/programs.lua` with an empty cache: first run 3.3 s (26 s optimized at once); once its optimized code is all cached, 0.75 s for a new process's first run and 0.009 s a run after that.

## Installation

```bash
pip install zrun-py
```

Like every zsuite package, it is named `<name>-py` on PyPI; the module is `zrun`. It installs `zgram-py` 0.4.0 and `zrules-py` 0.1.6 or newer with it. Prebuilt wheels cover CPython 3.10+ on **x86_64 Linux** and **x86_64 Windows**.

### From source

Requires [Zig](https://ziglang.org/) 0.16 and the zgram and zrules packages.

```bash
pip install pyoz
pyoz build --release     # builds the wheel into dist/
pip install dist/*.whl
python -m pytest test
```

`zig build -Doptimize=ReleaseFast` alone produces `zig-out/lib/zrun.so` for quick iteration.

## Quick start

A language is a zgram parser, zrules rules (for its names and scopes), and semantics:

```python
import builtins
import zrun

lang = zrun.Language(PARSER, RULES)      # a zgram parser, a zrules Rules
lang.function("FuncDef")                 # `fn name(params) { body }` defines functions

@lang.exec("While")
def while_(node, rt):
    while rt.eval(node.cond):
        if not rt.loop(node.body):       # False: the body broke out
            break

@lang.exec(["Let", "Assign"])
def assign(node, rt):
    rt.store(node.name, rt.eval(node.value))

@lang.exec("Return")
def return_(node, rt):
    raise rt.Return(rt.eval(node.value))

@lang.eval("BinOp")
def binop(node, rt):
    a, b = rt.eval(node.left), rt.eval(node.right)
    if node.op == "+":
        return a + b
    if node.op == "<":
        return a < b
    ...

@lang.eval("Call")
def call(node, rt):
    return rt.call(rt.eval(node.name), rt.eval(node.args))

@lang.host                               # plain Python: anything goes
def print(*args):
    builtins.print(*args)

program = lang.load(source, "fib.tiny")  # parse, check: zrun.LoadError lists the errors
program.run()                            # semantics as Python: the reference
program.run(mode="compiled")             # native code
```

[examples/tiny](https://github.com/dzonerzy/zrun/blob/main/examples/tiny/tiny.py) is this language, complete in 150 lines. Node fields are the grammar's labels (`node.cond`, `node.body`); variables are zrules' symbols, so `rt.load(name)` and `rt.store(name, v)` are slot accesses, not lookups.

## Writing semantics

- `@lang.eval(kind)`: an expression's value, `fn(node, rt) -> value`. `kind` is a node kind (a zgram `->` class or rule name) or a list of them.
- `@lang.exec(kind)`: a statement, `fn(node, rt)`.
- `lang.function(kind, params="params", body="body", name="name", hoist=True, missing="error", extra="error")`: nodes of `kind` define functions; zrun makes, calls and returns from them, frames and closures included.
- `@lang.host` / `@lang.host("name")`: a Python function the program calls through the builtin of that name.
- `lang.native_host(name, capsule)`: a function of a native library (a `zrun.native.v1` capsule), called by compiled code directly.

What `rt` offers:

| | |
|---|---|
| `rt.eval(x)`, `rt.exec(x)` | run a child node (or each of a list) |
| `rt.load(name)`, `rt.store(name, v)` | the variable a name node refers to |
| `rt.function(node)`, `rt.call(f, args, receiver=None)` | make and call functions (the program's or a host function) |
| `rt.loop(body)` | run a loop's body once: `False` if it broke out |
| `raise rt.Return(v)`, `rt.Break()`, `rt.Continue()` | control flow |
| `raise rt.Throw(value)`, `rt.error(node, message)` | the language's errors |
| `rt.receiver`, `rt.varargs`, `rt.context`, `rt.path` | the method's object, extra arguments, the host's context, the program's name |
| `rt.scope(node)`, `rt.symbol(node)`, `rt.type_of(node)`, `rt.fresh(node)` | zrules' view of names and types; new loop variables each time round |
| `rt.u8(data, i)` ... `rt.i64be(data, i)` | integers read from data, bounds checked |
| `rt.wrapping_add(a, b)`, `_sub`, `_mul`, `_shl`, `_shr`, `_ushr` | 64-bit arithmetic that wraps around |

Integers are 64-bit and checked: an overflow is a runtime error at the node (`zrun.IntegerOverflow`), not a silent wraparound; the `wrapping_` functions are for languages that want one (Lua, hashes).

Semantics are compiled from their Python source. What compiles, and how to write semantics that compile to fast code, is in [the guide to writing fast semantics](https://github.com/dzonerzy/zrun/blob/main/docs/writing-fast-semantics.md); `lang.python_semantics()` lists the ones that run as Python, and why.

## Modes

`program.run(mode=...)`:

- `"python"` (the default): semantics run as Python. The reference: compiled code must behave exactly like it.
- `"compiled"`: native code. Compiled fast the first time if the optimized code isn't cached; the optimized code is made in the background and runs from the run after it's ready (`zrun.configure(tiers=False)`: optimized at once).
- `"auto"`: as Python until the optimized code is ready, then compiled.

`program.report()` says where compiled code still went through Python and why, which state of the semantics' modules stayed Python, what the cache did, and which functions got typed entries.

## Engines

A program can be a set of entry points, loaded once and called many times:

```python
rules = lang.load(source)
rules.call("match", data, context=scan)                 # ~0.15 us a call from Python
rules.map("match", [(f,) for f in files], threads=8)    # native threads, no GIL
```

- **Calls** compile the program once and release the GIL while they run, taking it back only to touch Python; `report()["gil_taken"]` counts those times.
- **`map()`** runs the calls on native threads without returning to Python between items; the first failing item's error is raised as one thread calling them in order would raise it.
- **Data** (`bytes`, `bytearray`, `memoryview`, `mmap`) reaches the language as `zrun.Bytes`, read in place: slices share the memory.
- **Context**: `rt.context` is what the caller passed, for results and options.

## Types

With a zrules `types()` rule, say what values each type has:

```python
lang.types({"int": int, "float": float, "bool": bool, "str": str, "list": list})
```

A value is then checked against its node's declared type once, where it is made, and compiled code knows its kind from there: ints and floats unboxed, calls between typed functions with plain arguments in registers. A value that isn't what its type says is a runtime error at the node, never a wrong result.

## Ahead of time

```python
program = lang.compile(source, "rules.zrc")   # or program.save(path), after running it
program = lang.load_compiled("rules.zrc")     # another process: nothing compiled
```

A compiled module holds the program's source and its compiled code. Loading it checks a hash of the language's definition (grammar, semantics, host functions): a module made for another version of the language, another zrun or another CPU is refused, not misrun.

The compiled-code cache is in the platform's cache directory (`%LOCALAPPDATA%\zrun\Cache`, `~/Library/Caches/zrun`, `$XDG_CACHE_HOME/zrun` or `~/.cache/zrun`); `zrun.configure(cache=False)` or `cache="dir"` changes that.

## Sessions and the REPL

```python
lang.repl()                       # reads entries with input(), shows expression values
s = lang.session()
s.run("let x = 1;")
s.run("fn inc(n) { x = x + n; return x; }")
s.run("inc(5);")                  # 6: entries share the variables earlier ones defined
```

`python examples/lua/lua.py` with no script is a Lua REPL.

## Errors

A runtime error raises `zrun.Error`: `e.diagnostic` is a `zgram.Diagnostic` at the failing node, `e.stack` the language's calls that led there, and `str(e)` renders them:

```
fib.tiny:3:12: error: integer overflow [runtime]
    3 |     return fib(n - 1) + fib(n - 2);
      |            ^^^^^^^^^^^^^^^^^^^^^^^
  in fib(), called at fib.tiny:5:7
```

A Python exception a semantic raises (a `ZeroDivisionError`, a `KeyError`) is the program's runtime error at the node being run, worded as Python words it ("division by zero"); a host function's failure is worded `name: Type: message`. Errors found before running (syntax, the rules) raise `zrun.LoadError` with all the diagnostics.

## API Reference

Every class and method has its documentation in `help()` (and in the `.pyi` stubs the wheel ships).

| | |
|---|---|
| `Language(parser, rules=None, max_depth=1000)` | a language: a zgram parser, zrules rules |
| `@lang.eval(kind)`, `@lang.exec(kind)` | the semantics of a node kind (`native=False`: run as Python) |
| `lang.function(kind, ...)`, `@lang.host`, `lang.native_host(name, capsule)` | functions of the language, of Python, of a native library |
| `lang.types(mapping)` | what values the language's types have |
| `lang.load(source, path=None)` | parse and check a program: a `Program` |
| `program.run(mode=..., report=False)` | run it from its start |
| `program.call(name, *args, context=None)`, `program.map(name, items, threads=None)` | an engine's entry points |
| `program.report()`, `lang.python_semantics()` | what to look at to make it faster |
| `lang.compile(source, output)`, `program.save(path)`, `lang.load_compiled(path)` | compiled modules |
| `lang.session()`, `lang.repl()` | programs run one after another, sharing their names |
| `zrun.Bytes(data)` | data read in place |
| `zrun.configure(cache=, tiers=, perf_map=)` | process-wide settings |
| `zrun.collect()` | the cycle collector, now |
| `zrun.Error`, `zrun.LoadError`, `zrun.CompileError`, `zrun.IntegerOverflow` | errors |

## Architecture

```
zgram Tree + zrules Analysis (capsules, read in place)
     |
     v
[Program data]     -- nodes, fields, symbols, scopes as native arrays (program.zig)
     |
     v
[Front]            -- each semantic's Python source, read once (front.zig)
     |
     v
[Compiler]         -- the semantics run over the tree while compiling: what's known is
     |                decided, what isn't becomes LLVM IR (compile.zig, ir.zig)
     v
[LLVM]             -- zgram's, through its capsule: optimized, kept in the cache, loaded
     |                into its JIT (driver.zig, cache.zig, jit.zig)
     v
[Runtime]          -- tagged values, lists, dicts, records, frames; reference counts and
                      a cycle collector; Python's objects at the boundary (value.zig,
                      helpers.zig, gc.zig, proxies.zig, bridge.zig)
```

Key implementation details:

- **Partial evaluation**: compiling runs the semantics with the program's tree known. A node's fields, its kind, its operator text, a module's constant tables, a function's definition are values the compiler has; the code it emits is what remains. Parts it can't decide while compiling (a node evaluated only by a semantic run as Python) are compiled when first asked for, on their own.
- **Values**: 16-byte tagged words. Ints are 64-bit and checked; ints, floats and bools are never allocated. Lists, dicts and records are native and shared with Python through proxies when Python sees them, not copied.
- **Lists' elements kinds**: a list knows when all its items are ints or floats, so compiled code reads an item with its kind known.
- **Speculation**: a hot function gets a typed entry for the kinds of arguments it's called with (ints, floats, bools, lists), guarded where it's entered; calls from code that knows those kinds go to it directly, its arguments in registers.
- **Calls**: the program's functions are called through their code, the call stack kept as the reference mode's for errors; calls of host functions known while compiling jump to their compiled code.
- **Memory**: reference counting, with a cycle collector of CPython's design (generations, counted as containers are made). Functions read from the program's top level aren't counted on calls; one rebound while code runs is kept until the code returns.
- **Tiers**: a program whose optimized code isn't cached runs code LLVM compiled without optimizing, while the optimized code is made on other threads.

## Threads

- Compiled runs and calls release the GIL; code takes it back only to touch Python (host functions, semantics run as Python, Python values), and `report()["gil_taken"]` counts those times.
- `program.map()` runs calls on native threads; the program's variables and its module state are made immortal first, so the calls share them without counting. Calls mustn't change them.
- Each thread allocates values from lists of its own; LLVM's work in the background runs on worker threads, without the GIL.

## Known Issues

- Executables (a program with zrun's runtime, without Python) are planned, not built.
- x86_64 only: the LLVM zgram ships targets x86-64. No macOS wheels yet.
- Untyped code (Lua here) is 2.6-6x slower than a hand-written C interpreter; a program's top level isn't specialized as functions are.
- What compiles is a subset of Python (see the guide); the rest runs as Python, correctly but slower.
- The compiled-code cache has no size limit yet.
- On Windows with Python 3.12 or 3.13, `mode="python"` stops at a few hundred nested calls of the language's functions with "call stack too deep", before `max_depth`. Those Pythons allow 3000 calls nested through C on Windows (10000 elsewhere), and a call run as Python is several. Compiled code and other Pythons go to `max_depth`.

## Project Structure

```
src/
  lib.zig               # Python module: Language, Program, Runtime (rt), Session
  program.zig           # A loaded program's native data: tree, fields, symbols, scopes
  tree.zig              # zgram's parse tree as native code sees it (zgram.tree.v1)
  zrules_abi.zig        # zrules' analysis as native code sees it
  grammar.zig           # What zrun knows of a grammar: actions, kinds, labels
  front.zig             # Semantics read from their Python source
  compile.zig           # The compiler: semantics partially evaluated into LLVM IR
  ir.zig                # LLVM IR built in memory, through LLVM's C API
  jit.zig               # zgram's LLVM, through its zgram.llvm.v1 capsule
  driver.zig            # Compiled programs: IR to code, tiers, background compiling
  cache.zig             # Compiled code kept between runs
  aot.zig               # Compiled modules
  value.zig             # Values of compiled code and their runtime
  helpers.zig           # What compiled code calls: the execution context, the runtime
  bridge.zig            # Semantics run as Python inside compiled code
  proxies.zig           # Compiled lists, dicts and records seen from Python
  objects.zig           # Nodes, frames and functions of the reference mode
  types.zig             # zrun.I64, the errors and control-flow exceptions
  gc.zig                # The cycle collector
  pool.zig              # The values' allocator, per thread
  gil.zig               # Compiled code without the GIL
  adopt.zig             # Module state of the semantics made native
  bytes.zig             # zrun.Bytes: data read in place
  native.zig            # Native host functions (zrun.native.v1)
  wrapping.zig          # rt.wrapping_add and the other 64-bit wrapping operations
  pyhelp.zig            # Helpers over the Python C API
test/
  test_modes.py         # Differential tests: every program in every mode
  test_run.py           # Running programs: semantics as Python
  test_front.py         # Semantics read from their Python source
  test_typed.py         # The typed language: structs, methods, typed entries, errors
  test_lua.py           # Lua 5.4's test programs, like real Lua, in every mode
  test_engine.py        # program.call, map(), zrun.Bytes, native host functions
  test_scan.py          # A YARA-like engine: threads, map(), compiled modules
  test_boundary.py      # Values crossing between compiled code and Python
  test_state.py         # Module state the semantics change, made native
  test_scopes.py        # Block scopes and closures
  test_gc.py            # The cycle collector, leaks, functions rebound while called
  test_cache.py         # The compiled-code cache between processes
  test_tiers.py         # Fast code first, optimized in the background
  test_aot.py           # Compiled modules
  test_session.py       # Sessions and the REPL
examples/tiny/          # A small language: the quick start
examples/typed/         # A typed language: structs, methods, lists, optionals
examples/lua/           # Lua 5.4, with its test programs; a REPL
examples/scan/          # A YARA-like rule language: an engine, native host functions
docs/                   # The guide to writing fast semantics
build.zig               # Zig build configuration
pyproject.toml          # Python package configuration
```

## License

MIT
