Metadata-Version: 2.4
Name: inferris-lite
Version: 0.1.0
Summary: Executable mini-forms of LLM serving mechanisms - continuous batching, paged state, prefix caching, speculative decoding, the executor boundary, rank parallelism - as one dependency-free state machine with step-by-step traces
Project-URL: Homepage, https://github.com/gavinkvx/inferris
Keywords: llm,inference,serving,continuous-batching,paged-attention,prefix-caching,speculative-decoding,education
Requires-Python: >=3.12
Description-Content-Type: text/markdown

<!-- ./lite/py-lite/README.md -->

# inferris-lite

Executable mini-forms of LLM serving mechanisms — continuous batching,
token budgets, chunked prefill, paged state, preemption, prefix caching,
cache-aware scheduling, speculative decoding, the executor boundary, and
rank parallelism — as one dependency-free Python state machine that prints
what it is thinking, step by step.

Each mechanism is a mechanism note first (the problem, the invariant, the
smallest state machine that expresses it) and an implementation second.
The code grows mechanism by mechanism: `m00_engine_loop` through
`m11_parallelism`, each module the complete engine as of its mechanism,
each pinned by one test per line of its note. Reading order is the module
order; `import m11_parallelism` is the whole ladder.

## Install

```text
pip install inferris-lite
```

Python >= 3.12, zero runtime dependencies.

## Watch it think

```text
inferris-lite demo
```

Three requests — staggered arrivals, chunked prefill, page contention,
a scripted speculative verdict — on a rank-4 executor group:

```text
step   0  scheduled=['r0']  executed={'r0': 6}
step   1  scheduled=['r0', 'r1']  executed={'r0': 3, 'r1': 4}  accepted={'r0': 2}  finished={'r0': 'LENGTH'}
step   2  scheduled=['r1', 'r2']  executed={'r1': 2, 'r2': 6}  accepted={'r1': 1}  finished={'r1': 'LENGTH'}
...
r2: FINISHED (LENGTH)  generated=5  computed=14
```

`inferris-lite mechs` lists every mechanism and its key question. Your own
workloads go through `run`:

```text
inferris-lite run --request a:16:8 --request b:16:8:1 --spec-len 3 --world-size 2
inferris-lite run --spec workload.json --json
```

A JSON spec is an `engine` object (`max_step_tokens`, `page_size`,
`num_pages`, `spec_len`, `world_size`) plus a `requests` list
(`request_id`, `prompt_len` or explicit `tokens`, `max_new_tokens`,
`arrival_step`, `eos_after`, `accept_script`). Explicit shared `tokens`
prefixes are how you make the prefix cache light up in the `reused`
column. `--json` emits one object per step with every trace field, then a
terminal summary.

As a library:

```python
import m11_parallelism as m11

requests = [m11.Request("r0", list(range(6)), 4)]
engine = m11.Engine(requests, max_step_tokens=8, page_size=4, num_pages=8, world_size=2)
for trace in engine.run():
    print(trace.step, trace.scheduled, trace.executed, trace.accepted)
```

Every earlier mechanism is importable the same way — `m03_paged_state`
before pages met prefix reuse, `m09_speculative_decode` before the
executor boundary was cut — so you can diff neighbouring mechanisms to
see exactly what one mechanism adds.

## What this is, and is not

Tokens are integers. Execution is bookkeeping: the executor answers with
verdicts and page writes, not logits. Ranks are logical and run in
lockstep. There is no model, no tokenizer, no server — and that is the
point: what runs here is the mechanism semantics themselves, isolated
from implementation accident and performance optimization, small enough
to read in an afternoon and precise enough to break when you change an
invariant.

inferris-lite is the specification layer of
[inferris](https://github.com/gavinkvx/inferris), a Rust inference
engine; the mechanism notes live in the repo under `lite/mech/`.
