Metadata-Version: 2.4
Name: kasahare
Version: 0.1.0
Summary: A self-contained LLaMA-style inference engine with a Python binding.
Author: simple-llm
License: MIT
Keywords: llm,inference,transformer,llama,c++
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.20
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: requires-python

# simple-llm / kasahare

`kasahare` (which means "speak fast" in Twi) is the PyPI distribution name for simple-llm.
A small, self-contained C++17 inference engine for LLaMA-style transformer
language models.  No external dependencies beyond the C++ standard library
and a recent `g++` / `clang++`.  Designed to be read end-to-end in an
afternoon, then hacked on.

## What's in the box

| file | purpose |
|---|---|
| `include/simplellm/simd.h`        | AVX2 + FMA kernels (matmul, RMSNorm, SwiGLU, vec ops) |
| `include/simplellm/tensor.h`      | Lightweight matrix view + matmul / linear kernels |
| `include/simplellm/config.h`      | Static model description |
| `include/simplellm/tokenizer.h`   | Byte-level BPE tokenizer (no HF tokenizers dep) |
| `include/simplellm/sampler.h`     | Greedy / temperature / top-k / top-p sampler |
| `include/simplellm/model_data.h`  | On-disk model format reader / writer |
| `include/simplellm/model.h`       | Public surface: `Model::load`, `forward`, `generate` |
| `src/model.cpp`                   | LLaMA block: RMSNorm, GQA attention, RoPE, SwiGLU |
| `src/main.cpp`                    | `sl-llm` CLI |
| `tools/create_tiny_model.cpp`     | Synthesises a tiny model on disk for smoke testing |
| `tools/bench.cpp`                 | Micro-benchmark: `make bench` |
| `tests/test_basic.cpp`            | End-to-end smoke test: `make test` |

## Architecture

For each block (LLaMA-2 style):

```
x_n              = x_{n-1}                                  # residual
x_n              = attn_norm(x_n)
q, k, v          = Wq x_n, Wk x_n, Wv x_n                  # no biases
q, k             = RoPE(q, k, pos)
attn             = softmax(q . K^T / sqrt(d_head)) . V
x_n              = Wo(attn) + x_n                           # residual
x_n              = ffn_norm(x_n)
x_n              = W2( silu(W1 x_n) * (W3 x_n) ) + x_n      # SwiGLU, residual
```

Optimisations:

* `matmul` and `linear` are tiled over the inner dim with 8-wide AVX2
  + FMA.  Compiles to a single fused `vfmadd231ps` chain.
* `rmsnorm`, `vec_add`, `vec_mul`, `silu_mul` are SIMD-ised.
* A pre-allocated KV cache (`KvSlot` per head per layer) lets `forward()`
  be O(1) allocation cost.
* Tied input / output embeddings (LLaMA style) — one copy of the
  embedding matrix is read once per token.
* Decoder-time attention materialises only the per-step logits; we
  re-score every cached key on each new token.  For long contexts a
  Flash-Attention style kernel would beat this by ~2x, but the current
  implementation is dramatically simpler.

The block uses **GQA** (grouped query attention) so it accepts a config
where `n_kv_heads < n_heads` (or equal, for MHA, or 1, for MQA).

## Build

```sh
make            # builds build/bin/{sl-llm, create_tiny_model, bench}
make test       # builds and runs the smoke test
make bench      # runs the benchmark
make clean
```

Requirements: `g++` ≥ 9 (we use `g++ 12.2`), AVX2 + FMA support.  Override
the SIMD flags with `make CFLAGS="-O3 -march=native"` to autodetect, or
`make CFLAGS="-O3 -msse4.2"` to disable AVX2.

## Quick start

```sh
# 1) Generate a tiny model + tokenizer (~110 KB total)
./build/bin/create_tiny_model --out build

# 2) Run it
./build/bin/sl-llm \
    --model build/tiny.sllm \
    --tokenizer build/tiny.tok.txt \
    --prompt "hi" \
    --max-new 16 \
    --temperature 0.8

# 3) Show timing
./build/bin/sl-llm \
    --model build/tiny.sllm \
    --tokenizer build/tiny.tok.txt \
    --prompt "" \
    --max-new 64 --temperature 0 --timing
```

(The model produced by `create_tiny_model` has random weights; the output
is deterministic per `--seed` but not meaningful text.  Real text needs
real weights — see "Plugging in a real model" below.)

## CLI flags

```
--model <file.sllm>        model file
--tokenizer <file.txt>     tokenizer file
--prompt <text>            input text
--max-new <n>              number of tokens to generate (default 32)
--temperature <f>          sampling temperature, 0 = greedy (default 0.8)
--top-k <n>                top-k filtering (default 40)
--top-p <f>                top-p filtering (default 0.95)
--seed <n>                 PRNG seed (0 = random)
--show-config              print model config and exit
--no-print-prompt          don't echo the prompt before the generated tail
--timing                   print per-step timing to stderr
--stream                   flush stdout after each generated token
--help                     this message
```

## On-disk model format (`.sllm`)

```
+---------------------------------------------------+
| magic   (u32 LE, 0x534C4C4D == "SLLM")             |
| version (u32 LE, currently 1)                      |
| blen    (u32 LE)                                   |
| config blob  (blen bytes, key=value text)          |
| weights blob (u32 × N floats, little-endian)       |
+---------------------------------------------------+
```

The config blob is a human-readable key=value text.  Weights are stored
in a single contiguous float32 array in this order:

```
token_emb                (vocab_size * dim)
for each block:
    attn_norm            (dim)
    wq                   (dim * dim)
    wk                   (kv_dim * dim)
    wv                   (kv_dim * dim)
    wo                   (dim * dim)
    ffn_norm             (dim)
    w1                   (hidden * dim)   # SwiGLU gate
    w3                   (hidden * dim)   # SwiGLU up
    w2                   (dim * hidden)   # SwiGLU down
output_norm              (dim)
```

`tied_embeddings` reuses `token_emb` for the output projection (LLaMA
style).  The file is `mmap`-friendly, so you can load a model lazily in
a follow-up patch.

## Tokenizer format

A line-oriented text file:

```
# simple-llm tokenizer
vocab_size <N>
bos <id>
eos <id>
token <id> "<csv-escaped-payload>"
merge <rank> "<csv-escaped-left>" "<csv-escaped-right>"
```

Payloads are byte strings; CSV escape handles `"`, `\`, control chars
and arbitrary bytes via `\xHH`.  The BPE merge rank is determined by
file order (lower rank = higher priority).  A tokenizer with zero merges
falls back to a longest-match scan over the vocab, which is enough for
byte-level pre-tokenization.

## Performance

The numbers below are from `make bench` on a single CPU core (g++ 12.2,
AVX2 + FMA, FP32).  They are not competitive with llama.cpp — we
don't have quantisation, batching, GPU, threading, or FlashAttention —
but they show the hot path is reasonable.

| config | tokens/s |
|---|---|
| dim=256,  layers=4,  heads=4,  hidden=512,  seq=128  |   ~380 |
| dim=384,  layers=6,  heads=6,  hidden=1024, seq=256  |   ~60  |
| dim=512,  layers=8,  heads=8,  hidden=1536, seq=256  |   ~24  |

The decode loop is O(seq_len) per token, so a 2x longer context halves
the tok/s.  Most of the time is spent in the `linear()` calls for Q/K/V
projection and FFN; an OpenMP / threadpool pass over those would be the
biggest single win.

## Python binding

The engine is also a `pip install`-able Python package available on PyPI as `kasahare`:

```sh
pip install kasahare
```

Build it as editable for development, or build a wheel for distribution:


```sh
# Editable install (rebuilds on C++ changes — the binding .so lives in
# python/simplellm/_core*.so and is patched in by setuptools).
pip install -e .

# Wheel (good for shipping to a cluster with no compiler).
pip install build
make py-wheel              # produces dist/simplellm-0.1.0-*.whl
pip install dist/simplellm-0.1.0-*.whl
```

Run the Python tests:

```sh
make py-test
```

Then in Python:

```python
import numpy as np
import simplellm

m = simplellm.Model.load("tiny.sllm", "tiny.tok.txt")
ids  = m.tokenizer.encode("hello world", add_bos=True)
log  = m.forward(ids[0], reset_kv=True)         # -> np.ndarray (vocab_size,)
out  = m.generate(ids, max_new=32,
                  options=simplellm.SampleOptions(temperature=0.7, top_k=40))
print(m.tokenizer.decode(out))

# Streaming:
def on_token(tid): print(m.tokenizer.decode_token(tid), end="", flush=True)
m.reset_kv()
m.generate(ids, max_new=16, options=simplellm.SampleOptions(temperature=0.8), stream=on_token)

# In-memory models (no temp files, no paths):
m2 = simplellm.Model.from_bytes(open("tiny.sllm","rb").read(),
                                open("tiny.tok.txt","rb").read())
```

The binding is zero-copy on the C++ → Python transition for the
logits array: `forward()` returns a fresh `np.ndarray` of shape
`(vocab_size,)` that you can pass straight to `np.argmax`,
`torch.softmax`, etc.
projection and FFN; an OpenMP / threadpool pass over those would be the
biggest single win.

## Plugging in a real model

`tools/create_tiny_model.cpp` shows the on-disk layout.  To use a
trained model, the steps are:

1. Train or download a LLaMA-2 / Mistral / Qwen2 model.
2. Convert its HF / safetensors / GGUF weights to a single float32 buffer
   in the order listed above.
3. Write the config blob (a few `key=value` lines) with the matching
   dimensions.
4. Run `sl-llm --model ... --tokenizer ...`.

We do not ship a converter for a specific upstream format; the layout
is small enough that `numpy.tofile()` + a few header bytes does the
trick.  See `tools/create_tiny_model.cpp` for a working example of
writing a model from scratch.

## Limitations (intentional)

* FP32 only.  No FP16, BF16, INT8 or INT4.  Adding INT4 would 8x
  memory and ~3-4x decode throughput.
* Single-threaded.  No SIMD beyond AVX2/FMA.  No GPU.
* Decoder-only.  No training.
* No Flash-Attention — attention is O(seq_len) per token.
* No KV cache compression (quantised cache, sliding window).
* No beam search.  Sampling only (greedy / top-k / top-p / temperature).
* The tokenizer is byte-level BPE, not identical to any specific
  upstream BPE implementation — the merge-ranking and pre-tokenisation
  are simplified.  A trained model from a specific corpus may need its
  own tokeniser file to round-trip correctly.
* The on-disk format is not portable across endiannesses; this is fine
  for x86 / ARM servers but not for big-endian targets.

## Where to go from here

* Add a `mmap`-based weight loader so very large models don't need
  to live in RAM twice.
* Add INT8 / INT4 weight loading (block-quantised, like GGUF Q4_K).
* Add OpenMP parallelism to `matmul` (per-row across `M`).
* Replace the attention loop with a single fused kernel.
* Add a `Stream` API that yields tokens through a callback (handy for
  chat UIs).
* Add a HuggingFace `tokenizer.json` reader so trained tokenizers work
  out of the box.

## License

Do whatever you want with it.  No warranty — see "Limitations" above.
