Metadata-Version: 2.4
Name: kasahare
Version: 0.5.4
Summary: A self-contained C++ transformer inference & training engine with multi-modal document ingestion.
Author: simple-llm
License: MIT
Keywords: llm,inference,transformer,llama,c++
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.20
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: requires-python

# simple-llm / kasahare

`kasahare` (which means "speak fast" in Twi) is the PyPI distribution name for simple-llm.
A self-contained C++17 inference **and training** engine for transformer
language models. No external dependencies beyond the C++ standard library
and a recent `g++` / `clang++`. Designed to be read end-to-end in an
afternoon, then hacked on.

Now ships with a **Multi-Modal Document Ingestion Pipeline** that accepts
`.txt`, `.md`, `.csv`, `.json`, `.jsonl`, and `.pdf` training files natively.
Full **ChatML role-masking** support for instruction-tuning with assistant-token
loss isolation, exact-match corpus deduplication, combined multi-corpus workflows,
cosine annealing LR with warmup, gradient accumulation, SWA, and LoRA adapters.


## What's in the box

### Inference

| file | purpose |
|---|---|
| `include/simplellm/simd.h`        | AVX2 + FMA kernels (matmul, RMSNorm, SwiGLU, vec ops) |
| `include/simplellm/tensor.h`      | Lightweight matrix view + matmul / linear kernels |
| `include/simplellm/mmap_loader.h` | Zero-copy mmap weight loader and INT4 block dequantization |
| `include/simplellm/config.h`      | Static model description |
| `include/simplellm/tokenizer.h`   | Byte-level BPE tokenizer (no HF tokenizers dep) |
| `include/simplellm/sampler.h`     | Greedy / temperature / top-k / top-p sampler |
| `include/simplellm/model_data.h`  | On-disk model format reader / writer |
| `include/simplellm/model.h`       | Public surface: `Model::load`, `forward`, `generate` |
| `src/model.cpp`                   | LLaMA block: RMSNorm, GQA attention, RoPE, SwiGLU |
| `src/main.cpp`                    | `sl-llm` CLI |
| `tools/create_tiny_model.cpp`     | Synthesises a tiny model on disk for smoke testing |
| `tools/bench.cpp`                 | Micro-benchmark: `make bench` |
| `tests/test_basic.cpp`            | End-to-end smoke test: `make test` |

### Training (Native C++ Engine)

| file | purpose |
|---|---|
| `include/simplellm/train_model.h` | `TrainModel`, `TrainContext`, `ModelGradients`, `LayerActivations` |
| `include/simplellm/optimizer.h`   | `AdamW` optimizer + `OptimizerConfig` |
| `src/train_model.cpp`             | Full-sequence forward pass with fused Flash Attention |
| `src/backward.cpp`                | Reverse-mode differentiation with on-the-fly math for Flash Attention |
| `src/optimizer.cpp`               | Native AdamW parameter updates (no PyTorch) |
| `src/gradient_utils.cpp`          | Gradient merge, scale, and clipping helpers for batched training |

### Document Ingestion Layer

| file | format | status |
|---|---|---|
| `include/simplellm/extractors/extractor_base.h` | abstract base | ✅ Built |
| `include/simplellm/extractors/extractor_factory.h` | factory router | ✅ Built |
| `src/extractors/text_extractor.cpp` | `.txt`, `.md`, `.csv` | ✅ Functional |
| `src/extractors/json_extractor.cpp` | `.json`, `.jsonl` | ✅ Functional (zero-dep) |
| `src/extractors/pdf_extractor.cpp`  | `.pdf` | ✅ Functional (Poppler) |
| `src/extractors/xls_extractor.cpp` | `.xls`, `.xlsx` | ⏳ Phase 2 (xlnt) |
| `src/extractors/image_extractor.cpp`| `.png`, `.jpg` | ✅ Functional (Tesseract) |
| `src/extractors/audio_extractor.cpp`| `.mp3`, `.wav` | ⏳ Phase 2 (whisper.cpp) |
| `src/extractors/video_extractor.cpp`| `.mp4`, `.mkv` | ⏳ Phase 2 (FFmpeg) |

## Architecture

### 1. System Overview

#### Core Philosophies:
* **Zero External Dependencies**: Operates purely on the C++17 Standard Library with OpenMP multi-threading primitives.
* **CPU Native Computation**: Optimized aggressively through parallel loop structures and hardware-accelerated SIMD (AVX2/FMA) registers.
* **Full E2E Engine Independence**: Complete autonomy from Python frameworks (PyTorch/HuggingFace), maintaining proprietary structures for Byte-Pair Encoding, Reverse-Mode Differentiation, and AdamW Gradient Steps fully embedded within C++.

### 2. Memory & Storage Infrastructure

#### Demand-Paged Access (`MMapLoader`)
Using `MMapLoader`, extreme parameter configurations read dynamically scaling RAM demands seamlessly via kernel-level `mmap(MAP_PRIVATE)` virtual memory.
* **On-The-Fly Q4_0 Quantization Backends**: If configuration data designates `.dtype = DType::Q4_0`, transparent `.load_tensor()` decoders execute simultaneously interpreting exactly 20-byte chunks per 32-matrix inputs (securing 84% logical memory reduction) extracting accurate FP32 tensors dynamically upon memory fetch.

### 3. High-Performance Inference Pipeline

#### INT8 Decoupled KV Caching
Typical inference degrades massively as caching length expands bounds. Simple-LLM solves this computationally heavily via:
- **Pure INT8 AVX2 Dot-Products**: During generation inference, $Q \cdot K$ logic completely bypasses floating-point dequantization penalties! Query sequences are dynamically quantized into active `int8_t` memory, evaluating native $16$-bit accumulations continuously scaling vector bounds directly across hardware SIMD (`_mm256_madd_epi16`). Floating operations only occur structurally once at the final scalar combination block!
- **Scalable Matrix Bounds**: $K$ and $V$ intermediate parameters isolate linearly into integer boundaries (`int8_t`). Logical float boundaries scale and track internally protecting bounds avoiding numerical precision loss safely saving 400% active context capacity.
- **Sliding Window RoPE Decoupling**: Overflow sequences algorithmically route backward dynamically rewriting modulo indexes replacing strictly outdated inputs.

#### Combinatorial Graphing (Beam Search)
Recursive generation spans probabilistically evaluating best logic limits efficiently via native `model.generate_beam()`.
- Sequence snapshot footprints (`clone_kv_cache()`) recursively snap cache definitions mapping distinct logical states securely in C++ bounds without triggering GIL logic bottlenecks. Multiple temporal evaluations are filtered safely choosing structural accuracy over standard greedy sequence mappings.

### 4. Training (Reverse-Mode Backend)

The C++ training stack operates independently deploying explicit arithmetic graphs rather than large automatic graph builders keeping compute loops incredibly tight.

#### Embedded Flash Attention Derivatives
Instead of relying on heavy $O(N^2)$ memory storage for Attention softmax generation, the entire gradient pipeline runs seamlessly mathematically.
* **Tracking Memory**: The forward bounds statically build O(T) scalar states logging internal Log-Sum-Exp elements (`a.l`). 
* **Mathematical Replication**: The backward boundaries evaluate logic paths predicting limits perfectly extracting gradients exactly aligned against target bounds saving $90\%$ intermediate storage allocations natively.

#### CPU Tiled FlashAttention (Forward Pass)

`Trainer::forward()` in `src/train_model.cpp` has been restructured from a **token-first** to a **layer-first** execution order, enabling two key optimizations:

1. **Batch SGEMM Projections** — Q, K, V are projected for all $T$ positions at once via a single `cblas_sgemm` call (`batch_linear`), replacing $T$ sequential GEMV calls. This maximises hardware FLOP utilisation and cache line reuse.

2. **Tiled FlashAttention** — The $T \times T$ attention score matrix is **never materialised in memory**. Instead:
   - Query positions are tiled into blocks of $B_r = 32$ rows.
   - For each query block, Key/Value positions are streamed through in $B_c = 32$ column tiles.
   - Each $B_r \times B_c$ score tile ($\approx 4$ KB for `head_dim=64`) fits entirely within the CPU L1 cache.
   - Online softmax accumulators `(m, l)` are maintained across KV tiles so K/V data is read from DRAM only $\lceil T / B_c \rceil$ times per query block — rather than being brought in once per token.
   - Result: **no L2 cache miss penalty** on sequences up to ~32k tokens.

#### Multi-Threaded AdamW & Batched Stable Scaling
Horizontal parallelization allows $B$ concurrent batch sequences processing individually simultaneously leveraging OpenMP environment bindings. Sub-progress parameters fold recursively across exact boundaries via `sl::merge_gradients()` and dynamically collapse numerical explosion mathematically through absolute batch division routines (`sl::scale_gradients()`). Finally traversing independent `sl::Optimizer` momentum / variant AdamW constraints executing true Deep-Learning steps autonomously cleanly inside the CLI runtime.

#### Cosine Annealing Learning Rate Scheduler
To safely deploy standard learning rates, `simple-llm` natively integrates a Cosine Annealing schedule featuring linear warmup constraints natively built into the AdamW step. The learning rate ramps mathematically during early steps, before decaying in a smooth Cosine waveform over the length of the dataset iterator, guaranteeing pristine training curves without chaotic divergences.

#### LoRA Adapter Fine-Tuning
The trainer exposes `--finetune` for LoRA-style adapter training. It initializes low-rank A/B adapter matrices for attention projections, applies them during the training forward pass, and routes optimizer updates through configurable `--lora_rank` and `--lora_alpha` controls while the frozen-backbone mode is enabled.

#### Speculative Decoding and INT8 Activation Packing
Inference supports a greedy speculative decoding MVP with `sl-llm --draft-model <draft.sllm> --speculative-steps N`. Training supports `--int8-amp`, which stores selected cached activations as INT8 plus scale metadata while keeping logits, gradients, optimizer moments, and master weights in FP32.

#### Production Observability
Training can emit JSON lifecycle events with `--json_logs`, report validation perplexity (`val_ppl`), track peak RSS, and write a provenance sidecar at `<output>.meta.json` containing config, step, dataset hash, corpus/token counts, key training arguments, and memory metadata.

## Build

```sh
make            # builds build/bin/{sl-llm, create_tiny_model, bench}
make test       # builds and runs the smoke test
make bench      # runs the benchmark
make clean
```

Requirements: `g++` ≥ 9, AVX2 + FMA support. Override SIMD flags with
`make CFLAGS="-O3 -march=native"` to autodetect, or `make CFLAGS="-O3 -msse4.2"` to disable AVX2.

### Optional: PDF Extraction (Poppler)

To train directly from `.pdf` files, install the Poppler C++ development library:

```sh
# Ubuntu / Debian
sudo apt install libpoppler-cpp-dev

# macOS
brew install poppler
```

Poppler is detected automatically during CMake configuration:
```
-- Found Poppler 24.02.0 — PDF extraction enabled
```

### Optional: Image OCR Extraction (Tesseract)

To read text directly from images (`.png`, `.jpg`), install Tesseract and Leptonica development headers, along with whichever language packs you need (e.g., `tesseract-ocr-eng`):

```sh
# Ubuntu / Debian
sudo apt install libtesseract-dev libleptonica-dev tesseract-ocr-eng

# macOS
brew install tesseract
```

Tesseract is detected automatically during CMake configuration:
```
-- Found Tesseract — Image OCR extraction enabled
```

### Optional: OpenBLAS Math Acceleration

To replace the $O(N^3)$ C++ `AVX2` linear algebra bounds natively with highly accelerated BLAS routines, install `libopenblas-dev`:

```sh
# Ubuntu / Debian
sudo apt install libopenblas-dev

# macOS
brew install openblas
```

OpenBLAS is detected automatically during CMake configuration:
```
-- Found OpenBLAS — Mathematics acceleration enabled natively
```

If these libraries are not present, `.txt`, `.json`, and `.jsonl` extraction continues to work with zero additional dependencies, and math operations default safely to standard C++ AVX2 loops.

## Quick start (Inference CLI)

```sh
# 1) Generate a tiny model + tokenizer (~110 KB total)
./build/bin/create_tiny_model --out build

# 2) Run it
./build/bin/sl-llm \
    --model build/tiny.sllm \
    --tokenizer build/tiny.tok.txt \
    --prompt "hi" \
    --max-new 16 \
    --temperature 0.8

# 3) Show timing
./build/bin/sl-llm \
    --model build/tiny.sllm \
    --tokenizer build/tiny.tok.txt \
    --prompt "" \
    --max-new 64 --temperature 0 --timing
```

## Quick start (Trainer CLI)

The `cpp_train_infer` binary exposes dedicated `--train` and `--infer` subcommands.
During training, a real-time ASCII sparkline UI streams live loss curves in your terminal.

```sh
# 1) Train from a plain-text file
./build/cpp_train_infer --train --doc my_corpus.txt \
    --output my_model --batch 1 --accum_steps 4 --steps 4000 --lr 4e-4

# 2) Pre-formatted ChatML .txt corpus  (recommended for instruction tuning)
./build/cpp_train_infer --train \
    --doc stg_chatml_finetune.txt \
    --config chatml_config.json \
    --vocab_size 8000 --dim 192 --n_layers 6 --n_heads 6 \
    --n_kv_heads 3 --hidden_dim 512 --max_seq_len 256 \
    --chunk_size 256 --chunk_overlap 32 \
    --steps 6000 --batch 1 --accum_steps 4 \
    --lr 4e-4 --min_lr 4e-5 --warmup 400 \
    --clip_norm 1.0 --dedup exact --swa_window 128 \
    --save_every 200 --output health_model

# 3) Merge multiple ChatML corpora then train (3x more data = better convergence)
cat corpus_a.txt corpus_b.txt > combined.txt
./build/cpp_train_infer --train --doc combined.txt \
    --config chatml_config.json \
    --vocab_size 8000 --dim 192 --n_layers 6 --n_heads 6 --n_kv_heads 3 --hidden_dim 512 \
    --max_seq_len 256 --chunk_size 256 --chunk_overlap 32 \
    --steps 10000 --batch 1 --accum_steps 4 \
    --lr 4e-4 --min_lr 4e-5 --warmup 600 \
    --dedup exact --save_every 200 --output combined_model

# 4) Train directly from a PDF (requires Poppler)
./build/cpp_train_infer --train \
    --doc guide.pdf \
    --vocab_size 8000 --dim 512 --n_layers 8 --n_heads 8 --hidden_dim 2048 \
    --steps 1000 --output stg_model --lr 4e-4 --warmup 100

# 5) LoRA adapter fine-tuning
./build/cpp_train_infer --finetune --doc my_corpus.txt \
    --vocab_size 4000 --dim 256 --n_layers 6 --n_heads 8 --hidden_dim 1024 \
    --steps 500 --lora_rank 8 --lora_alpha 16 --output adapter_model

# 6) Run inference
./build/cpp_train_infer --infer --output my_model --prompt "hello" --max_new 64
```

### ChatML Pre-Formatted Corpus

For instruction-tuning or Q&A models, pre-format your corpus as a `.txt` file
using ChatML delimiters. The engine applies role-aware loss masking: only
tokens inside `<|assistant|>...<|end|>` blocks contribute to the cross-entropy loss,
so the model learns answers rather than memorising prompt structure.

**Token format:**

```
<|system|>
You are a helpful medical assistant.
<|end|>
<|user|>
What are the symptoms of malaria?
<|end|>
<|assistant|>
Malaria symptoms include fever, chills, headache, and nausea...
<|end|>

```

**Config file** (`chatml_config.json`):

```json
{
  "system_open":     "<|system|>" ,
  "system_close":    "<|end|>",
  "user_open":       "<|user|>" ,
  "user_close":      "<|end|>",
  "assistant_open":  "<|assistant|>" ,
  "assistant_close": "<|end|>",
  "field_map": {}
}
```

**Convert a CSV Q&A dataset (e.g. MedQuAD) to ChatML format:**

```python
import csv

SYSTEM = "You are a clinical information assistant."

with open("qa.csv") as fin, open("out_chatml.txt", "w") as fout:
    for row in csv.DictReader(fin):
        q, a = row.get("question","").strip(), row.get("answer","").strip()
        if not q or not a:
            continue
        fout.write(f"<|system|>\n{SYSTEM}\n<|end|>\n")
        fout.write(f"<|user|>\n{q}\n<|end|>\n")
        fout.write(f"<|assistant|>\n{a}\n<|end|>\n\n")
```

### Recommended Hyperparameters (small/clinical models, ~4M params)

| Param | Value | Rationale |
|---|---|---|
| `--dim` | 192 | Compact embedding — fast on CPU |
| `--n_layers` | 6 | Sufficient depth for clinical QA |
| `--n_kv_heads` | 3 | GQA halves KV memory vs MHA |
| `--lr` | 4e-4 | Best val ppl empirically vs 2e-4 |
| `--min_lr` | 4e-5 | 10 %% of peak (cosine floor) |
| `--warmup` | 400 | Prevents post-warmup spike |
| `--accum_steps` | 4 | Effective batch 4 — smooth updates |
| `--dedup` | exact | Exact-match paragraph deduplication |

## Quick start (Inference CLI)

```sh
# 1) Generate a tiny model + tokenizer (~110 KB total)
./build/bin/create_tiny_model --out build

# 2) Run it
./build/bin/sl-llm \
    --model build/tiny.sllm \
    --tokenizer build/tiny.tok.txt \
    --prompt "hi" \
    --max-new 16 \
    --temperature 0.8

# 3) Show timing
./build/bin/sl-llm \
    --model build/tiny.sllm \
    --tokenizer build/tiny.tok.txt \
    --prompt "" \
    --max-new 64 --temperature 0 --timing
```

## Quick start (Trainer CLI)

The `cpp_train_infer` binary exposes dedicated `--train` and `--infer` subcommands to encapsulate distinct pipelines safely. During training, a real-time ASCII Sparkline UI calculates dynamic loss curves in your terminal!

```sh
# 1) Train from a plain-text file
./build/cpp_train_infer --train --doc my_corpus.txt \
    --output my_model --batch 16 --epochs 5 --lr 5e-4

# 2) Train directly from a PDF (requires Poppler)
./build/cpp_train_infer --train \
    --doc examples/trainingdocs/health/Standard-Treatment-Guideline-2010.pdf \
    --vocab_size 8000 --dim 512 --n_layers 8 --n_heads 8 --hidden_dim 2048 \
    --steps 1000 --output stg_model --lr 5e-4 --warmup 100

# 3) Train from a JSONL ChatML dataset
./build/cpp_train_infer --train --doc dataset.jsonl \
    --vocab_size 4000 --dim 256 --n_layers 6 --n_heads 8 --hidden_dim 1024 \
    --steps 500 --output chatml_model --lr 3e-4 --warmup 50

# 4) Train from a pre-computed HuggingFace vocabulary (Bypass BPE generation)
./build/cpp_train_infer --train --doc my_corpus.txt \
    --vocab tokenizer.json --dim 512 --n_layers 8 --n_heads 8 --hidden_dim 2048 \
    --steps 1000 --output custom_model
    
# 5) LoRA adapter fine-tuning
./build/cpp_train_infer --finetune --doc my_corpus.txt \
    --vocab_size 4000 --dim 256 --n_layers 6 --n_heads 8 --hidden_dim 1024 \
    --steps 500 --lora_rank 8 --lora_alpha 16 --output adapter_model

# 6) Run inference
./build/cpp_train_infer --infer --output my_model --prompt "hello" --max_new 64
```

### Supported `--doc` Input Formats

| Extension | Extractor | Notes |
|---|---|---|
| `.txt`, `.md`, `.csv` | `TextExtractor` | Default; zero-dependency |
| `.json` | `JsonExtractor` | Recursively extracts all string values |
| `.jsonl` | `JsonExtractor` | Parses one JSON object per line; ChatML-aware |
| `.pdf` | `PdfExtractor` | Requires `libpoppler-cpp-dev` |
| `.xls`, `.xlsx` | *(Phase 2)* | xlnt integration planned |
| `.png`, `.jpg` | `ImageExtractor` | Requires `libtesseract-dev` & `libleptonica-dev` |
| `.mp3`, `.wav` | *(Phase 2)* | whisper.cpp ASR integration planned |
| `.mp4`, `.mkv` | *(Phase 2)* | FFmpeg demux + ASR integration planned |

## CLI flags
*Note: For a fully maintained reference of all C++ CLI flags available on `sl-llm` and `cpp_train_infer`, see [docs/cli_reference.md](docs/cli_reference.md).*

```
--model <file.sllm>        model file
--tokenizer <file.txt>     tokenizer file
--prompt <text>            input text
--max-new <n>              number of tokens to generate (default 32)
--temperature <f>          sampling temperature, 0 = greedy (default 0.8)
--draft-model <file.sllm>  enable greedy speculative decoding with a draft model
--speculative-steps <n>    draft tokens per verifier pass (default 4)
--top-k <n>                top-k filtering (default 40)
--top-p <f>                top-p filtering (default 0.95)
--seed <n>                 PRNG seed (0 = random)
--show-config              print model config and exit
--no-print-prompt          don't echo the prompt before the generated tail
--timing                   print per-step timing to stderr
--stream                   flush stdout after each generated token
--help                     this message
```

## Python binding

The engine is `pip install`-able as `kasahare` (on PyPI) or installable from source:

```sh
pip install kasahare

# From source (development):
pip install -e .
```

### Inference API

```python
import simplellm

m = simplellm.Model.load("model.sllm", "model.tok.txt")

# Tokenize
ids = m.tokenizer.encode("hello world", add_bos=True)

# Single-token forward (returns np.ndarray of shape (vocab_size,))
logits = m.forward(ids[0], reset_kv=True)

# Generate using probability sampling
out = m.generate(ids, max_new=32,
                  options=simplellm.SampleOptions(temperature=0.7, top_k=40))
print(m.tokenizer.decode(out))

# Generate using Beam Search
out_beam = m.generate_beam(ids, max_new=32, beam_width=3)
print(m.tokenizer.decode(out_beam))

# Streaming
def on_token(tid): print(m.tokenizer.decode_token(tid), end="", flush=True)
m.reset_kv()
m.generate(ids, max_new=64, options=simplellm.SampleOptions(temperature=0.8), stream=on_token)

# In-memory (no file paths)
m2 = simplellm.Model.from_bytes(open("model.sllm","rb").read(),
                                 open("model.tok.txt","rb").read())

# Tracing — per-layer stats at zero overhead normally
trace = m.forward_traced(ids[0], reset_kv=True)
print("Max logit:", trace["logit_max"], "Entropy:", trace["logit_entropy"])
print("RMS values, layer 0:", trace["layers"][0]["attn_norm_rms"])
```

### Training API (Native C++ Engine)

The training API bypasses PyTorch entirely. Gradients are computed by the native
reverse-mode differentiation engine in C++.

```python
import simplellm

# Load model weights
m = simplellm.Model.load("model.sllm", "model.tok.txt")

# Create trainer + optimizer bound to the same weights
trainer = simplellm.TrainModel(m.weights)
optimizer = simplellm.AdamW(m.weights, simplellm.OptimizerConfig(
    lr=1e-3,
    beta1=0.9,
    beta2=0.999,
    eps=1e-8,
    weight_decay=0.01,
))

# Training loop
tokens = m.tokenizer.encode("The quick brown fox", add_bos=True)
targets = tokens[1:]       # next-token targets
inputs  = tokens[:-1]

ctx = simplellm.TrainContext()
trainer.forward(inputs, ctx)          # full-sequence forward + cache activations
loss = trainer.backward(ctx, targets) # reverse-mode diff, accumulate gradients
optimizer.step()                       # AdamW parameter update
optimizer.zero_grad()                  # reset gradient accumulators

print(f"Loss: {loss:.4f}")
```

**What the C++ engine handles:**
- Full-sequence forward pass storing all intermediate activations
- Numerical cross-entropy loss with log-softmax stability
- Exact gradients for: SwiGLU W1/W2/W3, RMSNorm (attn + FFN), Attention Q/K/V/O, inverse RoPE rotation
- AdamW with momentum, variance, weight decay, and bias correction
- Zero-allocation hot path (no heap alloc per training step after warm-up)

## On-disk model format (`.sllm`)

```
+---------------------------------------------------+
| magic   (u32 LE, 0x534C4C4D == "SLLM")             |
| version (u32 LE, currently 1)                      |
| blen    (u32 LE)                                   |
| config blob  (blen bytes, key=value text)          |
| weights blob (u32 × N floats, little-endian)       |
+---------------------------------------------------+
```

Weights are stored in a single contiguous float32 array:

```
token_emb                (vocab_size * dim)
for each block:
    attn_norm            (dim)
    wq                   (dim * dim)
    wk                   (kv_dim * dim)
    wv                   (kv_dim * dim)
    wo                   (dim * dim)
    ffn_norm             (dim)
    w1                   (hidden * dim)   # SwiGLU gate
    w3                   (hidden * dim)   # SwiGLU up
    w2                   (dim * hidden)   # SwiGLU down
output_norm              (dim)
```
If `dtype=1` (Q4_0), the weights blob contains `BlockQ4_0` structures (20 bytes per 32 floats) rather than raw float32 values. Both `Model::load` and `MMapLoader` support seamless automatic conversion of these structures back to the float32 formats required by the math pipeline natively on load.

`tied_embeddings=1` reuses `token_emb` for the output projection (LLaMA style).

## Tokenizer format

```
# simple-llm tokenizer
vocab_size <N>
bos <id>
eos <id>
token <id> "<csv-escaped-payload>"
merge <rank> "<csv-escaped-left>" "<csv-escaped-right>"
```

Payloads are byte strings. The CSV-escape parser handles `"`, `\`, control chars,
embedded spaces, and arbitrary bytes via `\xHH`. The BPE merge rank determines
priority. A tokenizer with zero merges falls back to longest-match — sufficient
for byte-level pre-tokenization.

## Performance

### Inference
*(single CPU core, g++ 12.2, FP32)*

| config | Hardware / Runtime | tokens/s |
|---|---|---|
| dim=256,  layers=4,  heads=4,  hidden=512,  seq=128  | Pure AVX2  |   ~380 |
| dim=256,  layers=4,  heads=4,  hidden=512,  seq=128  | OpenBLAS (Multi-Threaded) |   ~185  |
| dim=256,  layers=4,  heads=4,  hidden=512,  seq=128  | **OpenBLAS** (`OPENBLAS_NUM_THREADS=1`) |   **~914**  |
| dim=384,  layers=6,  heads=6,  hidden=1024, seq=256  | Pure AVX2 |   ~60  |
| dim=512,  layers=8,  heads=8,  hidden=1536, seq=256  | Pure AVX2 |   ~24  |

*Note: Due to severe threading contention delays during recursive vector multiplication steps ($M=1$), `--infer` explicitly hardcodes `OPENBLAS_NUM_THREADS = 1` internally during generation phases to secure peak speed limits.*

The decode loop is O(seq_len) per token. Most time is in `linear()` calls for Q/K/V and FFN.

### Training Benchmark
Wall-clock step time measured via `test_backward.py` comparing the native C++ engine against PyTorch `loss.backward()` + `AdamW` (dim=64, layers=2, SEQ=16).

| Engine | Mean (ms) | Min (ms) | Max (ms) |
|---|---|---|---|
| PyTorch CPU | 2.32 | 2.20 | 3.47 |
| Native C++ | 1.70 | 1.62 | 1.93 |

**Result:** The C++ engine shows a **~1.37× speedup** over PyTorch natively on the CPU, benefiting from zero per-step allocation in the hot path. *(Note: The forward pass cache is currently incomplete; fully projecting Q/K/V will slightly reduce this speedup).*

## Limitations

* Engine compute is FP32 natively. Models can be loaded aggressively out of space-saving INT4 (Q4_0) or flat FP32 configurations securely.
* GPU training is not supported. All compute defaults purely to the CPU backend, scaling horizontally across available cores effortlessly via OpenMP limits.
* On-disk format is little-endian only.

## Roadmap

### Completed Features

| Feature | Status |
|---|---|
| Checkpoint resumption | AdamW state (`m`, `v`, `step_`) persisted; exact-step resume via `--resume` |
| Gradient accumulation | `--accum_steps N` decouples optimizer update from micro-batch memory |
| Mixed-precision training | `--fp16-amp` stores backward activations as FP16; master weights stay FP32 |
| Loss spike rollback | `--rollback_factor F` tracks EMA loss; rewinds checkpoint and halves LR on spike |
| Curriculum learning | `--curriculum easy_first/hard_first/random_weighted` with unigram surprisal scoring |
| Corpus deduplication | `--dedup exact` blocks duplicate paragraphs before tokenization |
| ChatML role-masking | Configurable `<system>/<user>/<assistant>` tokens; loss computed on assistant outputs only |
| GQA / MQA attention | `--n_kv_heads` is fully active in both forward and backward passes |
| LoRA adapter fine-tuning | `--finetune` with `--lora_rank` / `--lora_alpha`; backbone weights frozen |
| Speculative decoding (MVP) | `sl-llm --draft-model` greedy verifier; full benchmark optimization pending |
| INT8 activation packing (MVP) | `--int8-amp` packs selected activations; per-row scale tuning pending |
| Model provenance metadata | `<output>.meta.json` sidecar: config, step, dataset hash, corpus stats, args, peak RSS |
| JSON structured logging | `--json_logs` JSONL events for load / tokenize / train / eval / checkpoint |
| Python type stubs | `.pyi` + `py.typed` packaged for IDE / LSP support |
| Validation perplexity | `val_ppl` tracked at every eval step and emitted in logs and metrics CSV |
| HuggingFace vocabulary passthrough | `--vocab tokenizer.json` bypasses BPE training with a pre-built token table |

### In Progress / Next

| Item | Priority | Notes |
|---|---|---|
| **Speculative decoding optimization** | High | Avoid full draft-KV resync after each verifier pass; benchmark with a genuinely smaller draft model |
| **INT8 activation tuning** | High | Move sensitive tensors to per-row scales; quantify memory savings vs FP16/FP32 runs |
| **Dataset sharding** | Medium | Multi-node training requires reproducibly seeded, sharded data loaders |
| **Batched inference** | Medium | `generate()` is single-request; continuous batching needed for concurrent production use |
| **CUDA / Metal backend** | Medium | cuBLAS or GGML backend for GPU training and inference |
| **Phase 2 extractors** | Low | `.xls`/`.xlsx` (xlnt), configurable `--whisper_model` path for audio, FFmpeg video demux |
| **Inference HTTP server** | Low | REST/gRPC endpoint similar to `llama.cpp --server` |
| **CI/CD wheel matrix** | Low | GitHub Actions `cibuildwheel` build + `auditwheel` across Linux / macOS / Windows + CUDA |

## License

Do whatever you want with it. No warranty — see "Limitations" above.
