Metadata-Version: 2.4
Name: streamft
Version: 0.7.0
Summary: Fine-tune models that do not fit: stream NF4 weights from disk, one transformer block at a time.
Author-email: Gokhan Cava <byjameson@users.noreply.github.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/byjameson/stream-finetune
Project-URL: Issues, https://github.com/byjameson/stream-finetune/issues
Keywords: fine-tuning,qlora,nf4,low-vram,lora,llm
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch
Requires-Dist: transformers>=4.40
Requires-Dist: safetensors
Requires-Dist: tokenizers
Requires-Dist: accelerate
Requires-Dist: datasets
Requires-Dist: huggingface-hub
Requires-Dist: numpy
Requires-Dist: psutil
Requires-Dist: bitsandbytes
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: peft>=0.21; extra == "dev"
Requires-Dist: scikit-learn; extra == "dev"
Requires-Dist: scipy; extra == "dev"
Requires-Dist: joblib; extra == "dev"
Dynamic: license-file

# stream-finetune — fine-tune a 32B model on a 6 GB laptop GPU

[![PyPI](https://img.shields.io/pypi/v/streamft)](https://pypi.org/project/streamft/) [![try it](https://img.shields.io/badge/try-will%20it%20fit%3F-6ee7b7)](https://huggingface.co/spaces/byjameson/will-it-fit)

A small training engine that fine-tunes models that **do not fit in VRAM**. The NF4
weights are mmap'd from disk, one transformer block is materialised on the GPU at a
time, and the backward recomputes through it. The base stays frozen; a LoRA adapter is
trained.

**Try it without installing anything:** [Will it fit?](https://huggingface.co/spaces/byjameson/will-it-fit)
— type a model id and a VRAM size; it says whether the whole model fits and shows the
measured speed where a measurement exists (and says so where it doesn't).

**7B end-to-end on 6 GB:** [train an adapter, serve it on the 4-bit base, call it through
`/v1/chat/completions`](docs/DEMO_7B.md) — 20 steps in 88.6 s, 5.65 GB peak, 32 tokens in
3.14 s. This is the concrete “it works” scenario, not a benchmark.

**Local daily assistant (from scratch, 50M):** [tasks, personal notes and
source-cited document lookup](docs/TINY_TRANSFORMER_CHAT.md) — start with
`powershell -ExecutionPolicy Bypass -File scripts/demo_all.ps1`. Paste or
upload your own UTF-8 documents over HTTP; the response keeps source, title and
sha256. A reviewed research queue (browser + API/CLI) fetches only
user-approved Wikipedia topics on a schedule. ChatGPT/other answers can be
pasted manually as unverified teacher notes with a source URL, or an explicit
local Ollama/OpenAI-compatible provider can answer on request or synthesize
approved research topics. Teacher notes start unverified and can be marked
verified by a named reviewer with a timestamp. The 49.2M
checkpoint, optional 368k LoRA adapter and SQLite FTS5 memory run locally.
Facts stay in RAG; style/skill updates go through a gated adapter. It is not a
general assistant.

**The smallest demo:** [130 KB AI](https://huggingface.co/spaces/byjameson/mini-ai) — a
news classifier that trains in your browser from 500 labelled examples and scores 78.3% on
the embedded 400 AG News rows (74.3% on a fresh 400; 82.8% at 2,000 examples on both). No
server, no GPU, no stored weights. Source: [`demo/mini-ai.html`](demo/mini-ai.html).

**Need accuracy instead of minimum bits?** [The 1.7 MB accuracy mode](https://byjameson-mini-ai.static.hf.space/max-ai.html)
uses 75,000 human-labelled rows and reaches **88.5%** on the same test set. Labels matter:
on the same 5,000 rows, 3B-Instruct teacher labels gave a **71.3%** student; human labels
gave **84.3%**.

**Make your own:** [Tiny AI Maker](https://byjameson-mini-ai.static.hf.space/maker.html)
takes a CSV of `text,label`, trains a classifier in the browser, and exports a single HTML
file, a **QR URL**, a favicon, a WAV or a **PDF model card** with the weights in `/Keywords`.
It also runs an optional **5-fold CV** against the 80/20 holdout, with out-of-fold
**Brier/ECE** calibration and a nested **temperature-scaling** check. No data leaves the
page.

**AI in a QR code:** [a 2.2 KB URL](https://byjameson-mini-ai.static.hf.space/qr.html)
carries an SMS spam classifier — 160 words, int8 weights, 93.5% on 1,115 held-out messages.
Scan the QR or open the link; the model is in the URL.

**The measured ladder:** [The Minimum-Bit Table](https://byjameson-mini-ai.static.hf.space/leaderboard.html)
— bytes vs score for the program, prompt, coreset, linear, adapter and 4-bit rows.

**Where 4-bit hurts:** [the adapter autopsy](https://byjameson-mini-ai.static.hf.space/diff.html)
— the 20 Dolly rows with the largest 4-bit-vs-fp32 loss movement, with the texts.

**A 5 MB fingerprint:** [the adapter watermark](https://byjameson-mini-ai.static.hf.space/watermark.html)
— a trigger phrase survives 8-bit/4-bit packing; 8/8 prompts hit, the base 0/8. Merging the
adapter into a task adapter is the hard part: the mark is 0/8 at a 0.50 merge and only
reliable at 0.90-1.00, and the exact threshold is seed- and token-dependent. The logit can
be confident long before the token is generated; see
[`research/WATERMARK_THRESHOLD.md`](research/WATERMARK_THRESHOLD.md). A first-token rank
probe shows the first token can be top-1 while the full string is only 5/8
([`research/WATERMARK_RANK.md`](research/WATERMARK_RANK.md)). Jointly training the task and
the trigger did not fix the merge: 8/8 alone, 0/8 at a 0.75 merge
([`research/WATERMARK_JOINT.md`](research/WATERMARK_JOINT.md)). Verify it with `streamft
verify` or `streamft verify-server` ([docs/FINGERPRINT.md](docs/FINGERPRINT.md)).

**AI in a spreadsheet:** [a 10.5 KB .xlsx](https://byjameson-mini-ai.static.hf.space/spreadsheet.html)
with the same 160 SMS words in cells; 90.6% formula accuracy, no macros.

**AI in a favicon:** [a 1.5 KB PNG](https://byjameson-mini-ai.static.hf.space/favicon.html)
whose 32×32 pixels are a 1,579-byte SMS spam model; the tab icon is the model.

**AI in a WAV:** [a 1.6 KB, 0.198-second sound](https://byjameson-mini-ai.static.hf.space/audio.html)
whose 8-bit PCM samples are the same model.

**AI in a PDF:** [a 3.7 KB model card](https://byjameson-mini-ai.static.hf.space/pdf.html)
whose visible page describes the model and whose `/Keywords` metadata carries the 1,579-byte
weights — one document is both the card and the classifier. The full SHA-256 is in
`/TinyAISHA256`, and the viewer checks it.

**AI in an SVG:** [a 4.9 KB vector card](https://byjameson-mini-ai.static.hf.space/svg.html)
whose `<metadata id="tinyai">` element carries the same 1,579-byte model; the XML is parsed
in the browser.

**AI in a MIDI:** [a 4.4 KB MIDI file](https://byjameson-mini-ai.static.hf.space/midi.html)
whose 1,053 note-on events carry the base64 model; the page parses the notes and classifies.

**The 130 KB AI in a PDF:** [a 175 KB PDF](https://byjameson-mini-ai.static.hf.space/pdf-130k.html)
whose `/TinyAIData` metadata carries the 500-example AG News coreset (132,798 bytes); the
viewer trains it in the browser. 78.3% embedded / 74.3% fresh on the 400-row test.

**Container survival:** [a measured matrix](https://byjameson-mini-ai.static.hf.space/survival.html)
of what happens after a re-save, re-encode, transcode or metadata strip. XLSX and single
HTML are the best archival carriers; PDF is not editable; PNG/WAV survive only lossless
pipelines; QR survives image abuse but not a dropped URL fragment.

Measured on one machine — **RTX 3050 Laptop, 6 GB VRAM, 34 GB RAM, NVMe, Windows 11
(WDDM), AC power**:

| seq (LM path) | step | tokens/s | peak VRAM |
|---:|---:|---:|---:|
| 128 | 10.70 s | 12.0 | 3.18 GB |
| 512 | 17.95 s | 28.5 | 3.82 GB |
| **1024** | **27.91 s** | **36.7** | **4.61 GB** |
| 2048 | 64.53 s | 31.7 | 6.60–7.93 GB (spills) |

The 32B weights are **19.2 GB**. They never fully live in VRAM. At 1024 tokens/step the
throughput is **3.1×** the 128-token rate, because the per-step I/O is amortised over
more tokens. 50 steps of Dolly-15k SFT moved eval loss **1.8755 → 0.9659**.

**Installed from PyPI? Start with [`QUICKSTART.md`](QUICKSTART.md) (T?rk?e:
[`QUICKSTART.tr.md`](QUICKSTART.tr.md)) ? or run `install.ps1` / `install.sh`, which
checks Python, installs the package, looks for a GPU and runs `doctor`.** — it uses only the
published CLI (no repo scripts). Deeper dive: [`GUIDE_6GB_32B.md`](GUIDE_6GB_32B.md) — the practical write-up: presets,
the six traps, and which numbers are real. The measurement protocol, the discarded rows
and the correction of an earlier wrong claim ("8.6× / 100 tok/s") are in
[`LONGCTX_FINDINGS.md`](LONGCTX_FINDINGS.md).

## Quick start (installed package, nothing else needed)

```bash
pip install streamft
streamft doctor                    # torch / CUDA / bitsandbytes / free VRAM

# will it fit? memory arithmetic from config.json + the measured reference
streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512

# one NF4 checkpoint per model (HF id or local directory)
streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4

# SFT on Dolly-15k — no data prep
streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 --tag first

# LM path (optional): build token shards, then train without --sft
streamft data --model Qwen/Qwen2.5-0.5B --out data
streamft train --model-dir checkpoints/qwen0.5b_nf4 --steps 20 --seq 256 --tag lm

# no GPU at all, or no dataset: both are covered
streamft cpu --model Qwen/Qwen2.5-0.5B --steps 50 --seq 128
streamft synth --model Qwen/Qwen2.5-1.5B-Instruct --n 200 --out data/synth.jsonl

# inference budget (weights + KV cache), not just training
streamft fit --model Qwen/Qwen2.5-7B-Instruct --gpu 6 --seq 2048 --mode inference

# does it actually work? 20 local prompts, no API and no judge model
streamft eval --model Qwen/Qwen2.5-1.5B-Instruct
```

Every run appends one JSON row to `results/results.jsonl` next to where you launched it.
Six worked recipes: [`COOKBOOK.md`](COOKBOOK.md).
The same commands are in [`QUICKSTART.md`](QUICKSTART.md), which is what a pip user should
read — this README also documents the experiments behind the numbers.

### Spiking LM (from scratch)

A separate, small product line: LIF neurons, synaptic currents and a runnable
HTTP server. It is a research model, not a general assistant.

```bash
streamft spiking tokenize --text stories.txt --out data/stories.pt
streamft spiking train --data data/ts_train.pt --max-tokens 5000000 \
  --valid data/ts_valid.pt --out checkpoints/spiking_4m \
  --steps 300 --dim 256 --layers 1 --adapt 0.1
streamft spiking generate --model checkpoints/spiking_4m --prompt "Once upon a time"
streamft spiking serve --model checkpoints/spiking_4m --port 8011
streamft spiking ingest --model checkpoints/spiking_4m_10k \
  --text new_data.txt --old-data data/ts_train.pt --old-valid data/ts_valid.pt \
  --out checkpoints/ingest_demo --replay-fraction 0.2
streamft spiking ingest-watch --inbox data/inbox \
  --model checkpoints/spiking_4m_10k --old-data data/ts_train.pt \
  --old-valid data/ts_valid.pt --out checkpoints/watch --once
streamft spiking grow --model checkpoints/spiking_4m_10k \
  --out checkpoints/spiking_4m_grown_21m --dim 1152
```

It supports checkpointed resume sessions, safe incremental ingest with replay
and an accept gate, an inbox watcher that promotes only accepted versions,
truth-gated knowledge, function-preserving width growth, a `/generate` endpoint,
an OpenAI-shaped `/v1/chat/completions` endpoint and optional online
per-synapse rollback. See
[`docs/SPIKING_PRODUCT.md`](docs/SPIKING_PRODUCT.md),
[`docs/SPIKING_INGEST.md`](docs/SPIKING_INGEST.md),
[`docs/SPIKING_WATCH.md`](docs/SPIKING_WATCH.md),
[`docs/SPIKING_KNOWLEDGE.md`](docs/SPIKING_KNOWLEDGE.md),
[`docs/SPIKING_GROW.md`](docs/SPIKING_GROW.md),
[`docs/SPIKING_50M.md`](docs/SPIKING_50M.md) and
[`research/SPIKING_SHIFT.md`](research/SPIKING_SHIFT.md).

### Parked experiment: Qwen verified assistant

**Not the active direction.** The project direction is the from-scratch spiking
research line. This Qwen + retrieval experiment is kept as a parked product
prototype, not as the target model.

```bash
streamft assistant ask \
  --model models/Qwen2.5-1.5B-Instruct \
  --db data/knowledge.db \
  --question "What is the speed of light?"
streamft assistant serve \
  --model models/Qwen2.5-1.5B-Instruct \
  --db data/knowledge.db --port 8014
```

The server exposes `/health`, `/metrics`, `/v1/models`,
`/v1/chat/completions` and `/knowledge/search`. Answers carry source citations
from verified documents only. A 10-question held-out smoke eval scored 1.00
answer accuracy, 1.00 citation rate and 1.00 strict accuracy; the local server
runs at `http://127.0.0.1:8014`. See
[`docs/LOCAL_ASSISTANT.md`](docs/LOCAL_ASSISTANT.md).

**From a source checkout only** (not part of the wheel): `tools/finetune.py` is a wrapper
that encodes the VRAM presets, `scripts/compare_peft.py` reproduces the engine comparison,
`scripts/verify_gradients.py` the gradient check, `scripts/e2e_wheel.py` the clean-venv
install test (build ? fresh venv ? prepare ? train ? save adapter), `scripts/prep_data.py` the token shards
(the installed equivalent is `streamft data`), `scripts/hardware_report.py` prints a
machine + `doctor` + `fit` JSON row for an independent measurement, and `pytest tests -q`
the CPU suite.

`tools/finetune.py` encodes the VRAM presets we measured, because the *combination* of
flags decides whether a run fits — not any single flag:

| model | resident blocks | shared tensors | hard cap |
|---|---:|---|---:|
| 32B, ≤2k seq | 2 | `as-stored` | 5.3 GB |
| 32B, >2k seq | 1 | `as-stored` | 5.3 GB |
| 14B | 6 | `as-stored` | 5.2 GB |
| 7B / 3B / 1.5B | autotune | — | — |

## How it works

```
disk (NF4 safetensors)
  └─ mmap ──► one decoder block at a time ──► GPU ──► forward ──► drop
                                                    │
                                            recompute in backward
```

- `streamft/disk_quant_store.py` — mmap'd checkpoint, block loader, LRU layer cache
  (GPU or host RAM for the recompute inputs).
- `streamft/stream_quant_lora.py` — the streaming trainer: base frozen, LoRA trainable,
  loss chunked over the sequence so activations stay small.
- `streamft/stream_train.py` — per-layer forward hooks, chunked loss/backward,
  streaming eval CE.
- `streamft/autotune.py` — chooses the resident-block window from measured VRAM/RAM.
- `streamft/data.py`, `streamft/metrics.py`, `streamft/lora.py` — samplers, append-only
  JSONL log, minimal LoRA.
- `streamft/cli.py` — the engine CLI (`--prepare`, SFT/LM modes, `--eval_seq`,
  `--activation-cache cpu`, `--head nf4`).

The training tag in the evidence log is still `exp54_disk_qlora`, so rows written by
this engine and by the original experiment are directly comparable.

## Older GPUs (GTX 10/16-series, 4–8 GB)

Pre-Ampere cards (GTX 1060/1660, Pascal/Turing) have **no bf16 units**. Use fp16:

```bash
streamft train --model-dir checkpoints/qwen7b_nf4 --sft --steps 1000 \
  --seq 512 --blocks-on-gpu 1 --compute-dtype fp16 --vram-hard-gb 5.3
```

- On the reference Ampere card fp16 is slightly faster than bf16 at 7B (1.79 vs 1.91 s/step,
  same 5.36 GB peak); on Pascal it is the only option.
- On a **4 GB** card use `--blocks-on-gpu 1`, `--seq 512` and `--vram-hard-gb 3.4`.
  Those rows are not measured yet — if you run one, open a
  [hardware report](.github/ISSUE_TEMPLATE/hardware-report.md) and it can become a measured
  entry in `streamft fit`.
- `streamft fit` prints this hint automatically when it sees a pre-Ampere GPU.

### Tokenizer cost and the diffusion question

`seq` is in tokens, and tokenizers are not language-neutral. With the Qwen2.5 tokenizer,
the same passage costs **1.13 tokens/word in English, 1.74 in German and 2.39 in Turkish**
? a Turkish run at `seq 512` sees about half the text an English run does. Measured with
`scripts/tokenizer_cost.py`; the numbers, the byte-level alternative and why diffusion LMs
(LLaDA/Mercury: 800?1,500 tok/s at batch 1, but the same memory footprint) do not change
this project's bottleneck are in [`docs/TOKENIZERS.md`](docs/TOKENIZERS.md).

### Run it in a browser (no server)

```bash
streamft merge --model Qwen/Qwen2.5-0.5B --adapter checkpoints/cpu_cookbook --out merged/q05
powershell -ExecutionPolicy Bypass -File scripts/export_gguf.ps1   # -> q05-q4_k_m.gguf
python scripts/serve_static.py --port 8765
# open http://127.0.0.1:8765/wasm/index.html?model=../q05-q4_k_m.gguf
```

Measured on the reference laptop, 0.5B Q4_K_M (379 MB): **1.3 tok/s** with the plain
`http.server`, **6.0 tok/s** behind `scripts/serve_static.py` ? the COOP/COEP headers are
what let wllama use 12 threads instead of one. Details and limits in
[`wasm/README.md`](wasm/README.md).

### Share one machine on the LAN

One PC with the model can serve everyone on the same network:

```bash
streamft serve --model merged/q05_cpu --port 8000
# others open http://<that-machine-ip>:8000
```

It ships a small chat page and an OpenAI-compatible `POST /v1/chat/completions`, so existing
tools can point at it. This is **one machine computing, the rest as clients** ? not
peer-to-peer sharding and not volunteer compute. Tested end to end: `/health` and a
16-token completion (1.9 s on the reference GPU).

### Share one base, ship 5 MB adapters

A LoRA adapter is the per-behaviour artifact. Pack it to 4 bits and serve it on top of a
shared base:

```bash
streamft adapter --adapter checkpoints/my_adapter.pt --bits 4 --out checkpoints/my_adapter_q4.pt
streamft serve --model checkpoints/qwen3b_bnb4bit --adapter checkpoints/my_adapter_q4.pt
```

Measured on the reference machine (Qwen2.5-3B, 60-step Dolly LoRA):

| storage | size | bits / base-param |
|---|---:|---:|
| fp32 | 29.6 MB | 0.0766 |
| fp16 | 14.8 MB | 0.0382 |
| 8-bit | 8.56 MB | 0.0222 |
| **4-bit** | **4.88 MB** | **0.0126** |

**What the bit number means.** The base is 4-bit NF4 and the adapter is stored 4/8-bit.
`0.0126 bit/base-param` is the **marginal adapter cost**, not a 0.1-bit base. Count the base
and it is 4 bit/param (4-bit base) or 16 (bf16). The 0.1-bit VQ model was a failed
experiment; the sub-0.1 artifacts are adapters, prompts and coresets on a shared base.

**FAQ: “So are you using 0.1-bit?”** No. The deployed precision is **4-bit NF4** for the
base and 4/8-bit for the adapter. `0.0126` is an accounting ratio with the base shared. The
130 KB and 1.7 MB demos are separate artifacts (data and a linear model), not quantized
LLMs. There is no 0.1-bit general model here.

870 held-out Dolly rows, masked SFT loss: base 2.3752, original 2.3403, 8-bit 2.3401
(p = 0.81, no measurable loss), 4-bit 2.3420 (+0.0018, p = 0.001, 5% of the gain). A
second disjoint seed (pure LoRA, frozen base) gives 8-bit +0.00108 (p = 0.038) and 4-bit
+0.00114 (p = 0.029): a ~0.001 cost at both widths. The cost is small and real; 8-bit is
the conservative default. A second task, SST-2 (2,000 train / 400 test), goes from
**71.25%** base to **93.75%**, and
both packed adapters stay at **93.75%** (masked loss 0.7940/0.7942 vs 0.7930; p = 0.50 /
0.41). A third task, AG News (4-way), goes from 2.50% with the bare prompt (not followed)
to **62.25%**; 8-bit and 4-bit stay at **62.00%**. The adapter loads in 0.115 s. The LoRA
path is ~26% slower than the base on this card (8.75 vs 11.79 tok/s; a bf16 LoRA path
reaches 9.13); merge it into a bf16 base for zero per-token overhead, or accept the LoRA
path when storage/switching matters more than throughput.
Rank reduction is not viable: SVD to rank 8 loses 35.8% of the update.

**A prompt can be the better rung.** On the same base, a 441-byte 4-shot SST-2 prompt
scores 94.0% (adapter 93.75%) and a 242-byte 4-shot AG News prompt scores 61.75% (adapter
62.25%). If a few examples express the task, use the prompt; the adapter is for behaviour
a prompt cannot express, or when prompt tokens/context cost more than the storage.

**4-bit is a memory win, not a speed win.** On Qwen2.5-0.5B (bf16 fits on the card), bf16
generated **16.8 tok/s** vs **13.1** for 4-bit NF4 (~22% slower), while peak VRAM fell from
**1.00 GB to 0.49 GB**. The bit width is how a larger model fits; it does not make the
math faster.

### Measured on

Every speed number here comes from **one machine**: RTX 3050 Laptop 6 GB, Windows 11,
AC power, NF4 base, LoRA r=8, batch 1, Dolly-15k SFT. The four rows (3B/7B/14B/32B) are
listed in [`docs/MEASURED.md`](docs/MEASURED.md), together with the rules for adding one
from a different card:

```bash
python scripts/measured_row.py --row "<json line>" --config checkpoints/<model>
```

It validates the run (finished steps, `grad_layers == n_layers`, peak VRAM recorded),
derives the architecture fingerprint and prints the snippets for `fit.py`, `fitcore.py`
and the Space. Anything not in that table prints **"not measured"** in `streamft fit` ?
that is the point.

### Why 0.1-bit compression is not a universal target

A measured note on quantization vs distillation: post-hoc quantization dies below 3 bits;
a 340 KB student matches a 1.5B teacher on a narrow keyword task at 0.0019
bit/teacher-param; the same 4.4M student is within 1.3 points on DBPedia-14 but loses
22.5 on SST-2 and 47.9 on MNLI; open-ended generation dies at 3 bits. A cheap 45 KB TF-IDF
student is a **lower bound** on the 4.4M student's normalized skill (6 tasks, R² = 0.89),
not a predictor: QQP (C 0.21, S 0.49) breaks the near-identity. Details, caveats and
reproduction: [`docs/BEHAVIORAL_BITS.md`](docs/BEHAVIORAL_BITS.md).

## Verification

`python scripts/verify_gradients.py --model-dir checkpoints/qwen3b_bnb4bit --seq 128`

- **One resident block vs all 36 resident** (identical adapter weights, Qwen2.5-3B NF4,
  seq 128): the loss is identical to 10 decimals and the adapter gradients are
  **bit-identical** — 288 tensors, max relative error `0.0`, global relative error `0.0`.
  This is the streaming/recompute claim: dropping blocks and recomputing them in the
  backward changes nothing.
- **Manual block backward vs PyTorch autograd** through the same forward: global relative
  error **1.33%** (cosine `0.9998`). The same 1.33% appears with *all* blocks resident,
  so it is bf16 dequantisation precision, not the streaming path.
- Finite differences cannot check this on a 4-bit base: the LoRA delta is added to a bf16
  tensor, so a one-entry perturbation is below one bf16 ULP (measured for the same entry:
  `eps=2e-3` → `-0.12`, `eps=2e-2` → `+0.02`). The chunked-loss gradient itself is
  checked exactly in `tests/test_streamft_smoke.py`.

Raw output: `results/gradcheck.json` (git-ignored; the numbers above are copied from it).

## Compared with PEFT (matched runs)

Same data, same LoRA (r=8, alpha=16, q/k/v/o), same AdamW lr 1e-4, batch 1, same prompt
masking, same token-weighted masked eval on 8 held-out rows. PEFT uses
`prepare_model_for_kbit_training` with gradient checkpointing (the standard QLoRA setup).
Step times move by up to ~30% between runs on this laptop; quality is the stable number.

**3B (fits entirely):**

| | streamft (36 blocks resident) | PEFT / transformers |
|---|---:|---:|
| step time, median of 30 @ seq 512 | 0.94–1.24 s | **0.58–0.88 s** |
| peak VRAM | **3.85 GB** | 4.90 GB |
| eval before | 1.42087 | 1.42087 |
| eval after | **1.30720** (−0.1137) | 1.33002 (−0.0909) |

**7B / 14B / 32B (do not fit comfortably), after the masked-gradient fix:**

| model | step time, median @ seq 512 | peak VRAM | eval (start → after, 30 steps) |
|---|---:|---:|---:|
| 7B, streamft (28 blocks) | **1.55 s** | **5.51 GB** | 2.1928 → **1.1055** |
| 7B, PEFT | 8.96 s | 10.05 GB | 2.1914 → 1.1491 |
| 14B, streamft (6 blocks) | **5.86 s** | **3.86 GB** | 2.3748 → **0.9742** |
| 14B, PEFT | 18.56 s | 16.48 GB | 3 steps only |
| 32B, streamft (2 blocks, seq 1024) | **12.48 s** | **3.90 GB** | 1.8755 → **1.0008** |
| 32B, PEFT | fails before step 1 | — | — |

At 7B the crossover has already happened: PEFT's 4.4 GB of weights plus activations no
longer fit, so it spills (10.05 GB reserved) and pays ~5.8× the step time. Note that
streamft's 7B peak (5.51 GB) is *higher* than its 14B peak (3.86 GB) because the 7B run
kept all 28 blocks resident while the 14B run kept 6 — the window is a choice, and the
14B number is the one to copy for a 6 GB card.

**32B: PEFT does not reach the first step.** The 19.2 GB of weights load only by spilling
into shared memory (19.45 GB reserved on a 6 GiB card), and `prepare_model_for_kbit_training`
then dies allocating the fp32 norm copy: *CUDA out of memory. Tried to allocate 2.90 GiB;
18.00 GiB is already allocated.* The streaming engine runs the same checkpoint at
**12.5 s/step and 3.90 GB** (64/64 layers).

The 3B row is the honest answer to "is this faster?": **no** — on a model that fits, PEFT
is faster and uses 1.05 GB more. The 14B/32B rows are the design point: PEFT does not
OOM on Windows, it **spills into shared system memory** and pays 2.8–5.8× the step time.

### Correction: masked gradient (25 Oct, after an external review)

`chunked_lm_loss_and_backward_masked` divided every chunk by **that chunk's** valid-token
count. The reported loss was still the correct mean, so the curves looked right, but the
accumulated gradient over-weighted chunks with few unmasked tokens — i.e. the prompt-heavy
start of an SFT row. Measured on a toy model with an uneven mask: **112–427% relative
gradient error** at seq 512/1024 with 60% of the row masked (chunk 128), while balanced
masks were unaffected.

The fix divides every chunk by the **global** valid count (computed once, no-grad), so the
gradient is exactly d(mean CE over valid tokens)/dh. Two tests pin it:
`test_masked_chunked_gradient_matches_full_with_uneven_masks` and
`test_masked_chunked_handles_fully_masked_chunks`.

All SFT runs above were re-run with the fix. Quality held: 3B −0.114 (was −0.119),
7B −1.087 (was −1.083), 32B −0.875 (was −0.880); 14B is now 30 steps (−1.40) instead of
10. The numbers in the tables are the post-fix ones.

Reproduce: `python scripts/compare_peft.py --model checkpoints/qwen3b_bnb4bit --steps 30`
and `... --model checkpoints/qwen14b_nf4 --steps 30`.

### Reproducibility probe

Run this and compare the row; it is deterministic (seed 1234) apart from GPU-kernel
non-determinism:

```bash
streamft train --model-dir checkpoints/qwen3b_bnb4bit --steps 3 --seq 128 \
  --eval_chunks 1 --tag repro
```

Reference on the only machine tested so far (RTX 3050 Laptop 6 GB, torch 2.13.0+cu126,
bitsandbytes 0.50.2, Windows 11):

```text
losses        3.347981  3.647039  3.049560
ce_before     2.44565
ce_after      2.44568
peak_gb_max   3.068
adapter_sha_after  422424c59ef004c5
```

Different hardware will differ in the last digits (kernels, bf16 rounding). What must
match is the shape: the same three losses to ~1e-3, `grad_layers == n_layers`, and a
changed adapter hash. If it does not, that is a bug report worth posting with the
`streamft doctor` output.

## What this does not do

- **2k context does not fit** on 6 GB. It is ~0.6 GB short: with `--activation-cache cpu`
  the 2k peak is 6.60 GB, and what remains is the bf16 `lm_head` (1.56 GB). The 4-bit
  `--head nf4` path is **broken in the bitsandbytes version here**.
- **Longer context is not a quality lever by itself.** At a matched token budget
  (122,880 tokens), training at seq 512 gave a 2k-context CE of 1.2833; training at
  seq 2048 gave 1.3033. Data volume is the lever.
- **SFT `--seq` is a truncation cap**, not a padding target: one dataset row per step.
  Dolly rows are 43–1964 tokens, median 134. Never compute tokens/s from `--seq` on
  that path.
- Speed numbers are medians of warm runs, AC power only. Battery/Modern-Standby runs are
  invalid and were discarded.
- Single GPU, Windows/WDDM. Linux should work (no WDDM shared-memory cliff) but is not
  measured here.

## Repository layout

```
streamft/            the engine (package)
tools/finetune.py    one-command wrapper with the measured presets
scripts/             longctx_sweep.py — the LM-path measurement sweep
GUIDE_6GB_32B.md     practical guide
LONGCTX_FINDINGS.md  protocol, curve, the 2k wall, the E4 result, corrections
LONGCTX_PLAN.md      the measurement plan
ARTICLE.md           the DEV Community post source
results/             the engine appends results.jsonl here (git-ignored)
```

## The research line

This repository grew out of an attempt to replace dense attention with a **learned
block-sparse router**. That line is mostly a negative result and now lives on its own, as
an archive: **[byjameson/lcga-local](https://github.com/byjameson/lcga-local)** —
[`LCGA_FINDINGS.md`](https://github.com/byjameson/lcga-local/blob/main/LCGA_FINDINGS.md),
all experiments, every raw JSON, and the three measurement errors that were corrected.
The short version: on plain language modelling the learned router ties a budget-matched
static window; on retrieval data it reaches dense exact match at a 25% block budget.

History: this repository was previously named `lcga-local`; GitHub redirects the old
URLs. The engine lived at `experiments/exp54_disk_qlora.py` + `lcsa/` until the split.

## Articles

- [Fine-tuning a 32B model on a 6 GB laptop GPU — and the three measurements that nearly fooled me](https://dev.to/mycute_97b39c4e34027570f9/fine-tuning-a-32b-model-on-a-6-gb-laptop-gpu-and-the-three-measurements-that-nearly-fooled-me-2ojm)
- [Running a 4B model in 6 GB and training 3B parameters on a laptop](https://dev.to/mycute_97b39c4e34027570f9/running-a-4b-model-in-6-gb-and-training-3b-parameters-on-a-laptop-4al)
- [The same 32B post on Medium](https://medium.com/@myredditacc9393/fine-tuning-a-32b-model-on-a-6-gb-laptop-gpu-e749df447229)
- [Will it fit? A free planner for fine-tuning models that do not fit](https://dev.to/mycute_97b39c4e34027570f9/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu-o50)
- [Will it fit? on Medium](https://medium.com/@myredditacc9393/will-it-fit-a-free-check-before-you-download-19-gb-of-weights-9097e1fcf93f)

### Behavioral and watermark research

- [Behavioral vaccination: a 400-row mixed prime protects a behavior](https://dev.to/mycute_97b39c4e34027570f9/behavioral-vaccination-a-400-row-mixed-prime-protects-a-behavior-25ac)
- [4-bit Naive Bayes: symmetric fails on AG News, zero-point rescues it](https://dev.to/mycute_97b39c4e34027570f9/4-bit-naive-bayes-works-on-sms-fails-on-ag-news-4p62)
- [Behavioral epistasis: a watermark that works alone dies in every merge](https://dev.to/mycute_97b39c4e34027570f9/behavioral-epistasis-a-watermark-that-works-alone-dies-in-every-merge-1j9f)
- [The watermark's logit rises smoothly; its generation falls off a cliff](https://dev.to/mycute_97b39c4e34027570f9/the-watermarks-logit-rises-smoothly-its-generation-falls-off-a-cliff-4j0c)
- [Joint training did not fix the watermark merge: 8/8 alone, 0/8 at 0.75](https://dev.to/mycute_97b39c4e34027570f9/joint-training-did-not-fix-the-watermark-merge-88-alone-08-at-075-1hka)

**Spiking prototype:** a 4.3M-parameter LIF language model with synaptic current dynamics
learns TinyStories at 2.05 bpb after 300 steps and 1.94 after 1000 steps on the 6 GB card;
adding spike-frequency adaptation reaches 1.84. The rate control is weaker and saturates.
This is the from-scratch starting point for the brain-like line, not a claim that spiking
beats transformers. [`research/SPIKING_LM.md`](research/SPIKING_LM.md)

**Spiking rollback:** after a TinyStories -> WikiText shift, restoring the top 10% of
phase-A synapses by Fisher importance recovers 95.0%/94.1% of the old-task performance
while keeping 18.3% of the new-task gain on two seeds. Global rollback restores the old
task but discards all new learning. [`research/SPIKING_SHIFT.md`](research/SPIKING_SHIFT.md)

**Scaling check:** a 20.97M model trained for 10,000 steps / 164M tokens reached
1.327 bpb; the 4.3M model reached 1.336 on the same budget, and the larger model was
slightly worse on a fixed sample-quality probe. More parameters did not buy quality at
this budget; the next attempt needs 500M+ tokens. [`docs/SPIKING_PRODUCT.md`](docs/SPIKING_PRODUCT.md)

**Generative data diversity (first generative result):** a 4.3M model trained for
3,000 steps on 16M tokens. TinyStories-only: TinyStories **1.566** / WikiText
**4.455** bpb. Mixed 25% WikiText (12M TinyStories + 4M WikiText): TinyStories
**1.605** / WikiText **2.927** bpb. Longer mixed training (10k steps, 40M
tokens) improves both: TinyStories **1.464** / WikiText **2.794** bpb, versus
1.302/4.003 for the TinyStories-only 10k model. 50% and 75% WikiText add little
WikiText gain at a larger TinyStories cost (50%: 1.702/2.906; 75%:
1.734/2.820). The 21M/50M TinyStories-only models stay at WikiText 4.03, so
data diversity moves the general-text metric where parameter scaling did not.
The 21M 10k mix at matched 41M tokens (batch 32) is still worse on all three
bpb (1.669 / 3.011 / 3.384 vs 1.464 / 2.794 / 3.249); 5x parameters do not buy
quality at this budget. The batch-16 run (20.5M tokens) is worse still
(1.774/3.108). On a held-out
world-facts article set, the 4.3M mix reaches **3.249** bpb versus **3.949**
for TinyStories-only 10k, so the diversity gain transfers beyond WikiText-2.
A TinyStories-first curriculum does not dominate: it is slightly better on
TinyStories (1.439) but worse on WikiText (2.964) and world facts (3.374);
pure mix remains the best general-text tradeoff. Generation samples remain
locally repetitive and ungrammatical; 21M/50M TinyStories-only samples are more
story-shaped but still repetitive. Tighter sampling helps the TinyStories-only
4.3M model but hurts the mix model, so the generation gap is not just a
temperature problem. Lower bpb is not yet a generation-quality or deployment
win.
[`research/SPIKING_GENERATIVE.md`](research/SPIKING_GENERATIVE.md)

**Tiny Spiking Lab (product path, release blocked):** the local ingest workflow
improves a 76-article held-out domain from **5.99 to 4.45 bpb** in 300 steps
with 20% replay. It accepted the update on a 4096-token eval (old +2.55%), but
the full 8192-token eval shows old **+3.29%** and unmonitored WikiText **+7.1%**;
generation is formatting noise or corrupted text. A naive story ingest adapts
the target (space-story 2.19 -> 1.66, robot-themed samples) but full eval shows
TinyStories **+13.6%** and WikiText **+33.9%**. A safer recipe (70% replay from
a 20M TinyStories + 5M WikiText old mix, LR 5e-5, 120 steps) gives target
**2.19 -> 0.72**, TinyStories **+4.1%**, WikiText **-13.4%**. A second corpus
(Nova) replicates it: target **2.05 -> 0.60**, TinyStories **+4.4%**, WikiText
**-13.6%**. A second seed per corpus also replicates (TinyStories +4.1-4.4%,
WikiText -13.2-13.6%). The **narrow story-lab gate** passes on two synthetic
corpora and two seeds. **The real public-domain cross-book test fails**:
Alice-train -> Oz-valid passes bpb (target **-11.7%**, TinyStories **+3.5%**,
WikiText **-14.0%**) but generation alpha collapses on `Dorothy opened the
door` (min **0.534** seed 0, **0.259** seed 1, versus base 0.727-0.751). Gate
**BLOCKED**; the synthetic pass does not generalize to real text. A small-model
efficiency probe: 4.3M runs at **1,171 tok/s** on CUDA and **730 tok/s** on CPU
with **24.8 MB** peak VRAM; 21M runs at 563/204 tok/s with 88.3 MB. This is not
a general model; packaging/licence/model card are still missing.
[`research/SPIKING_TINY_LAB.md`](research/SPIKING_TINY_LAB.md)

**Tiny Transformer Chat (product path):** a **20.2M from-scratch transformer**
on the same clean chat data reaches chat bpb **1.359**, assistant loss **4.01**,
alpha min **0.731** and 0 empty samples. With role tokens banned at decoding it
passes the smoke gate; the 21M/50M spiking chat models remain BLOCKED at
2.29-2.31 bpb / 6.7-6.8 loss. A 49.2M transformer adds only 0.015 bpb, so the
20M checkpoint is the product model. It is a tiny, repetitive character model,
not a general assistant. Model card:
[`docs/TINY_TRANSFORMER_CHAT.md`](docs/TINY_TRANSFORMER_CHAT.md); local server:
`python -m streamft.chat_server`; demo: `scripts/chat_demo.ps1`.

**Sleep-like replay:** during a TinyStories -> WikiText phase, mixing 20% old windows
into every batch keeps old bpb at **2.079** (phase-A 2.077; naive 2.632) and retains
**99.2%** of the new-task gain. Replay is much better than post-hoc shadow rollback
here. [`research/SPIKING_REPLAY.md`](research/SPIKING_REPLAY.md)

**Replay + rollback:** controlled seed-1 arms confirm the mechanisms are not
additive. Replay alone keeps **98.9%** of the new gain; replay followed by Fisher
rollback keeps only **16.0%**. Replay prevents the forgetting; rollback after it
spends the new learning. [`research/SPIKING_REPLAY_ROLLBACK.md`](research/SPIKING_REPLAY_ROLLBACK.md)

**21M, two seeds:** replay-first holds at 21M. Replay keeps **99.6-99.8%** of the
new-task gain; replay+rollback keeps **6.5-12.0%**; replay+EWC keeps **99.3%**.
EWC is safe but does not beat simple replay.
[`research/SPIKING_REPLAY_EWC_21M.md`](research/SPIKING_REPLAY_EWC_21M.md)

**600-step phase B + generation probe:** at 21M and two seeds, replay keeps
**99.7%** of the new gain; naive samples collapse from **144-151 to 78-81 chars**
and repeated 4-grams quadruple. Replay keeps the phase-A profile; EWC is slightly
better on distinct-2/length but costs new-task bpb.

**50M long phase B:** replay keeps **98.2%** of the new gain and improves the
held-out continuation probe (content F1 **0.140 -> 0.182**); EWC keeps 91.7% and
lowers it to 0.115; naive drops it to 0.000. This is a small semantic signal, not
semantic competence. [`research/SPIKING_LONG_REPLAY.md`](research/SPIKING_LONG_REPLAY.md)

**Narrative-coherence benchmark:** 100 held-out four-way next-sentence cases —
4.3M **0.29**, 21M **0.41**, 50M **0.40** (chance 0.25). 50M does not beat 21M,
and EWC lambda 0.5/2.0 stays at 0.39/0.40 while costing new-task bpb. Parameter
scaling is not buying semantics. The decision is to revise architecture/data,
not just scale. [`research/SPIKING_SEMANTIC.md`](research/SPIKING_SEMANTIC.md)

**Top-down feedback (negative):** adding a zero-initialized last-layer-to-input
feedback path to the 21M model scored **0.39** narrative accuracy vs **0.45** for
the baseline after the same 500 steps, and slightly worse bpb. This topology did
not help. [`research/SPIKING_FEEDBACK.md`](research/SPIKING_FEEDBACK.md)

**Variable binding (non-TinyStories, negative):** on tiny three-statement stories
with **disjoint** train/test (person, color, object) triples, the 128-wide spiking
model reaches **0.168** (below the 0.25 chance) while the rate control reaches
**0.318**. Both memorize training triples and fail held-out binding; the current
architecture needs a memory/attention mechanism, not more parameters.
[`research/SPIKING_BINDING.md`](research/SPIKING_BINDING.md)

**Binding benchmark audit:** the first task is solvable **78%** by matching color
alone and **76%** by object alone, so a corrected conjunction split was built
(both attributes required, unique pair). There spike reaches **0.220**, rate
**0.326**, while a 2-layer transformer with dropout reaches **0.886** and a
4-layer transformer **1.000**. At 256 wide, spike/rate improve to **0.519/0.629**
on average over two seeds (seed range **0.466-0.572** / **0.502-0.756**) but are
**position-biased**: the spike model is best on the third statement (0.846/0.872)
and weak on the second (0.377/0.182). A 512-wide run is worse (**0.446/0.406**),
so width alone does not close the gap. A simple attention readout hurts
(**0.364/0.384**); a fixed pointer (**0.236/0.514**) and dropout
(**0.308/0.460**) also fail.

**Statement memory (positive):** a 256-wide spiking model with a statement-level
name/attribute memory plus a copy loss and a **self-supervised matching
attention** loss reaches **0.996/1.000** across two seeds on held-out
conjunction binding, is balanced across all three positions, and is
permutation-invariant (**0.996/1.000** consistency). The attention target is
derived from the query/statement attributes, not the answer. Without it, 6k
steps give **0.308/0.696**; at 12k the rate control reaches **0.874** but the
spike model is **0.470** and strongly position-biased (middle statement 0.159).
The architecture can bind; unsupervised credit assignment is still open.
[`research/SPIKING_BINDING_CONJ.md`](research/SPIKING_BINDING_CONJ.md)

**Continual binding (two seeds + hard split):** naive phase B on 16 new triples
drops old accuracy from **0.974/1.000** to **0.322/0.544** and held accuracy from
**0.730/0.926** to **0.240/0.280**. A 20% replay mix keeps old at
**0.992-0.996**, new at **1.000** and held at **0.788-0.866**. On a harder color
shift (old red/blue -> new green/yellow), naive old falls **0.930/1.000 ->
0.700/0.702** and held **0.336/0.000 -> 0.478/0.662**; replay reaches
**1.000/1.000/0.974** and **1.000/1.000/0.998**. Rollback 10% alone is weak
(0.392/0.674 old on the triple split) or destroys new colors (new 0.366 on color
seed 2); replay+rollback and replay+EWC are close but not consistently better.
Replay is the most reliable arm.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Without the matching-attention loss on harder splits (negative):** copy-only
color phase A is **0.332/0.214** (seed 0/2, at or below chance), and a no-overlap
2-name split collapses (**0.474/0.644**, even with the attention control). The
attention loss remains a required inductive bias at this budget; replay only
helps once phase A is learnable.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Four-statement stories:** dynamic memory, per-statement slots, slot reset, and
a hard name slot all stay at **0.214-0.236** old accuracy (majority **0.26**)
after 5k steps. Changing the slot key to the statement's **color+object
embeddings** and using **4-head attention** solves phase A: old **1.000**, all
four positions **1.000**, old permutation **1.000**. Copy-only phase B (no
matching-attention loss) keeps old **0.996/0.970**, new **1.000**, held
**0.872/0.728**, permutation **0.996/0.972**; replay reaches
**1.000/1.000** and held **0.926/0.760**. The structured key is the mechanism;
the task-specific attention loss is unnecessary and can hurt retention (naive
old **0.220/0.444** with it).
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Five-statement copy-only (positive):** five names, five attr-key slots,
sequence length 32. Phase A old **0.998** with all positions balanced and
permutation **1.000**. Copy-only phase B keeps old **0.950/0.994** (naive) or
**1.000** (replay), new **1.000**, and old permutation **0.944/0.994** or
**1.000**. Held-out composition is seed-sensitive (**0.244/0.356** on seed 0,
**0.982/0.980** on seed 1), so the next issue is composition, not retention.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Six-statement copy-only (positive, gate passed):** six statements, six names,
sequence length 37, six attr-key slots. Phase A old **1.000** with all positions
and permutation **1.000**. Copy-only phase B: naive old **0.998/0.914**, new
**1.000**, held **0.988/1.000**, permutation **0.998/0.924**; replay
**1.000/1.000**, held **1.000/0.998**, permutation **1.000/1.000**. Balanced
positions, permutation invariance, and held composition all pass on both seeds.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Seven-statement copy-only (positive, gate passed):** seven statements, seven
names, sequence length 42. Phase A old **1.000/0.998**, all positions and
permutation **1.000**. Copy-only phase B: naive old **1.000/0.986**, new
**1.000**, held **0.998/0.966**, permutation **1.000/0.986**; replay
**1.000/1.000**, held **0.988/0.994**, permutation **1.000/1.000**. The gate
passes on both seeds at seven statements.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Relaxed structure (partial):** variable filler lengths 0/1/2 with
token-triggered boundaries/read. Replay keeps old/new at **1.000** and
permutation at **1.000**, but held composition drops to **0.803/0.651**. A
0->1->2->mixed filler curriculum with 6k stories and 9k phase-A steps gives
**0.786/0.611** — no improvement. The gate fails on held-out composition, not on
old/new or balance; more filler-length practice does not fix it.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Factorized color-object attention (positive, gate passed):** separate color
and object keys/queries with multiplied attention distributions. Relaxed
structure: old/new/held **1.000/1.000/1.000**, permutation **1.000**, and held
**1.000** at every filler length, on both seeds. The earlier held gap was the
additive key, not data or training.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Eight-statement factorized relaxed (positive, gate passed):** eight names,
eight slots, sequence length 63, fillers 0/1/2. Replay: old/new/held
**1.000/1.000/1.000**, permutation **1.000**, held **1.000** at every filler
length on both seeds. The factorized mechanism scales to eight statements.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Multi-token answer (positive with curriculum):** answer is name + color, with
a query-color copy for the second token. Seed 0 reaches exact
**1.000/1.000/1.000** and permutation **1.000**; seed 1 initially fails the name
read (old **0.231**, new **0.354**, held **0.341**) though its color token is
**1.000**. A 3k red-only warmup + 6k red+blue + 3k phase B fixes both seeds to
exact **1.000/1.000/1.000**, permutation **1.000**, name/color 1.000 at every
filler length.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Learned boundaries, two-token answers (positive, gate passed):** statements
end with a generic `|`; learned write/read gates replace the known `.`/`?`
boundaries. Two seeds: phase-A old **1.000**, phase-B old/new/held
**1.000/1.000/1.000**, permutation **1.000**, name/color 1.000 at every filler
length. The remaining structure is a task-specific color/object factor and a
task-specific color-copy readout, not a general language model.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Raw boundaries, no separator (positive, gate passed):** statements are
concatenated with no separator token at all. A write candidate on the object
token plus a context read gate gives the same result: two seeds, phase-A old
**1.000**, phase-B old/new/held **1.000/1.000/1.000**, permutation **1.000**,
name/color 1.000 at fillers 0/1/2. The gate is supervised with object positions,
so this is learned write/read classification, not unsupervised segmentation.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Natural-language templates (positive, gate passed):** English-like
`tom has a big red ball.` / `who has a big red ball? mia red.` with learned
`.`/`?` gates. Two seeds: phase-A old **1.000**, phase-B old/new/held
**1.000/1.000/1.000** (seed 1 held **0.9989**), permutation **1.000**, all
filler cells above 0.95. Still a fixed 24-word template set, not free-form
language.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Wide natural-language templates (positive, gate passed):** verbs, articles
and adjectives vary (`tom owns the old red ball.` / `who found a big yellow
car? sam yellow.`). Two seeds: phase-A old **1.000**, phase-B old/new/held
**1.000/1.000/1.000**, permutation **1.000**, all 18 split/filler cells 1.000.
Still a fixed 31-word template set, not free-form language.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Paraphrase templates (scoped positive, 4 pairs):** names and queries move
(`tom has a red ball.` / `the red ball belongs to tom.` / `whose red ball is
it?`). Category states plus a template curriculum and 50% replay give two seeds
old/new/held **1.000/1.000/1.000**, permutation **1.000**. The full 16-pair set
is seed-sensitive and stays a negative result.
[`research/SPIKING_BINDING_CONTINUAL.md`](research/SPIKING_BINDING_CONTINUAL.md)

**Curriculum without the attention loss (negative):** a padded 1 -> 2 -> 3
statement curriculum trained with the copy loss only reaches spike **0.412**
(permuted 0.392, last-position-biased) and rate **0.212**. It does not replace
the self-supervised matching-attention loss at this budget.
[`research/SPIKING_BINDING_CONJ.md`](research/SPIKING_BINDING_CONJ.md)

## Support the work / Destek

This project is developed on a 6 GB laptop GPU. If you find it useful and want
to support further experiments, donations go toward more compute and larger
runs. There is no paid support, token, or investment offer.

- **EVM-compatible networks** (Ethereum, BNB Smart Chain, Polygon, Optimism,
  Arbitrum One, Base, Avalanche C-Chain):
  `0xa4dEc84Fe6F13c8A3b8707446D6892a6E272285c`
- **Solana:**
  `EDqs8w4JkS5vAAfxmsB7fAQwoauEYufYXvMWFW9aYPdc`
- **Bitcoin:**
  `bc1qsq4mdtf0dlfvppllqwdaknxrq6xkjgsrx6ctgm`

Please double-check the network and the full address before sending; crypto
transfers are irreversible. Donations are voluntary and do not buy influence
over results or reviews.

## License

Apache-2.0 — see [`LICENSE`](LICENSE).
