Metadata-Version: 2.4
Name: stillwarm
Version: 0.1.1
Summary: Automatic KV-cache session persistence for llama-server: save the model's 'notes' to disk, restore them after a restart, refuse incompatible files.
Author-email: Vimal Nakrani <vimalnak@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Vimal Nakrani
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Keywords: llama.cpp,llama-server,kv-cache,local-llm
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# stillwarm

**Your local LLM re-reads everything after every restart. stillwarm fixes that.**

llama.cpp's `llama-server` can save a conversation's KV cache (the model's
per-token "notes") to disk and load it back — but nothing does it automatically,
and nothing checks that the file you're loading matches the server you're
loading it into. stillwarm is a **zero-dependency** Python library + small CLI:

- **restore-before-request, save-after-response** — sessions survive restarts;
- a **metadata sidecar** per save — incompatible files are **refused with a
  reason, never silently misloaded**;
- a **silently-ineffective guard** — if a restore "succeeds" but the server
  re-prefills anyway (the measured SWA-without-`--swa-full` trap), you get a
  loud warning instead of silently paying cold-start cost;
- **LRU pruning** under a disk budget (KV files are ~128 KiB/token for an 8B
  model — a 32K-token session is ~4.3 GB);
- **`verify`** — an integrity mode that byte-checks a restored state against a
  probe captured at save time.

Measured on an M3 Max (llama.cpp b9871, Llama-3.1-8B Q4_K_M, same-session
thermally controlled): resuming a session from disk instead of re-reading it is
**71x faster at 8K tokens, 159x at 32K, 201x at 64K** (restore + first-token
time vs cold prefill), and ~50x cheaper in energy at 32K. Even with a purged
page cache, restore wins for any document longer than ~300 tokens.

## Quickstart

```python
from stillwarm import LlamaServer, ServerProfile, Session

# llama-server ... -fa on --slot-save-path ~/.stillwarm/   (must be running)
server  = LlamaServer("http://127.0.0.1:8080")
profile = ServerProfile(model_sha256="<sha256 of your .gguf>",
                        flash_attn="on", cache_type_k="f16", cache_type_v="f16",
                        swa_full=False, ctx_size=16384)
sw = Session(server, profile, cache_dir="~/.stillwarm",
             server_build_tag="b9871", disk_budget_gb=50)

r = sw.ask("paper-review", document + "\n\nQ: Summarize.\nA:", n_predict=256)
# ... llama-server restarts ...
r = sw.ask("paper-review", document + "\n\nQ: And the weaknesses?\nA:", n_predict=256)
print(r.restored, r.prompt_n)   # True, ~a few dozen tokens instead of thousands
```

`cache_dir` must be the directory passed to `--slot-save-path`. CLI:

```
stillwarm list  ~/.stillwarm            # sessions, sizes, ages
stillwarm inspect ~/.stillwarm/paper-review.bin
stillwarm prune ~/.stillwarm --budget-gb 50 [--dry-run]
```

## Compatibility contract (every rule measured, not guessed)

From a 480-row benchmark matrix ([stillwarm-bench], Block D portability taxonomy):

| you change... | what happens | stillwarm's response |
|---|---|---|
| flash-attention on<->off | restore fails (400) both directions | **REFUSE** up front |
| KV cache type (f16/q8_0/q4_0) | layout mismatch | **REFUSE** |
| model file | meaningless state | **REFUSE** |
| `--swa-full` (SWA models) | restore "works" but silently re-prefills | **REFUSE** + runtime guard |
| server ctx < saved tokens | restore fails (fit rule) | **REFUSE** with a clear message |
| server ctx >= saved tokens | works | OK |
| `-ngl` (GPU layers) | works (measured both directions) | OK — not part of the cache key |
| llama.cpp build (+-5 weeks measured) | works both directions | **ADVISORY** warning only |

## `verify` — honest semantics

At save time, stillwarm writes the slot file **first** (the file is the
pre-probe state), then generates a 64-token pinned-greedy probe and stores its
hash in the sidecar. `restore(..., verify=True)` restores, re-runs the probe,
compares hashes, then **re-restores** so the served state is exactly the
restored state.

This is a **determinism** check (does the restored state reproduce the
save-time continuation?), deliberately **not** a *cold-equivalence* check
(does it match a full recompute?). Cold-equivalence measurably fails without
any corruption: different prefill batch splits flip marginal tokens, and
quantized caches amplify this to near-certainty (q4_0: 5/5 divergent at 8K in
[stillwarm-bench] Block C) while both outputs stay coherent. A verify failure
therefore means *the file/model/config does not reproduce what was saved* —
a real integrity signal, not batching noise.

Cost: ~2x restore time + one 64-token decode (~1.5 s for an 8B model).

## Non-goals (v1 fence)

- **Not a serving framework** — one machine, one llama-server.
- **No multi-machine cache sharing**; no vLLM (LMCache owns that space).
- **llama-server only**: Ollama and LM Studio do not expose the slot
  save/restore endpoints this needs.
- No compression research. Small and finished beats large and abandoned.

## Platforms

- **macOS (Apple Silicon)** — tested; all measured numbers come from an M3 Max
  against llama.cpp **b9871**.
- **Linux** — expected to work (same llama-server HTTP API; the demo Space runs
  the identical mechanism on Linux/CPU); not part of the measured matrix.
- **Windows** — untested. The code is audited for portability (pathlib
  throughout, atomic writes via `os.replace`, no POSIX-only stdlib modules, and
  pruning tolerates `PermissionError` on files Windows holds open) — reports
  welcome.

## Tests

`pytest` — unit tests run anywhere; integration tests need a `llama-server`
binary and a tiny GGUF (defaults: SmolLM2-135M-Instruct Q8_0, ~145 MB; override
via `STILLWARM_TEST_SERVER_BIN` / `STILLWARM_TEST_MODEL`; they skip if absent).

## License

MIT — see [LICENSE](LICENSE).

[stillwarm-bench]: https://github.com/vimalnakrani08/stillwarm-bench
