Metadata-Version: 2.4
Name: hearth-llm
Version: 0.1.0
Summary: Keeps your local LLM contexts warm: persistent KV-cache proxy for llama-server
Author: Daniel Soromou
License: MIT
Project-URL: Homepage, https://github.com/caporalCoder/hearth
Project-URL: Issues, https://github.com/caporalCoder/hearth/issues
Project-URL: Changelog, https://github.com/caporalCoder/hearth/blob/main/CHANGELOG.md
Keywords: llm,llama.cpp,kv-cache,inference,proxy,ollama
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Internet :: Proxy Servers
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# hearth 🔥

**Keeps your local LLM contexts warm.**

[![CI](https://github.com/caporalCoder/hearth/actions/workflows/ci.yml/badge.svg)](https://github.com/caporalCoder/hearth/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/hearth-llm)](https://pypi.org/project/hearth-llm/)
[![Python ≥ 3.10](https://img.shields.io/badge/python-%E2%89%A53.10-blue)](https://pypi.org/project/hearth-llm/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

hearth is a zero-dependency caching proxy for
[llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` that
persists KV-caches to disk and restores them across sessions — and across
server restarts.

## The pain point

Local inference has a cold-context problem. Every time you start a new
session with llama.cpp or anything built on it, your entire context —
the codebase you loaded, the contract you're analyzing, the long
conversation you were having — is re-processed token by token from
scratch. On consumer hardware that's seconds to minutes of prompt
processing, thrown away the moment the process exits.

The KV-cache that represents all that work is just memory. hearth
snapshots it to disk, indexes it by conversation content, and restores
the longest matching prefix automatically on your next request. Quit,
reboot, come back tomorrow — your context is still warm.

## How it works

```
your app (any OpenAI client)
        │
        ▼
   hearth proxy  ──────────►  llama-server
   :8737                      :8738
        │                        │
        ▼                        ▼
   manifest.json  ◄────────  *.kv snapshots
        (~/.hearth/cache — shared directory)
```

1. Each chat request's messages are hashed as a **rolling prefix chain**
   (one digest per message boundary, seeded by model id).
2. hearth finds the longest saved prefix and tells llama-server to
   **restore** that snapshot into the slot via its native
   `/slots/{id}?action=restore` API. llama-server's own token-level
   prefix reuse then skips everything already computed.
3. The request is forwarded unchanged (streaming included) and the
   assistant's reply is captured on the way through.
4. After the response, the slot's new KV state is **saved** under the
   hash of the extended conversation — ready for the next turn, or the
   next restart.

Snapshots are evicted LRU under a configurable size cap. Nothing about
your client changes: it's the same OpenAI-compatible endpoint, plus two
response headers (`X-Hearth-Cache: hit|miss`, `X-Hearth-Restored-Msgs`).

## Install

Requires Python ≥ 3.10 and `llama-server` on your PATH
(`brew install llama.cpp`). hearth itself has **zero dependencies**.

```bash
pip install hearth-llm        # or: pipx install hearth-llm
# or run straight from a checkout — it's stdlib-only:
python3 -m hearth --help
```

## Quickstart

```bash
# Easiest: reuse a model you already pulled with Ollama
hearth serve --ollama gemma4:e2b

# Or any GGUF file
hearth serve --model ~/models/qwen2.5-7b-instruct-q4_k_m.gguf

# Or attach to a llama-server you manage yourself
# (it must run with --slot-save-path ~/.hearth/cache)
hearth serve --upstream http://127.0.0.1:8080
```

Then point any OpenAI client at `http://127.0.0.1:8737/v1`.

```bash
hearth models        # local Ollama models hearth can serve
hearth ls            # saved snapshots (size, tokens, hits, age)
hearth rm <key>      # drop one snapshot
hearth rm --all      # clear the cache
curl localhost:8737/hearth/stats   # hit/miss counters
```

## Benchmark

`scripts/bench.py` simulates the real workflow: load a large document,
ask a question, **kill the server entirely**, restart, ask a follow-up.

```bash
python3 scripts/bench.py --ollama gemma4:e2b
```

Cold = full prompt re-processing. Warm = hearth restores the snapshot
from disk. Measured on an Apple M3 Max (llama.cpp b10090):

|                        | cold  | warm  |
|------------------------|-------|-------|
| prompt tokens computed | 1,333 | 59    |
| prompt processing time | 209ms | 6ms   |

The warm session — a **brand-new llama-server process** — skipped 96%
of prompt processing because the KV state came off disk. The absolute
savings scale with model and context size: on a 7B model with a
50k-token codebase loaded, that difference is minutes, not milliseconds.

> Note: Ollama blobs for multimodal models (e.g. Gemma) bundle vision
> tensors that plain llama-server can't load; use a text-only GGUF for
> those. Text-only Ollama models work directly via `--ollama`.

## Status / roadmap

MVP. Single slot, requests serialized. Planned:

- [ ] multi-slot scheduling (parallel conversations, slot affinity)
- [ ] snapshot forking (branch a conversation from any saved prefix)
- [ ] token-level (not message-level) prefix matching via `/tokenize`
- [ ] cross-machine snapshot sharing (same model + build)
- [ ] `hearth warm <file>` — pre-bake a document/codebase into a snapshot
- [ ] TTL-based eviction policies alongside LRU

## Development

```bash
python3 -m unittest discover -s tests   # no model needed; uses a fake llama-server
```

See [CONTRIBUTING.md](CONTRIBUTING.md) — benchmark reports from
different hardware are especially welcome.

## License

[MIT](LICENSE) © 2026 Daniel Soromou
