Metadata-Version: 2.4
Name: innards
Version: 0.2.0
Summary: See inside open-weight LLMs per message: KV cache, context growth and memory, predicted vs measured, across MHA, GQA, MLA, sliding-window and hybrid models.
Keywords: llm,kv-cache,transformers,attention,gqa,mla,observability,profiling,inference
Author: Jai
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Dist: httpx>=0.28.1
Requires-Dist: huggingface-hub>=1.5
Requires-Dist: psutil>=5.9 ; extra == 'hf'
Requires-Dist: torch>=2.4 ; extra == 'hf'
Requires-Dist: transformers>=5.0 ; extra == 'hf'
Requires-Python: >=3.10
Project-URL: Homepage, https://pypi.org/project/innards/
Project-URL: Documentation, https://pypi.org/project/innards/#description
Provides-Extra: hf
Description-Content-Type: text/markdown

# Innards

![Innards: one input, every architecture, zero slowdown](https://<public-host>/innards/infographic.png)

Innards runs open-weight LLMs and shows, for every message, what happens inside: tokens, context growth, KV cache per layer, memory and timing.
It predicts KV-cache size from the model config, measures the real cache, and explains what each attention design (MHA, GQA, MLA, sliding window, hybrid) changed from its predecessor.
Observation runs in background threads beside generation, so the model generates exactly as it would without Innards.

## Install

<!-- test: install (checked by the fresh-environment wheel install in CI) -->
```bash
pip install innards            # core: KV calculator, lineage, schema, console sink
pip install "innards[hf]"      # + torch, transformers, psutil for in-process Hugging Face models

uv add innards                 # inside a uv project
uv add "innards[hf]"
uv pip install "innards[hf]"   # into an existing environment
```

The core install has two small dependencies (`httpx`, `huggingface-hub`) and works on any Python 3.10+. The `hf` extra adds PyTorch and Transformers.

## Quickstart

Chat with a model and get one record per message:

<!-- test: requires=hf,network -->
```python
from innards import Session

with Session(
    "Qwen/Qwen3-0.6B", max_new_tokens=48, sinks=["console"], chat_template_kwargs={"enable_thinking": False}
) as s:
    turn = s.chat("Explain paged attention in 3 lines")
    print(turn.output.text)
    print(turn.tokens.context_after, turn.kv.measured_bytes, turn.kv.predicted_bytes)

    turn = s.chat("Now in one line")
    print(turn.tokens.cached, "prompt tokens reused from the KV cache")
```

Send the same messages to models with different attention designs and compare:

<!-- test: requires=hf,network -->
```python
from innards import compare

results = compare(
    models=["gpt2", "Qwen/Qwen3-0.6B"],
    script=["What is a KV cache?", "Why does it grow with context?"],
    max_new_tokens=16,
)
print(results.table())
```

Predict KV size without loading weights:

<!-- test: requires=network check-output -->
```python
from innards.kv import predict

p = predict("Qwen/Qwen3-0.6B", context_tokens=32768, kv_dtype="float16")
print(p.formula)  # 2 × 28 layers × 8 KV heads × 128 head dim × 2 B = 114,688 B/token
print(p.kv_bytes)  # 3758096384
```

## Imports

| Import | What it is |
| --- | --- |
| `from innards import Session` | Synchronous session: chat with one model, one `TurnRecord` per message |
| `from innards import AsyncSession` | Async session: streams tokens while metrics arrive from background workers |
| `from innards import compare` | Sends the same messages to several models and aligns the results |
| `from innards import CompareResult` | Result of `compare`: `turns[model]`, `rows()`, `table()`, `to_dict()` |
| `from innards import Turn` | Alias of `TurnRecord`, the return type of `Session.chat` |
| `from innards.kv import predict` | Architectural KV bytes, capacity and per-layer breakdown from a config, no weights |
| `from innards.kv import KVPrediction` | Result of `predict` |
| `from innards.kv import ModelSpec, LayerKV` | Normalized architecture facts and per-layer cache geometry |
| `from innards.kv import load_config, spec_from_config` | Read a config (Hub id, path, dict) and normalize it |
| `from innards.kv import dtype_bytes, normalize_dtype` | KV element types: `float32`, `float16`, `bfloat16`, `float8` |
| `from innards.lineage import explain` | What an architecture changed from its predecessor, and why |
| `from innards.lineage import LineageEntry, chain, families, get` | Lineage entries, predecessor chain, family list and lookup |
| `from innards.schema import TurnRecord` | The per-message record (schema v1.0) |
| `from innards.schema import ModelInfo, RuntimeInfo, TokenCounts, ContextComposition, KVInfo, MemoryInfo, TimingInfo, OutputInfo, ObserverInfo` | The record's sections |
| `from innards.sinks import Sink, ConsoleSink, CallbackSink, make_sink, register_sink` | Where records go; any `write(record)`/`close()` object or plain callable works |
| `from innards.observe import Observer, BoundedEventQueue` | The non-blocking observer and its drop-counting queue |
| `from innards.backends import Backend, get_backend, register_backend, ContextFullError` | Engine adapter interface and registry |
| `from innards.backends.hf import HFBackend` | In-process Hugging Face Transformers backend (needs `innards[hf]`) |
| `from innards.strategies import FullHistory` | Context strategy that keeps every turn and reuses the KV cache |

## Functions

### `Session`

```text
Session(model: str, backend: str = "hf", strategy: str = "full_history",
        sinks: Iterable[str | Sink | Callable] | None = None, *, system: str | None = None,
        observe: bool = True, max_new_tokens: int = 256, record_timeout: float = 30.0,
        queue_size: int = 4096, run_id: str | None = None, session_id: str | None = None,
        load: bool = True, **backend_kwargs) -> Session
```

- `model`: Hugging Face Hub repo id or local path.
- `backend`: `"hf"` (in-process Transformers).
- `strategy`: `"full_history"`: every turn is kept, and the KV cache from earlier turns is reused.
- `sinks`: where finished records go, e.g. `["console"]`. Records are also returned by `chat`.
- `system`: optional system prompt.
- `observe`: `False` turns observation off entirely (no events, no worker threads).
- `max_new_tokens`: generation budget per turn (greedy decoding).
- `record_timeout`: how long `chat` waits for the background record after generation has finished.
- `**backend_kwargs`: for `hf`: `device` (`"cpu"`, `"cuda"`, `"mps"`), `dtype`, `chat_template_kwargs`, `generation_kwargs`, `revision`, `token`, `trust_remote_code`.

Returns a session bound to one model. Use it as a context manager, or call `close()`.

<!-- test: requires=hf,network check-output -->
```python
from innards import Session

s = Session("gpt2", device="cpu", max_new_tokens=16)
print(s.backend.spec.architecture)  # mha
s.close()
```

### `Session.chat`

```text
Session.chat(message: str, *, timeout: float | None = None) -> TurnRecord
```

- `message`: the user message.
- `timeout`: seconds to wait for the background record (default: `record_timeout`).

Returns the `TurnRecord` (also available as `Turn`) with output text, tokens, KV, memory and timing.

<!-- test: requires=hf,network -->
```python
from innards import Session

with Session("gpt2", max_new_tokens=16) as s:
    turn = s.chat("The key-value cache")
    print(turn.kv.cache_tokens, turn.kv.measured_bytes == turn.kv.predicted_bytes)  # e.g. 23 True
```

### `Session.stream` and `Session.turn_metrics`

```text
Session.stream(message: str) -> Iterator[str]
Session.turn_metrics(timeout: float | None = None) -> TurnRecord
```

- `stream` yields text as it is generated; closing the iterator early cancels generation.
- `turn_metrics` returns the record for the most recent turn, waiting for the background worker if needed.

```text
Session.progress -> TurnProgress | None
Session.reset() -> None
```

- `progress` gives the running turn's `turn`, `prompt_tokens`, `cached_tokens`, `generated_tokens` and `done`, plus `context_tokens`. `generated_tokens` advances as `stream` is consumed, so a live view can show the context growing token by token. After the turn ends it holds the final counts.
- `reset` forgets the conversation and the kept cache. Turn numbers keep counting. KV growth (`kv.growth_bytes`) and `kv.turns_to_limit` restart with the next turn.

<!-- test: requires=hf,network -->
```python
from innards import Session

with Session("gpt2", max_new_tokens=16) as s:
    for piece in s.stream("Attention is"):
        print(piece, end="", flush=True)
    print()
    print(s.turn_metrics().timing)
```

### `AsyncSession`

```text
AsyncSession(model: str, backend: str = "hf", **session_kwargs) -> AsyncSession
async AsyncSession.stream(message: str) -> AsyncIterator[str]
await AsyncSession.turn_metrics(timeout: float | None = None) -> TurnRecord
await AsyncSession.chat(message: str) -> TurnRecord
```

Takes the same arguments as `Session`. The model loads and generates in worker threads, so the event loop is never blocked.

<!-- test: requires=hf,network -->
```python
import asyncio
from innards import AsyncSession


async def main():
    async with AsyncSession("gpt2", max_new_tokens=16) as s:
        async for token in s.stream("Explain paged attention"):
            print(token, end="", flush=True)  # generation never waits on Innards
        turn = await s.turn_metrics()  # arrives from background workers
        print()
        print(turn.kv.measured_bytes)


asyncio.run(main())
```

### `compare`

```text
compare(models: Sequence[str], script: str | PathLike | Sequence[str], backend: str = "hf",
        **session_kwargs) -> CompareResult
```

- `models`: Hub repo ids or local paths, run one at a time.
- `script`: a list of user messages, one message, or a JSON file with a list of messages (strings or `{"content": ...}` objects, optionally under `"messages"`).
- `**session_kwargs`: passed to every `Session`.

Returns a `CompareResult`: `turns[model]` is the list of records; `rows()`, `table()` and `to_dict()` align them by turn.

<!-- test: requires=hf,network -->
```python
import json, pathlib, tempfile
from innards import compare

path = pathlib.Path(tempfile.mkdtemp()) / "script.json"
path.write_text(json.dumps({"messages": ["Hello", "What did I just say?"]}))
result = compare(["gpt2"], path, max_new_tokens=8)
print(result.rows()[0]["kv_measured_bytes"])
```

### `predict`

```text
predict(model_or_config: str | PathLike | dict | PretrainedConfig | ModelSpec, context_tokens: int,
        kv_dtype: str = "float16", *, memory_bytes: int | None = None, batch_size: int = 1,
        revision: str | None = None, token: str | None = None) -> KVPrediction
```

- `model_or_config`: Hub repo id (only `config.json` is downloaded), a config path or directory, a dict, a `PretrainedConfig` or a `ModelSpec`.
- `context_tokens`: tokens resident in the cache.
- `kv_dtype`: `float32`, `float16`, `bfloat16` or `float8` (aliases `fp32`, `fp16`, `bf16`, `fp8`).
- `memory_bytes`: optional KV budget; sets `capacity_tokens`.

Returns a `KVPrediction` with `bytes_per_token`, `kv_bytes`, `recurrent_state_bytes`, `total_bytes`, `per_layer_bytes`, `by_layer_type`, `capacity_tokens`, `formula` and `notes`. "Architectural" KV is what the attention design needs; an engine may hold more, and Innards reports the gap instead of hiding it. Sliding-window layers hold `window - 1` tokens: the current token attends to itself plus `window - 1` cached ones.

<!-- test: requires=network check-output -->
```python
from innards.kv import predict

p = predict("deepseek-ai/DeepSeek-V2-Lite", context_tokens=32768, memory_bytes=8 * 2**30)
print(p.architecture, p.bytes_per_token, p.capacity_tokens)  # mla 31104 163840
```

A config dict works offline:

<!-- test: check-output -->
```python
from innards.kv import predict

gemma_like = {
    "model_type": "gemma3_text",
    "num_hidden_layers": 26,
    "num_attention_heads": 4,
    "num_key_value_heads": 1,
    "head_dim": 256,
    "sliding_window": 512,
    "sliding_window_pattern": 6,
    "max_position_embeddings": 32768,
}
p = predict(gemma_like, context_tokens=8192, kv_dtype="bf16")
print(p.by_layer_type)  # {'global': 33554432, 'sliding': 11511808, 'recurrent': 0}
```

### `explain`

```text
explain(model_or_architecture: str | PathLike | dict | ModelSpec) -> LineageEntry
```

- `model_or_architecture`: a family name or alias (`"mha"`, `"mqa"`, `"gqa"`, `"mla"`, `"sliding_window"`, `"hybrid"`, `"ssm"`, `"fp8"`), a Hub repo id, a config path or dict, or a `ModelSpec`.

Returns a `LineageEntry` with `family`, `predecessor`, `change`, `problem_solved`, `trade_off`, `kv_formula` and `reference`. For a model, `detail` describes that model's own numbers.

<!-- test: check-output -->
```python
from innards.lineage import explain

entry = explain("gqa")
print(entry.predecessor, "->", entry.family)  # mha -> gqa
print(entry.problem_solved)
```

<!-- test: requires=network -->
```python
from innards.lineage import explain

print(explain("deepseek-ai/DeepSeek-V2-Lite"))  # MLA: what it changed from GQA and why
```

### `TurnRecord`

```text
TurnRecord.to_dict() -> dict
TurnRecord.to_json(**json_kwargs) -> str
TurnRecord.from_dict(data: dict) -> TurnRecord
TurnRecord.from_json(text: str) -> TurnRecord
```

One JSON record per message, schema v1.0: `model`, `runtime`, `tokens`, `context`, `kv`, `memory`, `timing`, `output` and `observer`. Missing measurements are `null`, never zero. Readers accept any 1.x record and reject other major versions.

<!-- test: check-output -->
```python
from innards.schema import TurnRecord

record = TurnRecord.from_dict(
    {
        "schema_version": "1.0",
        "session_id": "s-1",
        "turn": 1,
        "timestamp": "2026-10-01T10:42:07Z",
        "model": {"id": "Qwen/Qwen3-0.6B", "architecture": "gqa"},
        "runtime": {"backend": "hf"},
        "kv": {"predicted_bytes": 378929152, "measured_bytes": 378929152, "measured_source": "hf_cache"},
    }
)
print(record.kv.measured_bytes, record.memory.peak_bytes)  # 378929152 None
```

## CLI

<!-- test: cli -->
```bash
innards chat --model gpt2 --backend hf                    # chat; per-turn metrics on stderr
innards chat --model Qwen/Qwen3-0.6B --no-think --json    # TurnRecord JSON after each turn
innards predict --model Qwen/Qwen3-0.6B --context 32768 --kv-dtype fp8
innards predict --model deepseek-ai/DeepSeek-V2-Lite --context 32768 --memory 16GiB --json
innards explain sliding_window
```

`innards chat` streams the reply to stdout and prints a three-line summary per turn to stderr: prompt, cached and generated tokens; KV measured vs predicted by layer type; TTFT, prefill, decode speed, peak memory and dropped events. Type `/reset` to start over, `/exit` to quit. Options: `--system`, `--max-new-tokens`, `--device`, `--dtype`, `--no-think`, `--json`, `--no-metrics`.

## Supported

**Architectures** (KV calculator and HF measurements):

| Family | What is cached | Examples |
| --- | --- | --- |
| MHA | K and V for every head | GPT-2, Phi-3-mini (its config also sets a 2,047-token window on every layer) |
| MQA | One shared K/V head | GPT-BigCode, Falcon-7B |
| GQA | K/V per head group | Qwen3, Llama 3.2 |
| MLA | Compressed latent + RoPE key | DeepSeek-V2, DeepSeek-V3 |
| Sliding window (+ global) | Local layers keep `window - 1` tokens | Gemma 2/3, GPT-OSS, OLMo 3, Cohere 2, EXAONE 4, Mistral 7B v0.1; Llama 4 chunked layers |
| Hybrid | Attention KV in a few layers + fixed recurrent state | Granite 4.0-H, LFM2, Qwen3-Next, Qwen3.5/3.6 (Gated DeltaNet), Jamba, Bamba, Falcon-H1, Nemotron-H |
| SSM | Fixed recurrent state only | Mamba, Mamba-2 |

Legacy config keys (`n_layer`, `n_head`, `n_embd`, ChatGLM, Falcon, MPT) and nested `text_config` (multimodal checkpoints) are read too. KV dtypes: fp32, fp16, bf16, fp8.

**Backends:** `hf`, in-process Hugging Face Transformers 5.x on CPU, CUDA or Apple MPS. Remote engines (vLLM, Ollama, llama.cpp) and managed APIs use the same `TurnRecord` and plug in through `innards.backends.register_backend`. They are planned, not shipped yet.

**Sinks:** `console` (text or JSON lines). Any object with `write(record)` and `close()`, or a plain callable, also works as a sink.

**Environments:** anywhere Python runs: local terminal, Jupyter and VS Code, Docker, cloud VMs and containers (AWS, GCP, Azure), Kaggle and Colab.

**Python:** 3.10, 3.11, 3.12, 3.13 and 3.14. Python 3.15 is tested against its release candidates, with support from the final release (1 October 2026). The core install runs on 3.15 today. The `hf` extra needs PyTorch wheels for your Python version, and PyTorch has none for 3.15 yet.

## Non-blocking guarantee

Innards never sits in the generation path:

- **Hot path**: the only Innards code inside the token loop takes a timestamp and offers a small event to a bounded queue. It makes no network calls, disk writes, tensor copies or device synchronization.
- **Bounded queues that never block**: when the per-token queue is full, the event is dropped and counted, never waited on. Every record carries `observer.dropped_events`, so gaps are visible. The two per-turn events (start and end, sent outside the token loop) are never dropped, so every message gets a record.
- **Background workers**: KV math, cache measurement (tensor shapes and dtypes only, `numel × element_size`), memory sampling and sink writes all run in background threads.
- **Fail-open**: if a collector or sink fails, the error is logged and counted, and generation continues.
- **Same output**: greedy output with Innards is identical to plain `model.generate`, which the test suite checks.

**Overhead budget:** throughput within 1% of running without Innards, and no added time to first token. `scripts/overhead_bench.py` measures it (same model and prompts, Innards on vs off, interleaved runs, medians).

## License

Apache-2.0
