Metadata-Version: 2.4
Name: thinctx
Version: 1.0.1
Summary: ThinCtx: inference runtime and OpenAI-compatible server for the ThinCtx models (a small GPU memory for the context that grows 20 to 30 times more slowly than the original model's KV cache).
Author: Jan Rosecký (thinletter)
License: PolyForm-Small-Business-1.0.0
Project-URL: Homepage, https://thinletter.io
Project-URL: Models, https://huggingface.co/thinletter
Keywords: llm,inference,long-context,qwen,openai-compatible
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: POSIX :: Linux
Classifier: Intended Audience :: Developers
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-PolyForm-Small-Business-1.0.0.txt
Requires-Dist: torch<2.12,>=2.11
Requires-Dist: transformers<6,>=5.18
Requires-Dist: fla-core==0.5.2
Requires-Dist: safetensors>=0.4
Requires-Dist: huggingface_hub>=0.25
Requires-Dist: numpy>=1.26
Provides-Extra: conv1d
Requires-Dist: causal_conv1d>=1.4; extra == "conv1d"
Dynamic: license-file

# thinctx

Inference runtime and OpenAI-compatible server for the **ThinCtx models** — compact language models
whose context memory on the GPU stays small and grows 30 to 40 times more slowly than the original model's KV
cache, while the full context is held in system memory.

## Install and run

**Platforms.** Linux x86_64, or Windows 10/11 through WSL2, with an NVIDIA GPU; macOS is not supported. The
default PyTorch 2.11 wheel needs a driver with CUDA 13 support (with an older one, see below). In WSL the Windows
NVIDIA driver provides CUDA: do not install a Linux driver inside WSL. Triton compiles a small launcher on the first
start and needs gcc: `sudo apt install -y build-essential` (Debian / Ubuntu) if `gcc --version` fails.

Linux / WSL:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env   # only if uv is missing
uv venv --managed-python --python 3.12 ~/thinctx && source ~/thinctx/bin/activate
uv pip install thinctx
thinctx chat thinletter/ThinCtx-Qwen3.5-2B-small
```

With an older driver, run `uv pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128` before
installing thinctx. Without uv: `python3.12 -m venv ~/thinctx` (on Debian / Ubuntu it needs the python3.12-venv
package, and the python3.12-dev headers for Triton's launcher compile), then `pip install thinctx`.

Windows: in an admin PowerShell `wsl --install -d Ubuntu-24.04`, restart, open Ubuntu, then the Linux steps. Memory
sharing across conversations is Linux-native only (WSL runs without it). WSL keeps its file cache in Windows RAM: add
`autoMemoryReclaim=dropcache` under `[experimental]` in `%UserProfile%\.wslconfig`.

An OpenAI-compatible endpoint, or the engine from Python:

```bash
thinctx serve thinletter/ThinCtx-Qwen3.5-2B-small --port 8000 --auth user:pass
```

```python
import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-2B-small")   # a hub repo id or a local released folder
```

**First start.** The GPU kernels are compiled once, on the first run, and cached in `~/.triton/cache`, so the first
answer takes longer; later starts reuse the cache.

The server implements the OpenAI chat-completions API (`/v1/chat/completions`, streaming and
non-streaming, tools, `enable_thinking`); point any OpenAI client at `http://host:8000/v1`.

1.0.0 shares the memory of a common system prompt across conversations, moves idle conversations to system memory
and back, and keeps its public interface stable (below). It runs every ThinCtx model, including the compact `-small`
ones; `pip install -U thinctx`.

## Memory sharing across conversations

Conversations that start with the same system prompt share its stored context in system memory and its cached part
on the GPU. On the 2B, each additional conversation then costs about 180 MB of GPU memory and 30–60 MB of system
memory, almost independent of the prompt length (8k–32k tokens); without sharing it costs 320–620 MB of GPU memory
and 175–440 MB of system memory. On the 2B-small, an additional conversation costs 100–110 MB of GPU memory and
20–30 MB of system memory. A new conversation whose system prompt is already cached starts generating in 0.15–0.27 s
instead of 1.3–6.9 s (8k–32k tokens, RTX 3090). With `--warm_prompts`, the first conversation starts that fast too.
Idle conversations can be moved to system memory and resumed in about the same time as a live turn.
Single-conversation decode speed changes by at most ~1.5 %. The system memory a conversation's context takes stays
close to the context's size as it grows.

Memory limits (`thinctx serve` flags, defaults in brackets):

- `--prefix_cache_mb` (4096) and `--prefix_gpu_mb` (2048): system and GPU memory the prompt cache may hold; the least
  recently used prompts are dropped first.
- `--prefix_cache_ttl` (1800): seconds after which a cached prompt nobody reused is dropped (0: never).
- `--max_sessions` (the model's profile): conversations kept ready on the GPU; beyond it the least recently used idle
  one moves to system memory.
- `--session_ttl` (1800): seconds after which an idle conversation moves to system memory (0: never).
- `--session_gpu_mb` (0: no limit): GPU memory for all conversations together; beyond it idle ones move to system
  memory.
- `--tenant_cache_mb` / `--tenant_gpu_mb` (0: none): each API key's share of the prompt cache.

## Prompt cache and forks

A conversation that starts with a system prompt the server has already seen starts without reprocessing it, and it
shares that prompt's memory with every other conversation that starts the same way: the stored context in system
memory and the system prompt's part on the GPU are held once. Conversations are named by the request's `session`
field.

- **System prompt.** Conversations that start with the same system prompt (and tool list) process it once. Every
  later conversation starts from the cached copy, so its first token comes sooner, and only its own turns add
  memory. `--warm_prompts FILE` (a JSON list of `{"system": ..., "tools": [...], "namespace": ...}`) processes system
  prompts when the server starts, so even the first request is fast.
- **Regenerate, edit, fork** (opt-in: `--prefix_turns N`). The ends of the last N prompts of each conversation are
  saved too.
  - Sending the same request again regenerates the answer without reprocessing anything.
  - An edited last message resumes from the latest saved point it extends.
  - `POST /fork {"session": "new", "from": "old", "turn": 2}` starts a new conversation at the start of the second
    answer of another one. A chat request can do the same with `"fork_from": {"session": "old", "turn": 2}`.
- **Isolation.** With `--api_keys FILE` (one `NAME:KEY` per line, sent as `Authorization: Bearer KEY`), each key has
  its own conversations and its own cached prompts, so one tenant's prompt never speeds up another's.
  `--prefix_share global` reuses prompts across keys; `--prefix_share off` turns the prompt cache off.
  `--tenant_cache_mb` / `--tenant_gpu_mb` give each key its own share of the cache, so one key cannot push out
  another's prompts. `GET /cache` lists the cached prompts and `POST /cache/clear` drops them: a key sees and clears
  its own, the `--auth` user and the keys named in `--admin_keys` every key's.
- **Memory.** `--prefix_cache_mb` (default 4096) and `--prefix_gpu_mb` (default 2048) bound what the prompt cache
  holds; the least recently used entries are dropped first, and entries nobody reused for `--prefix_cache_ttl`
  seconds (default 1800) are dropped. Only prompts of at least 1024 tokens are saved (`--prefix_min_tokens`).
- **Idle conversations.** A conversation idle for `--session_ttl` seconds (default 1800), beyond `--max_sessions`
  conversations, or while all conversations together exceed `--session_gpu_mb` of GPU memory, moves to system
  memory; its next request continues it exactly where it was. System memory for a conversation's context grows with
  the context in small steps, so it stays close to the context's actual size.
- **Results.** A conversation resumed from a cached prompt, or moved to system memory and back, gives exactly the same
  tokens as one that never was.
  - Compared with a server that has no cache, the end of the system prompt adds one processing boundary. That changes
    the numbers only within bf16 rounding: in our tests the mean divergence was below 0.001 nats per token, and a
    greedy token can flip only where two candidates are nearly tied.
  - `--prefix_align chunk` saves at the processing block before the end instead. The results are then identical to a
    server without the cache, at the cost of reprocessing up to 2048 tokens of the prompt.

## Stability

From 1.0.0 on, thinctx follows semantic versioning for its public interface:

- **Python API:** `thinctx.load(...)` and the engine it returns, `ChatSession` (including `fork`).
- **Command line:** `thinctx serve` and `thinctx chat` with the flags their `--help` documents.
- **HTTP endpoints:** `/v1/chat/completions` (the OpenAI chat-completions schema), `/fork`, `/cache`, `/cache/clear`,
  `/info` and `/stats`.
- **Model formats:** every released model format (`thinctx-model/1`, `/2` and `/3`) stays loadable by later 1.x
  releases.

Anything else, including module internals and undocumented options, may change in any release. A part of the public
interface is only removed after it has printed a deprecation warning for at least one minor release.

## Requirements

Linux x86_64 or WSL2, CPython 3.12, an NVIDIA GPU, PyTorch 2.11, gcc (Triton's first start). Installs `torch`,
`transformers`, `fla-core` and `safetensors`. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA
driver, install the CUDA 12.8 build first: `pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128`.
`causal_conv1d` is optional and makes prefill faster (without it a slower pure-PyTorch fallback is used).

## License

Released under the [PolyForm Small Business License 1.0.0](https://polyformproject.org/licenses/small-business/1.0.0)
(`LICENSE`): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted
for inflation). `LICENSE` adds free use, for anyone, for personal non-commercial purposes, for non-commercial research
and education, and for evaluation and testing (non-production). Production use by larger organisations needs a
commercial licence: info@thinletter.io. The models are derivatives of Qwen3.5 (Apache-2.0); each model repo ships
Qwen's license and a `NOTICE`.
