Metadata-Version: 2.4
Name: thinctx
Version: 0.3.1
Summary: ThinCtx: inference runtime and OpenAI-compatible server for the ThinCtx models (a small GPU memory for the context that grows 20 to 30 times more slowly than the original model's KV cache).
Author: Jan Rosecký (thinletter)
License: PolyForm-Small-Business-1.0.0
Project-URL: Homepage, https://thinletter.io
Project-URL: Models, https://huggingface.co/thinletter
Keywords: llm,inference,long-context,qwen,openai-compatible
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: POSIX :: Linux
Classifier: Intended Audience :: Developers
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-PolyForm-Small-Business-1.0.0.txt
Requires-Dist: torch<2.12,>=2.11
Requires-Dist: transformers<6,>=5.18
Requires-Dist: fla-core==0.5.2
Requires-Dist: safetensors>=0.4
Requires-Dist: huggingface_hub>=0.25
Requires-Dist: numpy>=1.26
Provides-Extra: conv1d
Requires-Dist: causal_conv1d>=1.4; extra == "conv1d"
Dynamic: license-file

# thinctx

Inference runtime and OpenAI-compatible server for the **ThinCtx models** — compact language models
whose context memory on the GPU stays small and grows 20 to 30 times more slowly than the original model's KV
cache, while the full context is held in system memory.

```bash
pip install thinctx

thinctx serve thinletter/ThinCtx-Qwen3.5-2B --port 8000 --auth user:pass   # OpenAI-compatible endpoint
thinctx chat  thinletter/ThinCtx-Qwen3.5-2B                                # terminal REPL
```

```python
import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-2B")   # a hub repo id or a local released folder
```

The server implements the OpenAI chat-completions API (`/v1/chat/completions`, streaming and
non-streaming, tools, `enable_thinking`); point any OpenAI client at `http://host:8000/v1`.

This release runs the compact `-small` models (smaller weights and a compressed context store in system memory,
for smaller GPUs; they also load faster), adds a prompt cache to the server (below), and needs less GPU memory
while reading very long prompts. The `-small` repos need this version; `pip install -U thinctx`.

## Prompt cache and forks

A conversation that starts with a system prompt the server has already seen starts without reprocessing it. In this
version each conversation keeps its own copy of the context in memory; sharing memory across conversations comes in
1.0.0. Conversations are named by the request's `session` field.

- **System prompt.** Conversations that start with the same system prompt (and tool list) process it once. Every
  later conversation starts from the cached copy, so its first token comes sooner.
- **Regenerate, edit, fork** (opt-in: `--prefix_turns N`). The ends of the last N prompts of each conversation are
  saved too. Each save copies the conversation's context into system memory during that turn: for the 2B, about
  0.1 / 0.2 / 0.4 s and 0.2 / 0.5 / 1.8 GB at 8k / 32k / 128k tokens.
  - Sending the same request again regenerates the answer without reprocessing anything.
  - An edited last message resumes from the latest saved point it extends.
  - `POST /fork {"session": "new", "from": "old", "turn": 2}` starts a new conversation at the start of the second
    answer of another one. A chat request can do the same with `"fork_from": {"session": "old", "turn": 2}`.
- **Isolation.** With `--api_keys FILE` (one `NAME:KEY` per line, sent as `Authorization: Bearer KEY`), each key has
  its own conversations and its own cached prompts, so one tenant's prompt never speeds up another's.
  `--prefix_share global` reuses prompts across keys; `--prefix_share off` turns the prompt cache off.
- **Memory.** `--prefix_cache_mb` (default 4096) bounds the system memory the prompt cache uses, and the least
  recently used entries are dropped first. Only prompts of at least 1024 tokens are saved (`--prefix_min_tokens`).
- **Results.** A conversation resumed from a cached prompt gives exactly the same tokens as the conversation that
  cached it.
  - Compared with a server that has no cache, the end of the system prompt adds one processing boundary. That changes
    the numbers only within bf16 rounding: in our tests the mean divergence was below 0.001 nats per token, and a
    greedy token can flip only where two candidates are nearly tied.
  - `--prefix_align chunk` saves at the processing block before the end instead. The results are then identical to a
    server without the cache, at the cost of reprocessing up to 2048 tokens of the prompt.

## Requirements

Linux x86_64, CPython 3.12, an NVIDIA GPU, PyTorch 2.11. Installs `torch`, `transformers`, `fla-core` and
`safetensors`. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build
first: `pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128`. `causal_conv1d` is optional
and makes prefill faster (without it a slower pure-PyTorch fallback is used).

## License

Released under the [PolyForm Small Business License 1.0.0](https://polyformproject.org/licenses/small-business/1.0.0)
(`LICENSE`): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted
for inflation). `LICENSE` adds free use, for anyone, for personal non-commercial purposes, for non-commercial research
and education, and for evaluation and testing (non-production). Production use by larger organisations needs a
commercial licence: info@thinletter.io. The models are derivatives of Qwen3.5 (Apache-2.0); each model repo ships
Qwen's license and a `NOTICE`.
