Metadata-Version: 2.4
Name: thinctx
Version: 0.1.1
Summary: ThinCtx: inference runtime and OpenAI-compatible server for the ThinCtx models (a small GPU memory for the context that grows 20 to 30 times more slowly than the original model's KV cache).
Author: Jan Rosecký (thinletter)
License: PolyForm-Small-Business-1.0.0
Project-URL: Homepage, https://thinletter.io
Project-URL: Models, https://huggingface.co/thinletter
Keywords: llm,inference,long-context,qwen,openai-compatible
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: POSIX :: Linux
Classifier: Intended Audience :: Developers
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-PolyForm-Small-Business-1.0.0.txt
Requires-Dist: torch<2.12,>=2.11
Requires-Dist: transformers<6,>=5.18
Requires-Dist: fla-core==0.5.2
Requires-Dist: safetensors>=0.4
Requires-Dist: huggingface_hub>=0.25
Requires-Dist: numpy>=1.26
Provides-Extra: conv1d
Requires-Dist: causal_conv1d>=1.4; extra == "conv1d"
Dynamic: license-file

# thinctx

Inference runtime and OpenAI-compatible server for the **ThinCtx models** — compact language models
whose context memory on the GPU stays small and grows 20 to 30 times more slowly than the original model's KV
cache, while the full context is held in system memory.

```bash
pip install thinctx

thinctx serve thinletter/ThinCtx-Qwen3.5-2B --port 8000 --auth user:pass   # OpenAI-compatible endpoint
thinctx chat  thinletter/ThinCtx-Qwen3.5-2B                                # terminal REPL
```

```python
import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-2B")   # a hub repo id or a local released folder
```

The server implements the OpenAI chat-completions API (`/v1/chat/completions`, streaming and
non-streaming, tools, `enable_thinking`); point any OpenAI client at `http://host:8000/v1`.

## Requirements

Linux x86_64, CPython 3.12, an NVIDIA GPU, PyTorch 2.11. Installs `torch`, `transformers`, `fla-core` and
`safetensors`. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build
first: `pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128`. `causal_conv1d` is optional
and makes prefill faster (without it a slower pure-PyTorch fallback is used).

## License

Released under the [PolyForm Small Business License 1.0.0](https://polyformproject.org/licenses/small-business/1.0.0)
(`LICENSE`): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted
for inflation). `LICENSE` adds free use, for anyone, for personal non-commercial purposes, for non-commercial research
and education, and for evaluation and testing (non-production). Production use by larger organisations needs a
commercial licence: info@thinletter.io. The models are derivatives of Qwen3.5 (Apache-2.0); each model repo ships
Qwen's license and a `NOTICE`.
