Metadata-Version: 2.4
Name: tq-serve
Version: 0.4.26
Summary: TurboQuant model server manager — auto-configured llama-server with KV cache compression
Author: Sheng Chiao
License-Expression: MIT
Project-URL: Homepage, https://github.com/xt8086/tq
Project-URL: Repository, https://github.com/xt8086/tq
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: psutil>=5.9
Requires-Dist: huggingface_hub>=0.20
Requires-Dist: tomli>=2.0; python_version < "3.11"
Requires-Dist: tomli_w>=1.0
Requires-Dist: rich>=13.0
Provides-Extra: chat
Requires-Dist: httpx>=0.25; extra == "chat"
Requires-Dist: prompt_toolkit>=3.0; extra == "chat"
Requires-Dist: pypdf>=3.0; extra == "chat"
Requires-Dist: PyMuPDF>=1.24; extra == "chat"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-mock>=3.0; extra == "dev"

# tq — TurboQuant Model Server Manager

Auto-configured llama-server with KV cache compression.

## Install

```bash
# macOS (Apple Silicon) — one-liner
curl -fsSL https://raw.githubusercontent.com/xt8086/tq/main/install.sh | bash
```

> **Note:** `tq-serve` is a pure Python package — it does NOT include the `llama-server` binary.
> The binary comes from [TheTom/llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant),
> a fork of llama.cpp with TurboQuant KV cache compression built in.
> `tq install` (run automatically by the installer) downloads the correct binary for macOS Metal.
>
> Requires macOS Apple Silicon.

## Quick Start

```bash
tq doctor                        # Verify setup
tq list                          # List local GGUF models
tq search "qwen2.5 coder 7b"    # Search HuggingFace (numbered, pick # to download)
tq serve 1                       # Launch with auto-configured TurboQuant
tq serve 1 --lan                 # Allow other devices on WiFi to connect
tq chat                          # Interactive coding agent
```

## How It Works

`tq serve` automatically:

1. Detects your hardware (GPU, RAM)
2. Parses model metadata (quant type, layers, context length)
3. Calculates optimal TurboQuant cache settings
4. Launches llama-server with the right flags

Example: A Q4_K_M model on Apple M1 with 8GB RAM gets:
- `ctk=q8_0` (protect K cache)
- `ctv=turbo4` (compress V cache 3.8x)
- Context capped to safe memory limit
- Idle auto-stop after 5 min

## Commands

| Command | Description |
|---------|-------------|
| `tq list` | List local GGUF models |
| `tq search <query>` | Search HuggingFace for GGUF models (numbered, with download prompt) |
| `tq download <model>` | Download a model from HuggingFace |
| `tq remove <model>` | Remove a downloaded model |
| `tq serve 1` | Launch with auto TQ config |
| `tq serve 1 --lan` | Allow access from other devices on WiFi |
| `tq serve 1 --dry-run` | Show command without running |
| `tq status` | Check if server is running |
| `tq stop` | Stop the server |
| `tq logs` | View server logs |
| `tq validate <model>` | Pre-flight check |
| `tq install` | Download TurboQuant+ binary (from TheTom/llama-cpp-turboquant) |
| `tq doctor` | Verify setup |
| `tq config show` | Show/edit configuration |
| `tq chat` | Interactive coding agent (local AI) |

## API

The server exposes an OpenAI-compatible API:

```
POST http://127.0.0.1:8080/v1/chat/completions
```

No auth needed. Works with any OpenAI client:

```python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="your-model.gguf",
    messages=[{"role": "user", "content": "Hello"}]
)
```

## Configuration

Config stored at `~/.tq/config.toml`:

```bash
tq config show              # Show all settings
tq config set port 9090     # Change port
tq config set idle_timeout 600  # 10 min idle timeout (0 to disable)
```

## TurboQuant Cache Types

| Type | Bits | Compression | Use Case |
|------|------|-------------|----------|
| f16 | 16 | 1x | No compression (baseline) |
| q8_0 | 8 | 2x | Safe for K cache |
| turbo4 | 4.25 | 3.8x | Best quality/compression for V |
| turbo3 | 3.25 | 4.9x | Aggressive, for large models |

## Requirements

- Python 3.10+
- macOS (Apple Silicon)
- ~2GB free RAM minimum (depends on model)
