Metadata-Version: 2.5
Name: anet-kllm
Version: 0.1.0
Summary: Pull, run and seal LLMs and diffusion models on SGLang and vLLM.
Project-URL: Homepage, https://kllm.agentnetwork.org.cn
Project-URL: Documentation, https://docs.agentnetwork.org.cn/kllm/en/latest/
Project-URL: Source, https://github.com/ANetResearch/kLLM
Project-URL: Changelog, https://github.com/ANetResearch/kLLM/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/ANetResearch/kLLM/issues
Author-email: The kLLM Authors <hi@anet0.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: diffusion,inference,llm,sealing,sglang,vllm
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Provides-Extra: client
Requires-Dist: cryptography>=42; extra == 'client'
Requires-Dist: fastapi; extra == 'client'
Requires-Dist: numpy; extra == 'client'
Requires-Dist: safetensors; extra == 'client'
Requires-Dist: torch; extra == 'client'
Requires-Dist: transformers; extra == 'client'
Requires-Dist: uvicorn; extra == 'client'
Provides-Extra: dev
Requires-Dist: cryptography>=42; extra == 'dev'
Requires-Dist: numpy; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: safetensors; extra == 'dev'
Requires-Dist: torch; extra == 'dev'
Provides-Extra: image
Requires-Dist: cryptography>=42; extra == 'image'
Requires-Dist: diffusers>=0.36; extra == 'image'
Requires-Dist: fastapi; extra == 'image'
Requires-Dist: numpy; extra == 'image'
Requires-Dist: pillow; extra == 'image'
Requires-Dist: safetensors; extra == 'image'
Requires-Dist: torch; extra == 'image'
Requires-Dist: transformers; extra == 'image'
Requires-Dist: uvicorn; extra == 'image'
Provides-Extra: keyed
Requires-Dist: cryptography>=42; extra == 'keyed'
Provides-Extra: seal
Requires-Dist: cryptography>=42; extra == 'seal'
Requires-Dist: numpy; extra == 'seal'
Requires-Dist: safetensors; extra == 'seal'
Requires-Dist: torch; extra == 'seal'
Provides-Extra: serve
Requires-Dist: sglang>=0.5; extra == 'serve'
Provides-Extra: vllm
Requires-Dist: vllm>=0.11; extra == 'vllm'
Description-Content-Type: text/markdown

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://kllm.agentnetwork.org.cn/img/kllm-hero-dark.png" />
  <source media="(prefers-color-scheme: light)" srcset="https://kllm.agentnetwork.org.cn/img/kllm-hero-light.png" />
  <img src="https://kllm.agentnetwork.org.cn/img/kllm-hero-light.png" alt="kLLM: sealed models on stock SGLang and vLLM" width="100%" />
</picture>

[![PyPI](https://img.shields.io/badge/PyPI-0.1.0-C9241B?style=flat-square&labelColor=1A1C1F)](https://pypi.org/project/anet-kllm/)
[![CI](https://img.shields.io/github/actions/workflow/status/ANetResearch/kLLM/ci.yml?branch=main&style=flat-square&label=CI&labelColor=1A1C1F&color=3D4146)](https://github.com/ANetResearch/kLLM/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-Apache--2.0-3D4146?style=flat-square&labelColor=1A1C1F)](https://github.com/ANetResearch/kLLM/blob/main/LICENSE)
[![Docs](https://img.shields.io/badge/docs-docs.agentnetwork.org.cn-3D4146?style=flat-square&labelColor=1A1C1F)](https://docs.agentnetwork.org.cn/kllm/en/latest/)
[![Python](https://img.shields.io/badge/python-3.10%2B-3D4146?style=flat-square&labelColor=1A1C1F)](https://github.com/ANetResearch/kLLM/blob/main/pyproject.toml)
[![Engines](https://img.shields.io/badge/engines-SGLang%20%7C%20vLLM-3D4146?style=flat-square&labelColor=1A1C1F)](https://github.com/ANetResearch/kLLM#quickstart)
[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-ANetResearch-3D4146?style=flat-square&labelColor=1A1C1F)](https://huggingface.co/ANetResearch)

| **[Docs](https://docs.agentnetwork.org.cn/kllm/en/latest/)** | **[Website](https://kllm.agentnetwork.org.cn)** | **[Models on Hugging Face](https://huggingface.co/ANetResearch)** | **[Quickstart](https://github.com/ANetResearch/kLLM#quickstart)** | **[Citation](https://github.com/ANetResearch/kLLM#citation)** |
|:---:|:---:|:---:|:---:|:---:|

# kLLM

kLLM pulls and runs models on [SGLang](https://github.com/sgl-project/sglang) and [vLLM](https://github.com/vllm-project/vllm), with a workflow in the manner of Ollama. It can also seal a model under a key. A sealed model keeps its tensor names, shapes and data types, so the stock engine serves it unmodified. With the right key, it produces the original model's output; with a wrong key, the text is meaningless and the image is noise.

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://kllm.agentnetwork.org.cn/img/kllm-key-demo-dark.png" />
  <source media="(prefers-color-scheme: light)" srcset="https://kllm.agentnetwork.org.cn/img/kllm-key-demo-light.png" />
  <img src="https://kllm.agentnetwork.org.cn/img/kllm-key-demo-light.png" alt="The same prompt and seed with the right key and with a wrong key. Text (Qwen3-8B): a coherent answer versus a repeated token. Image (Z-Image): a watercolor cat astronaut versus noise." width="100%" />
</picture>

<img src="https://kllm.agentnetwork.org.cn/img/kllm-h3-demo.gif" alt="MiniMax H3 output for the same prompt, with and without the key: on the left, the hospital corridor scene described in the prompt; on the right, an unrelated scene of two people seated on a leather sofa on a stage." width="100%" />

MiniMax H3: the same prompt with the key (left) and without the key (right). Without the key, the prompt reaches the model as garbled tokens and the video shows an unrelated scene. The full videos are on the [kLLM website](https://kllm.agentnetwork.org.cn/#demo-title).

## Quickstart

### Install

kLLM 0.1.0 is available on PyPI as anet-kllm. It requires Linux, an NVIDIA GPU and Python 3.10 or later.

```bash
pip install 'anet-kllm[seal,serve]'     # SGLang; use 'anet-kllm[seal,vllm]' for vLLM
kllm doctor                        # checks the driver, CUDA, disk space and engine packages
```

Alternatively, install from source:

```bash
git clone https://github.com/ANetResearch/kLLM && cd kLLM
pip install -e '.[seal,serve]'     # SGLang; use '.[seal,vllm]' for vLLM
```

### Run a catalog model

Catalog models are public models sealed so that anyone can try kLLM. Everyone who submits a model's access form receives the same key, so a catalog model is a demonstration, not a private model.

1. Open the model's Hugging Face repository, such as [ANetResearch/Qwen3-8B-kllm](https://huggingface.co/ANetResearch/Qwen3-8B-kllm). Sign in and submit the access form. Approval is immediate.
2. Download the model and its key, `kllm.key`, which is included in the repository and requires no passphrase.
3. Run or serve the model with the key.

```bash
hf auth login                                          # or set HF_TOKEN
hf download ANetResearch/Qwen3-8B-kllm --local-dir Qwen3-8B-kllm
kllm serve Qwen3-8B-kllm --key Qwen3-8B-kllm/kllm.key  # OpenAI-compatible API on port 8000; add --engine vllm for vLLM
```

Alternatively, pull and run the model through the local store:

```bash
kllm pull qwen3:8b-sealed                              # or: kllm pull hf://ANetResearch/Qwen3-8B-kllm
kllm run qwen3:8b-sealed                               # uses the kllm.key pulled with the model
```

### Seal your own model

To keep a model private, seal it under a new key:

```bash
kllm key new mykey                                     # prompts for a passphrase
kllm pull qwen3:8b
kllm create qwen3-mine --from qwen3:8b --key mykey --transform perm
kllm run qwen3-mine --key mykey
```

For a local Hugging Face checkpoint directory, create a key file with `kllm keygen -o my.key`, seal the checkpoint with `kllm seal models/Qwen3-8B -o models/Qwen3-8B-sealed --key my.key`, and serve it with `kllm serve models/Qwen3-8B-sealed --key my.key`.

`kllm run qwen3:8b` runs a library model without sealing it. `kllm ps` lists running models; `kllm stop` stops one.

## Catalog

The catalog contains 19 sealed models at [huggingface.co/ANetResearch](https://huggingface.co/ANetResearch). Each is sealed from a public base model and keeps that model's license. `kllm search sealed` lists the models and their library tags.

| Model | Base model | Modality | Size |
|---|---|---|---|
| [Qwen3-0.6B-kllm](https://huggingface.co/ANetResearch/Qwen3-0.6B-kllm) | `Qwen/Qwen3-0.6B` | Text | 1.5 GB |
| [Qwen3-1.7B-kllm](https://huggingface.co/ANetResearch/Qwen3-1.7B-kllm) | `Qwen/Qwen3-1.7B` | Text | 4.1 GB |
| [Qwen3-4B-kllm](https://huggingface.co/ANetResearch/Qwen3-4B-kllm) | `Qwen/Qwen3-4B` | Text | 8.8 GB |
| [Qwen3-4B-Instruct-2507-kllm](https://huggingface.co/ANetResearch/Qwen3-4B-Instruct-2507-kllm) | `Qwen/Qwen3-4B-Instruct-2507` | Text | 8.8 GB |
| [Qwen3-8B-kllm](https://huggingface.co/ANetResearch/Qwen3-8B-kllm) | `Qwen/Qwen3-8B` | Text | 16.4 GB |
| [Qwen3-14B-kllm](https://huggingface.co/ANetResearch/Qwen3-14B-kllm) | `Qwen/Qwen3-14B` | Text | 29.5 GB |
| [Qwen3-32B-kllm](https://huggingface.co/ANetResearch/Qwen3-32B-kllm) | `Qwen/Qwen3-32B` | Text | 65.5 GB |
| [Qwen3-30B-A3B-Instruct-2507-kllm](https://huggingface.co/ANetResearch/Qwen3-30B-A3B-Instruct-2507-kllm) | `Qwen/Qwen3-30B-A3B-Instruct-2507` | Text, mixture of experts | 61.1 GB |
| [Qwen3-30B-A3B-Instruct-2507-FP8-kllm](https://huggingface.co/ANetResearch/Qwen3-30B-A3B-Instruct-2507-FP8-kllm) | `Qwen/Qwen3-30B-A3B-Instruct-2507-FP8` | Text, mixture of experts, FP8 | 31.2 GB |
| [Qwen3-Coder-30B-A3B-Instruct-FP8-kllm](https://huggingface.co/ANetResearch/Qwen3-Coder-30B-A3B-Instruct-FP8-kllm) | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` | Text, mixture of experts, FP8 | 31.2 GB |
| [Qwen2.5-7B-Instruct-kllm](https://huggingface.co/ANetResearch/Qwen2.5-7B-Instruct-kllm) | `Qwen/Qwen2.5-7B-Instruct` | Text | 15.2 GB |
| [Qwen2.5-Coder-7B-Instruct-kllm](https://huggingface.co/ANetResearch/Qwen2.5-Coder-7B-Instruct-kllm) | `Qwen/Qwen2.5-Coder-7B-Instruct` | Text | 15.2 GB |
| [DeepSeek-R1-Distill-Qwen-7B-kllm](https://huggingface.co/ANetResearch/DeepSeek-R1-Distill-Qwen-7B-kllm) | `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B` | Text | 15.2 GB |
| [DeepSeek-V4-Flash-kllm](https://huggingface.co/ANetResearch/DeepSeek-V4-Flash-kllm) | `deepseek-ai/DeepSeek-V4-Flash` | Text, mixture of experts | 159.6 GB |
| [DeepSeek-V4-Pro-kllm](https://huggingface.co/ANetResearch/DeepSeek-V4-Pro-kllm) | `deepseek-ai/DeepSeek-V4-Pro` | Text, mixture of experts | 864.7 GB |
| [GLM-5.3-kllm](https://huggingface.co/ANetResearch/GLM-5.3-kllm) | `zai-org/GLM-5.3` | Text, mixture of experts | 755.7 GB |
| [GLM-5.3-Flash-kllm](https://huggingface.co/ANetResearch/GLM-5.3-Flash-kllm) | `zai-org/GLM-5.3-Flash` | Text and image input, mixture of experts | 328.4 GB |
| [Z-Image-kllm](https://huggingface.co/ANetResearch/Z-Image-kllm) | `Tongyi-MAI/Z-Image` | Image generation | 20.5 GB |
| [Z-Image-Turbo-kllm](https://huggingface.co/ANetResearch/Z-Image-Turbo-kllm) | `Tongyi-MAI/Z-Image-Turbo` | Image generation | 32.9 GB |

## Supported architectures

Any model that the selected engine supports can be served unsealed. Sealing supports the architectures below.

| Family | Architecture | Modality | Transforms | Engines |
|---|---|---|---|---|
| Qwen3 | `Qwen3ForCausalLM` | Text | orth, perm (FP8: perm) | SGLang, vLLM |
| Qwen3-MoE | `Qwen3MoeForCausalLM` | Text, mixture of experts | orth, perm (FP8: perm) | SGLang, vLLM |
| Qwen2 and Qwen2.5, including DeepSeek-R1-Distill-Qwen | `Qwen2ForCausalLM` | Text | orth, perm | SGLang, vLLM |
| GLM-5 (5, 5.1, 5.2, 5.3) | `GlmMoeDsaForCausalLM` | Text, mixture of experts | perm | SGLang, vLLM |
| DeepSeek V4 (Flash, Pro) | `DeepseekV4ForCausalLM` | Text, mixture of experts | perm | SGLang, vLLM |
| GLM-5.3-Flash | `Glm5NextForConditionalGeneration` | Text and image input, mixture of experts | perm | SGLang, vLLM |
| MiniMax H3 | FL2VA | Video with audio | orth, perm | SGLang |
| Z-Image | `ZImagePipeline` | Image generation | orth, perm | kLLM latent server |

To request another model, open a [model request](https://github.com/ANetResearch/kLLM/issues/new?template=model-request.yml) or run `kllm request MODEL_ID`.

## Editions

kLLM Community is the open-source edition in this repository. For every model, it keeps the efficiency of the stock engine. kLLM Enterprise is a separate runtime written in Rust and Go, with dedicated optimizations for selected models and workloads, such as MiniMax H3 and agent-team workloads. These optimizations are not open source. Contact the team through [kllm.agentnetwork.org.cn/enterprise](https://kllm.agentnetwork.org.cn/enterprise) or at hi@anet0.com.

## Documentation

The [documentation](https://docs.agentnetwork.org.cn/kllm/en/latest/) covers installation, the catalog, sealing, serving, the command-line reference and guides for each model family. For contributions, see [CONTRIBUTING.md](https://github.com/ANetResearch/kLLM/blob/main/CONTRIBUTING.md).

## Citation

```bibtex
@inproceedings{yuan2025scx,
  title     = {{SCX}: Stateless {KV}-Cache Encoding for Cloud-Scale Confidential Transformer Serving},
  author    = {Yuan, Mu and Zhang, Lan and Zeng, Liekang and Jiang, Siyang and Yang, Bufang and Duan, Di and Xing, Guoliang},
  booktitle = {Proceedings of the ACM SIGCOMM 2025 Conference},
  year      = {2025},
  doi       = {10.1145/3718958.3750509}
}

@inproceedings{yuan2026stip,
  title     = {{STIP}: Three-Party Privacy-Preserving and Lossless Inference for Large Transformers in Production},
  author    = {Yuan, Mu and Zhang, Lan and Cheng, Yihang and Song, Miao-Hui and Xing, Guoliang and Li, Xiang-Yang},
  booktitle = {Network and Distributed System Security (NDSS) Symposium 2026},
  address   = {San Diego, CA},
  year      = {2026},
  doi       = {10.14722/ndss.2026.230035}
}
```

## License

kLLM is licensed under [Apache-2.0](https://github.com/ANetResearch/kLLM/blob/main/LICENSE). Each catalog model retains its base model's license.

<!-- Enable once the repository is public; the chart renders empty for a private repository.
## Star history

<a href="https://star-history.com/#ANetResearch/kLLM&Date">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=ANetResearch/kLLM&type=Date&theme=dark" />
    <img alt="Star history chart" src="https://api.star-history.com/svg?repos=ANetResearch/kLLM&type=Date" />
  </picture>
</a>
-->

---

kLLM is a project of Agent Network.
