Metadata-Version: 2.4
Name: pytllm
Version: 1.0.1
Summary: TLLM runs 70B large language models on a single 4GB GPU without quantization, distillation or pruning. 405B Llama 3.1 on 8GB, DeepSeek-V3 671B on ~12GB, Kimi K3 2.8T on under 4GB, Qwen3.8-27B on 3.3GB.
Home-page: https://github.com/Cognitivess/pytllm
Author: CognitivessAI
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tqdm
Requires-Dist: torch>=2.4
Requires-Dist: transformers<5.13,>=4.49
Requires-Dist: accelerate>=1.0
Requires-Dist: safetensors
Requires-Dist: huggingface-hub
Requires-Dist: scipy
Requires-Dist: sentencepiece
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-dist
Dynamic: summary

# TLLM

[**Quickstart**](#quickstart) |
[**Disk bottleneck solved**](#disk-bottleneck-solved) |
[**Configurations**](#configurations) |
[**MacOS**](#macos) |
[**Example notebooks**](#example-python-notebook) |
[**FAQ**](#faq)

**TLLM** dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run **405B Llama 3.1** on **8GB**, **DeepSeek-V3 (671B)** on **~12GB**, and **Kimi K3 (2.8T)** — the largest open-source model released to date — on **under 4GB**, because sparse MoE models stream one expert at a time rather than a whole layer.

TLLM fixes the biggest real-world limitation of layer-streaming inference: the per-token disk re-read (see [Disk bottleneck solved](#disk-bottleneck-solved)).

[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)](LICENSE)

## Disk bottleneck solved

Layer-streaming inference reads one layer at a time from disk to GPU and evicts it
immediately after the layer runs. That saves VRAM, but it means **every single
generated token re-reads essentially the entire model from disk**. For a 70B
model (~140GB in bf16) that is 0.5–2 tokens/second on a 4GB GPU versus 15–30
with the model in VRAM — and for Kimi K3 some users report several minutes per
token. It runs, but it is a tool for offline batch processing, not interactive
chat.

**The real bottleneck isn't VRAM, it's disk.** TLLM adds a layered weight cache
that breaks this:

- On the **first** token, layers are read from disk once (as before).
- Layers that fit are then kept **resident in VRAM**; the rest **spill to
  pinned host RAM**; only what exceeds both is re-read from disk on later
  tokens.
- So the disk read becomes a **one-time cost** instead of a per-token cost.
  Subsequent tokens hit the cache, and a model that fits in VRAM approaches
  full-VRAM throughput.

The cache is **on by default** and degrades gracefully: if VRAM or RAM runs
low, layers fall back to the original streaming path, so TLLM never uses more
memory than uncached streaming on the same hardware. See the `cache_layers`,
`vram_cache_reserve_gb`, and `pinned_cache_gb` options in
[Configurations](#configurations).

## Updates

[2026/08] **Disk bottleneck fix**: layered VRAM + pinned-RAM weight cache. Layers are read from disk once and reused across tokens instead of re-read on every token. On by default, graceful degradation.

[2026/08] **Qwen3.8-27B** support: Qwen's new dense VL (Gated DeltaNet + Gated Attention, native vision) runs in **3.33GB** of VRAM, measured end to end on one RTX 3090. Needs `transformers` 5.8+.

[2026/07] **Kimi K3 (2.8T)** support: the largest open-source model runs on a single card in **3.72GB** of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: `pip install compressed-tensors flash-attn` (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and `transformers` 4.56.x, as its remote code does not load on 5.x.

[2026/06] **v3.0**: FP8 model support + the latest models. Run **DeepSeek-V3 (671B) on ~12GB** and **Qwen3-235B on ~3GB**, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single `AutoModel`.

## Table of Contents

* [Disk bottleneck solved](#disk-bottleneck-solved)
* [Quick start](#quickstart)
* [Model Compression](#model-compression---3x-inference-speed-up)
* [Configurations](#configurations)
* [Run on MacOS](#macos)
* [Example notebooks](#example-python-notebook)
* [Supported Models](#supported-models)
* [Acknowledgement](#acknowledgement)
* [FAQ](#faq)

## Quickstart

### 1. Install package

First, install the pytllm pip package.

```bash
pip install pytllm
```

### 2. Inference

Then, initialize the model via `AutoModel`, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model.

(*You can also specify the path to save the splitted layered model through **layer_shards_saving_path** when init the model.*

```python
from pytllm import AutoModel

MAX_LENGTH = 128
# just pass a hugging face repo id — works with almost any popular model:
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# go bigger with the exact same one line:
#model = AutoModel.from_pretrained("Qwen/Qwen3.8-27B")          # 27B dense VL, 3.33GB
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")     # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, runs in ~12GB

# or use a model's local path...
#model = AutoModel.from_pretrained("/home/ubuntu/.cache/huggingface/hub/models--Qwen--Qwen3-32B/snapshots/...")

input_text = [
        'What is the capital of United States?',
        #'I like',
    ]

input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])

print(output)

```

Note: During inference, the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space in the huggingface cache directory.

## Model Compression - 3x Inference Speed Up!

We support model compression based on block-wise quantization-based model compression. Which can further **speed up the inference speed** for up to **3x** , with **almost ignorable accuracy loss!** (see more performance evaluation and why we use block-wise quantization in [this paper](https://arxiv.org/abs/2212.09720))

#### How to enable model compression speed up:

* Step 1. make sure you have [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) installed by `pip install -U bitsandbytes `
* Step 2. make sure pytllm verion later than 2.0.0: `pip install -U pytllm`
* Step 3. when initialize the model, passing the argument compression ('4bit' or '8bit'):

```python
model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct",
                     compression='4bit' # specify '8bit' for 8-bit block-wise quantization
                    )
```

#### What are the differences between model compression and quantization?

Quantization normally needs to quantize both weights and activations to really speed things up. Which makes it harder to maintain accuracy and avoid the impact of outliers in all kinds of inputs.

While in our case the bottleneck is mainly at the disk loading, we only need to make the model loading size smaller. So, we get to only quantize the weights' part, which is easier to ensure the accuracy.

## Configurations

When initialize the model, we support the following configurations:

* **compression**: supported options: 4bit, 8bit for 4-bit or 8-bit block-wise quantization, or by default None for no compression
* **profiling_mode**: supported options: True to output time consumptions or by default False
* **layer_shards_saving_path**: optionally another path to save the splitted model
* **hf_token**: huggingface token can be provided here if downloading gated models like: *meta-llama/Llama-2-7b-hf*
* **prefetching**: prefetching to overlap the model loading and compute. By default, turned on.
* **delete_original**: if you don't have too much disk space, you can set delete_original to true to delete the original downloaded hugging face model, only keep the transformed one to save half of the disk space.

### Layer cache (disk bottleneck fix)

These control the layered weight cache that stops TLLM from re-reading the model
from disk on every token (see [Disk bottleneck solved](#disk-bottleneck-solved)):

* **cache_layers**: `True` (default). Keep streamed layers resident in VRAM,
  spilling to pinned RAM, and only re-read from disk what exceeds both. Set to
  `False` to restore uncached per-token streaming behaviour.
* **vram_cache_reserve_gb**: amount of VRAM to leave free for activations and
  the KV cache rather than filling it with cached weights. Default `2`. Raise
  it if you see OOMs during generation; lower it to cache more layers.
* **pinned_cache_gb**: how much page-locked host RAM to use as a second cache
  tier before falling back to disk. Default uses a fraction of free RAM.
  Pinned RAM copies to the GPU faster than a disk read, so the spill tier is
  still much cheaper than re-reading every token.

The cache degrades gracefully: if admitting a layer would exceed the available
VRAM or RAM, that layer falls back to the original stream-from-disk path, so
peak memory never exceeds uncached streaming on the same hardware.

## MacOS

Just install pytllm and run the code the same as on linux. See more in [Quick Start](#quickstart).

* make sure you installed [mlx](https://github.com/ml-explore/mlx?tab=readme-ov-file#installation) and torch
* you probably need to install python native see more [here](https://stackoverflow.com/a/65432861/21230266)
* only [Apple silicon](https://support.apple.com/en-us/HT211814) is supported

Example [python notebook](tllm/examples/run_on_macos.ipynb)

## Example Python Notebook

Example colabs here:

See the [example notebook](tllm/examples/run_all_types_of_models.ipynb) covering ChatGLM, QWen, Baichuan, Mistral and more.

#### example of other models (ChatGLM, QWen, Baichuan, Mistral, etc):

<details>

* ChatGLM:

```python
from pytllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("THUDM/chatglm3-6b-base")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=True)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=5,
    use_cache= True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])
```

* QWen:

```python
from pytllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen-7B")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=5,
    use_cache=True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])
```


* Baichuan, InternLM, Mistral, etc:

```python
from pytllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("baichuan-inc/Baichuan2-7B-Base")
#model = AutoModel.from_pretrained("internlm/internlm-20b")
#model = AutoModel.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=5,
    use_cache=True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])
```

</details>

## Supported Models

TLLM works out of the box with **virtually every popular open LLM** — just pass its Hugging Face ID to `AutoModel.from_pretrained(...)`. That covers all the major families:

**Llama** (2 / 3 / 3.1 / 3.3 / 4) · **Qwen** (1 / 2 / 2.5 / 3 / 3.5 / 3.8, including MoE, FP8, and native VL) · **DeepSeek** (V2 / V3 / R1) · **Mistral & Mixtral** · **Phi** · **Gemma** · **ChatGLM** · **Baichuan** · **InternLM** · **Yi** · **Kimi K3** — and most new models the day they're released.

### Tiny GPU, huge models

The trick: TLLM only ever keeps **one layer on the GPU at a time**, so the VRAM you need depends on the model's layer size — not its total size. That's how a 671B model fits on a hobbyist card:

| Model | Size | GPU VRAM |
|---|---|---|
| Qwen3 / Mistral / Phi (≈8B) | 8B | **~1–2 GB** |
| Qwen3-30B / Mixtral (MoE) | 30–47B | **~1–3 GB** |
| Qwen3.8-27B (dense VL) | 27B | **3.33 GB** |
| Qwen3-235B (MoE) | 235B | **~3 GB** |
| Llama 3.x 70B (full precision) | 70B | **~4 GB** |
| Llama 3.1 405B | 405B | **~8 GB** |
| DeepSeek-V3 | **671B** | **~12 GB** |

Same one line of code for all of them — no special setup.

## Acknowledgement

A lot of the original code is based on SimJeg's great work in the Kaggle exam competition. Big shoutout to SimJeg:

[GitHub account @SimJeg](https://github.com/SimJeg),
[the code on Kaggle](https://www.kaggle.com/code/simjeg/platypus2-70b-with-wikipedia-rag),
[the associated discussion](https://www.kaggle.com/competitions/kaggle-llm-science-exam/discussion/446414).

## FAQ

### 1. MetadataIncompleteBuffer

safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer

If you run into this error, most possible cause is you run out of disk space. The process of splitting model is very disk-consuming. See [this](https://huggingface.co/TheBloke/guanaco-65B-GPTQ/discussions/12). You may need to extend your disk space, clear huggingface [.cache](https://huggingface.co/docs/datasets/cache) and rerun.

### 2. ValueError: max() arg is an empty sequence

Most likely you are loading QWen or ChatGLM model with Llama2 class. Try the following:

For QWen model:

```python
from pytllm import AutoModel #<----- instead of TLLMLlama2
AutoModel.from_pretrained(...)
```

For ChatGLM model:

```python
from pytllm import AutoModel #<----- instead of TLLMLlama2
AutoModel.from_pretrained(...)
```

### 3. 401 Client Error....Repo model ... is gated.

Some models are gated models, needs huggingface api token. You can provide hf_token:

```python
model = AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf", #hf_token='HF_API_TOKEN')
```

### 4. ValueError: Asking to pad but the tokenizer does not have a padding token.

Some model's tokenizer doesn't have padding token, so you can set a padding token or simply turn the padding config off:

 ```python
input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False  #<-----------   turn off padding
)
```

## Citing TLLM

If you find
TLLM useful in your research and wish to cite it, please use the following
BibTex entry:

```
@software{tllm2026,
  author = {TLLM},
  title = {TLLM: streaming large language model inference with a layered weight cache},
  version = {1.0.0},
  year = {2026},
}
```

## Contribution

Welcomed contributions, ideas and discussions!
