Metadata-Version: 2.1
Name: qwen-tokenizer
Version: 0.3.0
Summary: Lightweight tokenizer for Qwen, including Qwen3.5 and Qwen3.6.
Author-Email: Anderson <andersonby@163.com>
License: MIT
Requires-Python: >=3.8
Requires-Dist: tiktoken>=0.7.0
Description-Content-Type: text/markdown

# Qwen Tokenizer

## Introduction

Qwen Tokenizer is a lightweight tokenization library that does not require the `transformers` package. It uses `tiktoken` as its only runtime dependency.

Version 0.3.0 adds the tokenizer shared by the official `Qwen/Qwen3.5-*` and `Qwen/Qwen3.6-*` models, including `Qwen/Qwen3.6-27B`. Earlier Qwen models continue to use the legacy tokenizer, so selecting a Qwen3.5 or Qwen3.6 model does not change tokenization for Qwen2.5 applications.

## Installation

To install Qwen Tokenizer, use the following command:

```bash
pip install qwen_tokenizer
```

## Basic Usage

Below is a simple example demonstrating how to use the Qwen3.6 tokenizer:

```python
from qwen_tokenizer import get_tokenizer


qwen_tokenizer = get_tokenizer("qwen3.6-27b")
text = "Hello! 毕老师！1 + 1 = 2 ĠÑĤÐ²ÑĬÑĢ"

result = qwen_tokenizer.encode(text)
print(result)
```

### Output

```
[9419, 0, 220, 97169, 97621, 6115, 16, 478, 220, 16, 283, 220, 17, 9551, 254, 68933, 128, 97, 91914, 28495, 68933, 128, 105, 68933, 128, 95]
```

Hugging Face-style model IDs are also accepted:

```python
qwen_tokenizer = get_tokenizer("Qwen/Qwen3.6-27B")
```

Qwen3.5 and Qwen3.6 use a 248,044-token base vocabulary plus 33 added tokens. For example, `<|im_start|>`, `<think>`, and `</think>` map to IDs `248045`, `248068`, and `248069` respectively.

## License

This project is licensed under the MIT License.
