Metadata-Version: 2.4
Name: prunetokens
Version: 0.1.4
Summary: Prune unused tokens from LLM models
Description-Content-Type: text/markdown
Requires-Dist: safetensors>=0.4.0
Requires-Dist: torch>=2.0.0
Dynamic: description
Dynamic: description-content-type
Dynamic: requires-dist
Dynamic: summary

## Remove all non-English tokens from the model

`pip install prunetokens`

`prunetokens /path/to/model_directory_with_safetensors`

Requirements: `safetensors` to read and write file. No VRAM required.

| Model | Before | After | Saved | Vocab |
|---|---|---|---|---|
| Qwen3.5-9B-AWQ-4bit | 8.5 GB | 7.0 GB | ~18% | 248k → 146k |
| MiniCPM5-1B | 2.1 GB | 1.9 GB | ~10% | 130k → 94k |

***

Pruned models:
- https://huggingface.co/albex123/Qwen3.5-9B-AWQ-4bit-pruned

## UPDATE 0.1.4:
|         Model       | Before | After  |Saved | Vocab       |
|---------------------|--------|--------|------|-------------|
| Qwen3.5-9B-AWQ-4bit | 8.5 GB | 6.8 GB | ~20% | 248k → 137k |
| Qwen-3.5-4B-AWQ | 4.1 GB | 3.5 GB | ~14% | 248k → 138k |
