Metadata-Version: 2.4
Name: freetoken
Version: 0.1.1
Summary: FreeToken inference runtime
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/FlashML-org/FreeToken
Project-URL: Repository, https://github.com/FlashML-org/FreeToken
Project-URL: Issues, https://github.com/FlashML-org/FreeToken/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: apache-tvm-ffi==0.1.13.post3
Requires-Dist: einops<1,>=0.8
Requires-Dist: fastapi<1,>=0.115
Requires-Dist: flashlib==0.3.0
Requires-Dist: gguf<1,>=0.19
Requires-Dist: huggingface_hub<2,>=1.5
Requires-Dist: msgpack<2,>=1.1
Requires-Dist: modelscope<2,>=1.37
Requires-Dist: numpy<2.5,>=2.0
Requires-Dist: openai<3,>=2.0
Requires-Dist: partial-json-parser<1,>=0.2
Requires-Dist: prompt_toolkit<4,>=3.0
Requires-Dist: pydantic<3,>=2.9
Requires-Dist: pyzmq<28,>=27
Requires-Dist: safetensors<1,>=0.6
Requires-Dist: torch<2.12,>=2.11
Requires-Dist: tqdm<5,>=4.66
Requires-Dist: transformers<6,>=5.5
Requires-Dist: triton==3.6.0
Requires-Dist: uvicorn<1,>=0.30
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Provides-Extra: fi
Requires-Dist: flashinfer-python[cu13]<0.7,>=0.6; extra == "fi"
Provides-Extra: sgl
Requires-Dist: sglang-kernel==0.4.5; extra == "sgl"
Provides-Extra: accel
Requires-Dist: freetoken[fi,sgl]; extra == "accel"
Dynamic: license-file

<div align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/FlashML-org/FreeToken/main/assets/freetoken-logo-dark.svg">
    <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/FlashML-org/FreeToken/main/assets/freetoken-logo-light.svg">
    <img alt="FreeToken" src="https://raw.githubusercontent.com/FlashML-org/FreeToken/main/assets/freetoken-logo.svg" width=55%>
  </picture>
  
  [![Slack](https://img.shields.io/badge/Slack-join%20the%20community-4A154B?logo=slack&logoColor=white)](https://join.slack.com/t/flashml/shared_invite/zt-3zpdh5j10-9dwTXrgLiqpVxizhA9KVbA)
[![Discord](https://img.shields.io/badge/Discord-join%20the%20community-5865F2?logo=discord&logoColor=white)](https://discord.gg/xzwSnMdsX)
</div>



A local, MoE-offload inference runtime with an OpenAI- and Anthropic-compatible
HTTP API — Run DeepSeek-V4-Flash on your 5090 with 20+ TPS.



## Quick start

See [docs/install.md](https://github.com/FlashML-org/FreeToken/blob/main/docs/install.md) for requirements and installation.

```bash
ft serve --model ~/models/Qwen3.6-35B-A3B   # API server on http://127.0.0.1:1919
ft launch claude                            # point an agent at it (codex / opencode / openclaw)
ft shell                                    # or chat in the terminal
```

## Documentation

- [Install](https://github.com/FlashML-org/FreeToken/blob/main/docs/install.md) — requirements and setup
- [Supported models](https://github.com/FlashML-org/FreeToken/blob/main/docs/models.md) — model × quantization
- [CLI reference](https://github.com/FlashML-org/FreeToken/blob/main/docs/cli.md) — `ft` commands and environment variables

## Acknowledgment

FreeToken was deeply inspired by [mini-sglang](https://github.com/sgl-project/mini-sglang), and
learned the design and reused code from the following projects:
[SGLang](https://github.com/sgl-project/sglang),
[vLLM](https://github.com/vllm-project/vllm),
[FlashInfer](https://github.com/flashinfer-ai/flashinfer),
[flash-linear-attention](https://github.com/fla-org/flash-linear-attention),
[LightLLM](https://github.com/ModelTC/lightllm) and [llama.cpp](https://github.com/ggml-org/llama.cpp).

## License

[Apache License 2.0](https://github.com/FlashML-org/FreeToken/blob/main/LICENSE).
