Metadata-Version: 2.1
Name: tokenspeed
Version: 0.1.0
Summary: TokenSpeed is a speed-of-light LLM inference engine.
License: MIT License
        
        Copyright (c) 2026 LightSeek Foundation <contact@lightseek.org>
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/lightseekorg/tokenspeed
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRDPARTYNOTICES
Requires-Dist: aiohttp
Requires-Dist: compressed-tensors
Requires-Dist: dill
Requires-Dist: einops
Requires-Dist: fastapi
Requires-Dist: grpcio-health-checking==1.81.1
Requires-Dist: grpcio-reflection==1.81.1
Requires-Dist: hf-transfer
Requires-Dist: huggingface-hub
Requires-Dist: modelscope
Requires-Dist: msgspec
Requires-Dist: ninja
Requires-Dist: numpy
Requires-Dist: openai>=2.24.1
Requires-Dist: openai-harmony
Requires-Dist: orjson
Requires-Dist: packaging
Requires-Dist: partial-json-parser
Requires-Dist: peft
Requires-Dist: pillow
Requires-Dist: prometheus-client
Requires-Dist: psutil
Requires-Dist: pybase64
Requires-Dist: pybind11
Requires-Dist: pydantic
Requires-Dist: py-spy
Requires-Dist: pytest-asyncio
Requires-Dist: python-multipart
Requires-Dist: pyzmq
Requires-Dist: requests
Requires-Dist: setproctitle
Requires-Dist: tiktoken
Requires-Dist: tokenspeed-kernel>=0.1.3.dev0
Requires-Dist: tokenspeed-mooncake>=0.3.11.post20260527
Requires-Dist: tokenspeed-scheduler>=0.1.3
Requires-Dist: tokenspeed-smg==1.8.0.post20260720
Requires-Dist: tokenspeed-smg-grpc-proto==0.4.14.post20260720
Requires-Dist: tokenspeed-smg-grpc-servicer==0.7.0.post20260720
Requires-Dist: torch==2.11.0
Requires-Dist: torch-memory-saver==0.0.9.post1
Requires-Dist: torchvision
Requires-Dist: tqdm
Requires-Dist: transformers==5.12.0
Requires-Dist: uv
Requires-Dist: uvicorn
Requires-Dist: uvloop
Requires-Dist: xgrammar==0.2.2
Requires-Dist: viztracer

<p align="center">
  <img src="./assets/banner/tokenspeed-banner.png" alt="TokenSpeed: Tokens at the speed of light" width="100%" />
</p>

TokenSpeed is a speed-of-light LLM inference engine designed for **agentic workloads**, with TensorRT-LLM-level performance and vLLM-level usability. Our goal is to be the most performant inference engine for production agentic workloads.

Core components:

- **Modeling layer**: local-SPMD design with a static compiler that generates
  collective communication from module-boundary placement annotations, so users
  do not hand-write parallelism logic.
- **Scheduler**: C++ control plane and Python execution plane. Request
  lifecycle, KV cache ownership, and overlap timing are encoded as a
  finite-state machine, with safe KV resource reuse enforced by the type system at compile time.
- **Kernels**: pluggable, layered kernel system with a portable public API and
  a centralized registry including one of the fastest **MLA**
  (Multi-head Latent Attention) implementations on Blackwell for agentic workload.
- **Entrypoint**: SMG-integrated AsyncLLM for low-overhead CPU-side request
  handling.

## News

- [2026/07] [TML Inkling](https://thinkingmachines.ai/news/introducing-inkling/) at Day 0: FP4 Inference on NVIDIA and [AMD](https://huggingface.co/lightseekorg/Inkling-MXFP4) with [TokenSpeed](https://thinkingmachines.ai/news/introducing-inkling/#inkling-availability). [[blog](https://lightseek.org/blog/tokenspeed-inkling.html)]
- [2026/06] Deep dive into the design and optimization of TokenSpeed-Kernel. [[blog](https://pytorch.org/blog/lightseek-tokenspeed-kernel/)]
- [2026/05] 🚀 TokenSpeed hits 580 TPS on Qwen3.5-397B-A17B for agentic workloads. [[blog](https://pytorch.org/blog/up-to-580tps-new-speed-record-of-qwen3-5-397b-a17b-on-gpu-for-agentic-workloads-with-tokenspeed/)]
- [2026/05] TokenSpeed announced — a speed-of-light LLM inference engine for agentic workloads. [[blog](https://lightseek.org/blog/lightseek-tokenspeed.html)]

## Blogs and Talks

For technical blogs, conference talks, and engineering articles from LightSeek Foundation, visit the [LightSeek Blog](https://lightseek.org/blog/).

## Performance Comparison

<img src="./assets/perf/tokenspeed-kimi-k2.5-performance.png" alt="TokenSpeed vs. TensorRT-LLM Pareto curves on agentic workload (Kimi K2.5, B200)" width="800" margin="10px"></img>

## Documentation

Start here:

- [Docs Index](https://lightseek.org/tokenspeed/)
- [Getting Started](https://lightseek.org/tokenspeed/guides/getting-started)
- [Launching a Server](https://lightseek.org/tokenspeed/guides/launching)
- [Model Recipes](https://lightseek.org/tokenspeed/recipes/models)
- [Server Parameters](https://lightseek.org/tokenspeed/configuration/server)
- [Compatible Parameters](https://lightseek.org/tokenspeed/configuration/compatible-parameters)
- [Parallelism](https://lightseek.org/tokenspeed/serving/parallelism)
