Metadata-Version: 2.4
Name: lk_moe
Version: 2.3.0
Summary: A highly optimized MOE hybrid inference engine for CPU-GPU collaborative execution of large-scale MOE models.
License: Proprietary
Project-URL: Homepage, https://pypi.org/project/lk-moe/
Requires-Python: >=3.10,<3.14
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRD_PARTY_LICENSES
Dynamic: license
Dynamic: license-file
Dynamic: requires-python

## lk_moe

[lk_moe](https://pypi.org/project/lk-moe/) is a highly optimized MOE hybrid inference engine that serves as the computational core for MOE layer execution. It is designed to maximize resource utilization across CPU, memory, and GPU, enabling large-scale MOE models to run efficiently beyond GPU memory constraints.

### Key Capabilities

- **Hybrid Compute Backend**: Each MOE layer can dynamically choose between the original GPU computation path or lk_moe's hybrid inference path, offering flexible deployment strategies.
- **CPU-GPU Collaborative Inference**: Distributes expert computation across CPU (via NUMA-optimized threading) and GPU, reducing GPU memory requirements while maintaining high throughput.
- **NUMA-Aware Parallelism**: Optimized thread binding and memory allocation across NUMA nodes, achieving cross-node communication as low as 3% and L3 cache hit rates above 50%.
- **Quantization Kernel Support**: Native support for multiple quantization formats including BF16, FP8, NVFP4, MXFP4, and AWQ 4-bit, with efficient kernel implementations.
- **Memory + VRAM Load Balancing**: Total model footprint spans both VRAM and system memory, allowing models larger than GPU memory to run without performance collapse.

### Framework Integration

lk_moe integrates with popular LLM serving frameworks as a drop-in MOE computation backend:

| Framework | Integration | Description |
|-----------|-------------|-------------|
| **vLLM** | [LvLLM](https://github.com/guqiong96/Lvllm) | lk_moe as MOE layer backend for vLLM serving |
| **sglang** | [Lsglang](https://github.com/guqiong96/Lsglang) | lk_moe as MOE layer backend for sglang serving |
| **DeepSeek V4** | [Lvllmds4](https://github.com/guqiong96/Lvllmds4) | Specialized integration for DeepSeek V4 (SM120) |

### Installation

``bash
pip install lk-moe
``


### License

lk_moe is provided under a proprietary software license agreement. See [LICENSE](./LICENSE) for terms and conditions. Third-party open-source components and their respective licenses are listed in [THIRD_PARTY_LICENSES](./THIRD_PARTY_LICENSES).
