Metadata-Version: 2.4
Name: aegaeon
Version: 0.1.0
Summary: Aegaeon: A multi-LLM inference engine that multiplexes requests at the token level.
Home-page: https://github.com/pkusys/aegaeon
Author: Alibaba Team
License: Apache-2.0
Requires-Python: >=3.10,<3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp>=3.9
Requires-Dist: fastapi>=0.115
Requires-Dist: pydantic>=2.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: uvicorn>=0.30
Requires-Dist: modelscope>=1.22.0
Requires-Dist: matplotlib>=3.10.0
Requires-Dist: numpy
Requires-Dist: psutil
Requires-Dist: ray>=2.23.0
Requires-Dist: safetensors>=0.4.5
Requires-Dist: scikit_learn>=1.6.0
Requires-Dist: tomli; python_version < "3.11"
Requires-Dist: viztracer
Requires-Dist: simpy==4.1.1
Requires-Dist: aenum
Requires-Dist: torch==2.11.0+cu129
Requires-Dist: vllm==0.26.0+cu129
Requires-Dist: torchaudio==2.11.0+cu129
Requires-Dist: torchvision==0.26.0+cu129
Requires-Dist: torchcodec==0.15.0+cu129
Requires-Dist: cupy-cuda12x
Dynamic: author
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<h1 align="center">Aegaeon</h1>

<p align="center">
  <strong>Serve many LLMs on a shared GPU pool with token-level autoscaling.</strong>
</p>

<p align="center">
  <a href="https://doi.org/10.1145/3731569.3764815">Paper</a> ·
  <a href="https://asceticism-pku.github.io/aegaeon-preview/en/">Documentation</a>
</p>

## Why Aegaeon?

Model-as-a-Service platforms must serve a long tail of LLMs with sparse and unpredictable traffic. Existing GPU pooling approaches either colocate models, which GPU memory limits to two or three models per GPU, or autoscale at request boundaries, where many concurrently active models cause head-of-line blocking.

The SOSP '25 paper [*Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market*](https://doi.org/10.1145/3731569.3764815) presents Aegaeon, a multi-LLM serving system that moves model autoscaling to token granularity. Its token-level scheduler and full-stack optimizations for preemptive model switching enable up to seven models per GPU. In evaluation, Aegaeon sustained 2–2.5× higher request arrival rates or 1.5–9× higher goodput than existing systems; in its Alibaba Cloud Model Studio beta deployment, it reduced the GPU count from 1,192 to 213, an 82% saving.

### Key Features

- **Token-level autoscaling** — switches among active models between token steps instead of waiting for whole requests to finish.
- **SLO-aware scheduling** — coordinates Prefill and Decode work against TTFT and TPOT objectives.
- **Fast model switching** — combines host-side weight caching, prefetching, a Rust quick loader, and explicit KV-cache movement.
- **Multi-model runtime** — supports dynamic model deployment, multiple replicas, and load-aware request routing.
- **vLLM backend** — reuses model implementations and compute kernels across dense, hybrid, and MoE architectures.
- **Serving and observability** — provides OpenAI-style Chat and Completion APIs, SSE streaming, and runtime state.

### Demo

The demo compares Aegaeon with a request-level baseline while both serve the same two models. It shows model switches and streamed outputs.

<p align="center">
  <img src="./.readme/aegaeon-demo.gif" alt="Aegaeon multi-model serving demo" width="900">
</p>

## Quick Start

### Install the CUDA 12.9 release from PyPI

In a fresh Python 3.10 environment on Linux x86_64 with glibc >=2.31, install the published version and its matching CUDA 12.9 dependencies with one command:

```bash
python -m pip install aegaeon==0.1.0 --extra-index-url https://download.pytorch.org/whl/cu129 --extra-index-url https://wheels.vllm.ai/568afb3a13806beb53bb2e6bd518269357b237c0/cu129
```

The extra indexes are the official PyTorch CUDA 12.9 index and the pinned vLLM CUDA 12.9 build index. Aegaeon is downloaded from PyPI; pip installs all declared runtime dependencies automatically. No repository checkout or requirements file is needed for this installation. Keep both indexes: the CUDA variants pinned by this release are not all available on the default PyPI index.

The wheel contains Aegaeon's precompiled CUDA and Rust extensions. Installing it does not require Aegaeon's CUDA/C++/Rust build toolchain. A compatible NVIDIA driver is required. This release was built for compute capability 8.0 and validated on NVIDIA A100; other GPU architectures and the CUDA 13.0 ARM wheel are not validated. See the [installation guide](https://asceticism-pku.github.io/aegaeon-preview/en/installation/) for source builds and checks.

### Build from source

Use Linux x86_64 and CPython 3.10. Prepare the matching CUDA Toolkit, C++ compiler, and Rust/Cargo as described in the [installation guide](docs/en/installation.md), then run from the repository root in a fresh environment.

Choose **one** CUDA version:

```bash
# CUDA 12.9
python -m pip install -r requirements-cu129.txt
AEGAEON_CUDA=cu129 python -m pip install --no-build-isolation -e .
```

```bash
# CUDA 13.0 (GPU serving still requires validation)
python -m pip install -r requirements-cu130.txt
AEGAEON_CUDA=cu130 python -m pip install --no-build-isolation -e .
```

Follow the [Quickstart](docs/en/quickstart.md) to prepare a model and send your first request. Driver requirements and manual toolchain commands are in the [installation guide](docs/en/installation.md).

## Documentation

| Topic | Guide |
|---|---|
| Installation and first request | [Installation](https://asceticism-pku.github.io/aegaeon-preview/en/installation/) · [Quickstart](https://asceticism-pku.github.io/aegaeon-preview/en/quickstart/) |
| Supported models and limitations | [Features and Compatibility](https://asceticism-pku.github.io/aegaeon-preview/en/capabilities/) · [Model Preparation](https://asceticism-pku.github.io/aegaeon-preview/en/models/) |
| Deployment and model lifecycle | [Distributed Deployment](https://asceticism-pku.github.io/aegaeon-preview/en/deployment/) · [Model Management](https://asceticism-pku.github.io/aegaeon-preview/en/model-management/) |
| API usage | [Online Inference](https://asceticism-pku.github.io/aegaeon-preview/en/serving/) · [HTTP API](https://asceticism-pku.github.io/aegaeon-preview/en/api/) |
| Internals and contributions | [Architecture](https://asceticism-pku.github.io/aegaeon-preview/en/architecture/) · [Development](https://asceticism-pku.github.io/aegaeon-preview/en/development/) |
