Metadata-Version: 2.4
Name: sglang-fs-linker
Version: 0.1.0
Summary: Out-of-tree filesystem UnifiedCacheLinker for SGLang
Author: maobaolong
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/maobaolong/sglang-fs-linker
Project-URL: Repository, https://github.com/maobaolong/sglang-fs-linker
Project-URL: Issues, https://github.com/maobaolong/sglang-fs-linker/issues
Keywords: sglang,kv-cache,filesystem,linker
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Dynamic: license-file

# SGLang filesystem linker

Implement `UnifiedCacheLinker` in a separately installed Python package and
select its class and module at launch. You do not need to register the class
inside SGLang or modify its backend selection code. This follows the explicit
class/module/extra-options pattern of vLLM's KV connector configuration.

## Install and launch

Use Linux with an NVIDIA CUDA GPU and a SGLang environment with the external
linker loader from [SGLang PR #40595](https://github.com/sgl-project/sglang/pull/40595).
This package does not install or replace SGLang or its CUDA-specific PyTorch
runtime. No tagged SGLang release is claimed to support the loader yet.
The tested SGLang revision is `9f417bd5054f3ed2db54eeb07c741df6ff7c255c`. The example uses one GPU (TP=DP=PP=CP=1), ordinary MHA/GQA in NHD layout,
and float16, bfloat16, or float32 KV. SWA, MLA, Mamba, quantized KV, unified memory,
and speculative decoding are intentionally unsupported by this example.

Install the independently maintained package into that environment:

```bash
python -m pip install sglang-fs-linker==0.1.0
python -c 'from sglang_fs_linker.linker import FileSystemLinker; print(FileSystemLinker.__module__)'
```

To install from this repository instead, run `python -m pip install .` from its
root. SGLang itself does not contain or register this package.

Start SGLang with a public model (no Hugging Face token required):

```bash
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-0.6B \
  --revision c1899de289a04d12100db370d81485cdf75e47ca \
  --host 127.0.0.1 --port 30000 \
  --tp 1 --page-size 16 --attention-backend triton \
  --disable-cuda-graph --mem-fraction-static 0.3 \
  --max-total-tokens 4096 --enable-cache-report \
  --enable-unified-cache-external-linker \
  --unified-cache-external-linker-config '{
    "linker": "FileSystemLinker",
    "linker_module_path": "sglang_fs_linker.linker",
    "linker_extra_config": {
      "directory": "/tmp/sglang-fs-kv",
      "namespace": "qwen3-0.6b-example-v1"
    }
  }'
```

The JSON has two required strings, `linker` and `linker_module_path`, and an
optional object, `linker_extra_config`. Install the plugin and its dependencies
in every worker environment. Only an explicitly selected module is imported,
inside the cache-owning worker; frontend configuration validation does not
import it. Use trusted plugin packages, since importing them executes Python.

This config takes precedence over `--unified-cache-external-linker-backend`.
Without the JSON config, existing `mooncake` (default) and `mori` launches keep
their constructor and behavior. The enable flag is still required. The
Mooncake-specific `--enable-linker-mla-dedup` cannot be used with this config.
There is no `kv_role`: this interface provides both lookup/load and offload.

## Implement a linker

Subclass `sglang.srt.mem_cache.unified_cache.unified_cache_linker.UnifiedCacheLinker`
and provide this constructor:

```python
def __init__(self, server_args, params, *, components, extra_config):
    ...
```

`params` is `CacheInitParams`, including the worker's device allocator and rank
groups. `components` is a set of active `ComponentType` values. `extra_config`
is a private copy of the JSON options. Validate your supported model layouts and
topology before allocating resources. The abstract interface documents all
required methods, including reset and close.

`PoolTransfer.device_indices` identifies the GPU slots; `keys` identifies the
corresponding prefix-hashed pages. A load must wait for
`start_layer_wise_loading()` before copying into those slots. Publish completion
only after device writes are safe, because the tree releases locks on completion.
The `layer_done_counter` provides `set_consumer(index)` and `wait_until(layer)`
to the worker and KV pool. Asynchronous implementations must synchronize each
layer correctly; this example finishes all copies synchronously before returning.

SGLang continues to own prefix matching, GPU allocation, admission and tree locks.
The plugin owns storage and transfer completion. Existing Full/SWA component
restrictions in the tree wrapper still apply; the plugin mechanism does not
extend model support by itself.

## Verify disk reuse

With the server above running, use another shell in the same environment:

```bash
sglang-fs-linker-verify --base-url http://127.0.0.1:30000
```

The verifier sends a cold request with a unique cache salt, flushes the GPU radix
cache, then repeats the request. It requires a positive external hit in
`meta_info.cached_tokens_details.host`, zero device-cache hits, identical output
tokens, and output log probabilities within 0.05. The server also reports
`FileSystemLinker loaded N pages from disk`. A warm request without a radix flush
would only demonstrate local prefix caching.

## Development and verification

Clone both repositories into a parent directory. Check out the SGLang revision
with the loader; the FS package stays outside that checkout:

```bash
git clone --branch feat/out-of-tree-unified-cache-linker https://github.com/maobaolong/sglang.git
git -C sglang checkout 9f417bd5054f3ed2db54eeb07c741df6ff7c255c
git clone https://github.com/maobaolong/sglang-fs-linker.git
```

For the NVIDIA H200 validation environment, use Docker with NVIDIA Container
Toolkit. Run this from the parent directory containing both repositories:

```bash
docker run --rm -it --gpus all --ipc=host \
  -v "$PWD:/work" -w /work \
  --entrypoint bash \
  lmsysorg/sglang@sha256:9e148f5ac788e856a06166bd6347a831831eb9fcfab4d1770874823a7c29a1a1
```

Inside the container, select one available GPU and prepare the runtime:

```bash
export CUDA_VISIBLE_DEVICES=0
export PYTHONPATH=/work/sglang/python
python -m pip install --no-deps sglang-kernel==0.4.7
python -m pip install /work/sglang-fs-linker
python -c 'import sglang; print(sglang.__file__)'
python -m pytest -q /work/sglang-fs-linker/tests
```

The printed SGLang path must belong to the mounted checkout. The image provides
Python 3.12 and PyTorch 2.13.0+cu130, but its preinstalled kernel package is older
than the tested checkout requires. No native rebuild is needed. Run the launch
command above; run the installed verifier in a second shell in that container.

The filesystem tests use real CPU tensors, temporary files, and separately
constructed producer/consumer instances. They check exact KV restoration into
different slots, persistence across reset/close, failed writes, missing prefixes,
cancellation, and corrupt pages. No Mooncake or MORI service is needed.

The SGLang loader and existing linker regressions remain in SGLang. In the same
container, run them separately:

```bash
cd /work/sglang
python -m pytest -q \
  test/registered/unit/mem_cache/test_external_linker_factory.py \
  test/registered/unit/mem_cache/test_registry.py \
  test/registered/unit/server_args/test_server_args.py \
  test/registered/unit/mem_cache/test_unified_cache_linker.py \
  test/registered/unit/mem_cache/test_linker_pool_assembler.py
```

For packaging-only checks (no GPU or SGLang required), run these commands from
the `sglang-fs-linker` repository root:

```bash
uv build
uvx twine check --strict dist/*
uvx --from ruff==0.15.1 ruff check src tests
uvx --from ruff==0.15.1 ruff format --check src tests
```

## Filesystem example limits

Each file contains all K/V layers of one page. A temporary file followed by
atomic replacement keeps incomplete pages invisible. Reset and close preserve
files, so the cache can survive worker restarts. The namespace, model path, page
size, dtype, and tensor geometry isolate files; change `namespace` whenever model
weights, revision, adapters, or RoPE settings change. Do not reuse a namespace
across incompatible model configurations.

This is a validation example, not a performance backend: transfers synchronize
the device, stage through CPU tensors, and perform blocking filesystem I/O. It
has no disk eviction, capacity limit, or crash-durability guarantee. Remove its
directory only when its workers are stopped. Missing files are misses before a
load is queued; deletion or corruption after admission fails the load instead of
allowing inference on incomplete KV. Only use a directory managed by trusted
processes.
