turbo_attn (imported as `tkv`)
Copyright 2026 Arbi City (Dmitri Evseev <dmitri.evseev@arbi.city>)

This software is licensed under the Apache License, Version 2.0. You may
obtain a copy of the License at:

    http://www.apache.org/licenses/LICENSE-2.0

A copy of the License is included as `LICENSE` at the root of this repository.

================================================================================
ORIGINATED CONTRIBUTIONS
================================================================================

The following techniques and components were originated by Arbi City and
are released under Apache-2.0; attribution to Arbi City for these
techniques is requested as a courtesy (see `ATTRIBUTION.md` for
technique-by-technique provenance, novelty notes, and file references)
though not itself a condition of the Apache-2.0 license.

Kernel-level contributions (Arbi City, 2026):

  * TKV inline-dequant load_K/load_V override — dequantizes compressed
    KV bytes directly into MMA register tiles at SMEM-fill time,
    eliminating a decompress buffer. It is implemented as a loader
    mixin over turbo_attn's own CuTeDSL prefill mainloop
    (`TurboPrefillForward`), not over any third-party attention
    kernel. Files:
    tkv/kernels/cuda/prefill/turbo_prefill_load_tkv.py (`_TkvLoadMixin`),
    tkv/kernels/loaders/tkv.py (`tkv_fill`).

  * V4 split-D Q-once smem layout — full sQ + aliased sKV (96 KB)
    allowing K and V to share a single 32 KB chunk-sized smem region
    sequentially under barrier protection.

  * Dual-kernel SWA decode dispatcher — split-K kernel for sliding-window
    layers at sw=0; unified BLOCK_M=1 kernel for sw>0 SWA layers.
    Files: tkv/kernels/_cuda_decode_splitk.cu,
    tkv/kernels/_cuda_decode_unified.cu.

  * Cooperative bf16→fp32 smem dtype swap in the decode kernel.

  * Multi-warp compress pack at w=2 (occupancy 41.5% → 80.7%).
    File: tkv/kernels/cuda_compress_store.py.

  * Per-layer bit-allocation pipeline using logit-KLD distortion +
    greedy bang-per-byte solver, with `lossless` / `balanced` /
    `aggressive` profiles. Files: tkv/calibration/optimize_quant.py,
    tkv/calibration/calibrate_centroids.py.

  * Fisher-weighted Lloyd's calibration using attention-mass as the
    weighting proxy. Files: tkv/calibrate_online.py and
    tkv/calibration/.

Integration-level contributions (Arbi City, 2026):

  * vLLM v1 attention-backend plugin with FULL_AND_PIECEWISE CUDAGraph
    capture across both prefill and decode.
    File: tkv/integrations/vllm/plugin.py, backend.py.

  * Per-group BlockPool patch for hybrid models (attention + Mamba/GDN);
    proposed upstream to vLLM.

  * SGLang attention-backend integration (pool, backend, metadata,
    CUDAGraph). Files: tkv/integrations/sglang/.

  * Hybrid GatedDeltaNet support end-to-end (validated on Qwen3.5-27B-AWQ
    TP=2 at 1.36M context).

  * Hybrid MoE support end-to-end (validated on LFM2-8B-A1B and
    Qwen3.6-MoE TP=2).

================================================================================
THIRD-PARTY COMPONENTS AND ATTRIBUTION
================================================================================

This product includes software developed by third parties listed below.
Their licenses apply to those components in addition to the Apache-2.0
License covering this repository as a whole.

--------------------------------------------------------------------------------
TurboQuant codec
--------------------------------------------------------------------------------

The codec layer (Walsh-Hadamard rotation, Lloyd-Max codebook construction,
sign-removed rotation, n_centroids parameterization) is the contribution
of Zandieh et al., "TurboQuant" (ICLR 2026), and underlies turbo_attn's
compression scheme. turbo_attn extends the codec with:

  * bf16 norm storage (was fp32 in initial drafts)
  * Multi-corpus calibration (c4 + chat + code + math, multi-run pooled)
  * Per-layer per-(k,v) bit allocation via logit-KLD distortion
  * Fisher-weighted Lloyd's centroid fitting
  * Asymmetric K/V across all nine {2,4,8}² combinations

See ATTRIBUTION.md §"Codec layer" for the technique-by-technique split.

--------------------------------------------------------------------------------
FlashAttention-4 CuTeDSL support modules (vendored)
--------------------------------------------------------------------------------

Directory: `tkv/kernels/cute/_fa/`

This directory contains a vendored fork of twelve SUPPORT modules from
the FlashAttention-4 CuTeDSL sources (SM80-class: Ampere / Ada /
Blackwell GeForce only) by Tri Dao and the FlashAttention team,
distributed under the BSD-3-Clause license. The original LICENSE is
preserved at `tkv/kernels/cute/_fa/LICENSE`.

The vendored copy is pinned to upstream commit
`15270e66dc88d70d08f6ba6003eb17e7307149b0` (2026-04-03). Every one of
the twelve modules is reachable from — and used by — the Turbo prefill
kernels: `ampere_helpers`, `block_info`, `block_sparsity`,
`cute_dsl_utils`, `fast_math`, `mask`, `named_barrier`, `pack_gqa`,
`seqlen_info`, `softmax`, `tile_scheduler`, `utils`. Nine are verbatim upstream
(module-path rewrite only); three — `pack_gqa.py`, `softmax.py`,
`utils.py` — carry local patches. See
`tkv/kernels/cute/_fa/UPSTREAM.md` for the exact per-file provenance,
the baseline SHA and how it was established, and the full patch ledger.

What this directory vendors is a utility substrate, not an attention
mainloop. No FlashAttention forward-pass kernel class is forked here or
anywhere else in this repository, and nothing under `tkv/` imports the
`flash_attn` package. turbo_attn's prefill kernels
(`TurboPrefillForward` and its Bypass / Hybrid subclasses) are
original CuTeDSL kernels: they own their mainloop, epilogue, base class
and parameter plumbing, and extend only turbo_attn's own
`_TurboPrefillBase`.
What they take from this directory is the surrounding substrate —
masks, softmax, seqlen/block info, pack-GQA, named barriers, block
sparsity, tile schedulers and the SM80 gemm helpers. Parts of
`tkv/kernels/cuda/prefill/turbo_prefill_base.py`
(`_TurboPrefillBase.__init__`'s
attention-shape plumbing, `_check_type`, `advance_pipeline`) also
derive from this BSD-3-Clause source; that file carries an in-place
attribution notice.

The TQ-specific inline-dequant logic itself (see the ORIGINATED
CONTRIBUTIONS section above) lives entirely in
`tkv/kernels/cuda/prefill/turbo_prefill_load_tkv.py` and
`tkv/kernels/loaders/tkv.py`, mixed into turbo_attn's own
`TurboPrefillForward` mainloop rather than into any FlashAttention
class.

The `flash-attn-4` PyPI package (the `fa4` optional extra) is a
benchmark-only dependency used to compare against upstream; no
turbo_attn runtime path imports it.

--------------------------------------------------------------------------------
vLLM
--------------------------------------------------------------------------------

turbo_attn registers as a vLLM attention backend via vLLM's plugin
entry-point system. vLLM is © 2023 the vLLM team and is distributed
under the Apache-2.0 license. See https://github.com/vllm-project/vllm.

A small set of patches against vLLM is required while upstream merges
the `CacheDType` Literal relaxation; see `docker/PATCHES.md`.

--------------------------------------------------------------------------------
SGLang
--------------------------------------------------------------------------------

turbo_attn registers as an SGLang attention backend via a runtime
monkey-patch of `_init_pools`. SGLang is © the SGLang Project and is
distributed under the Apache-2.0 license. See
https://github.com/sgl-project/sglang.

================================================================================
CITATION
================================================================================

If you build on turbo_attn — kernels, integration patterns, or the
per-layer bit-allocation pipeline — please cite this work via the
metadata in `CITATION.cff` and credit the originating contribution
listed above. If you build on the underlying TurboQuant codec, please
cite Zandieh et al., ICLR 2026, in addition.

Per the Apache License, Version 2.0, §4(d), this NOTICE file must be
reproduced as part of any redistribution that includes a NOTICE file.
