turbo_attn (imported as `tkv`)
Copyright 2026 Arbi City (Dmitri Evseev <dmitri.evseev@arbi.city>)

This software is licensed under the Apache License, Version 2.0. You may
obtain a copy of the License at:

    http://www.apache.org/licenses/LICENSE-2.0

A copy of the License is included as `LICENSE` at the root of this repository.

================================================================================
ORIGINATED CONTRIBUTIONS
================================================================================

The following techniques and components were originated by Arbi City and
are released under Apache-2.0; attribution to Arbi City for these
techniques is requested as a courtesy (see `ATTRIBUTION.md` for
technique-by-technique provenance, novelty notes, and file references)
though not itself a condition of the Apache-2.0 license.

Kernel-level contributions (Arbi City, 2026):

  * TKV inline-dequant load_K/load_V override — dequantizes compressed
    KV bytes directly into MMA register tiles at SMEM-fill time,
    eliminating a decompress buffer. Originally implemented as a
    subclass of FlashAttention-4's own CuTeDSL forward mainloop
    (`tkv/kernels/cuda/prefill/fa4_tq.py`, `FlashAttentionForwardSm120TQ`);
    that FA4-mainloop-coupled fallback was deleted in a 2026-07 re-scope
    (no public flash-attn-4 API accepts this technique's packed/norm/
    centroid K/V, so it could not be rebuilt against the real upstream
    package) — the technique itself lives on in the owned native
    mainloop's loader mixin. Files:
    tkv/kernels/cuda/prefill/fa4_tq_load.py (`_TQLoadMixin`),
    tkv/kernels/loaders/tkv.py (`tkv_fill`).

  * V4 split-D Q-once smem layout — full sQ + aliased sKV (96 KB)
    allowing K and V to share a single 32 KB chunk-sized smem region
    sequentially under barrier protection.

  * Dual-kernel SWA decode dispatcher — split-K kernel for sliding-window
    layers at sw=0; unified BLOCK_M=1 kernel for sw>0 SWA layers.
    Files: tkv/kernels/_cuda_decode_splitk.cu,
    tkv/kernels/_cuda_decode_unified.cu.

  * Cooperative bf16→fp32 smem dtype swap in the decode kernel.

  * Multi-warp compress pack at w=2 (occupancy 41.5% → 80.7%).
    File: tkv/kernels/cuda_compress_store.py.

  * Per-layer bit-allocation pipeline using logit-KLD distortion +
    greedy bang-per-byte solver, with `lossless` / `balanced` /
    `aggressive` profiles. Files: tkv/calibration/optimize_quant.py,
    tkv/calibration/calibrate_centroids.py.

  * Fisher-weighted Lloyd's calibration using attention-mass as the
    weighting proxy. Files: tkv/calibrate_online.py and
    tkv/calibration/.

Integration-level contributions (Arbi City, 2026):

  * vLLM v1 attention-backend plugin with FULL_AND_PIECEWISE CUDAGraph
    capture across both prefill and decode.
    File: tkv/integrations/vllm/plugin.py, backend.py.

  * Per-group BlockPool patch for hybrid models (attention + Mamba/GDN);
    proposed upstream to vLLM.

  * SGLang attention-backend integration (pool, backend, metadata,
    CUDAGraph). Files: tkv/integrations/sglang/.

  * Hybrid GatedDeltaNet support end-to-end (validated on Qwen3.5-27B-AWQ
    TP=2 at 1.36M context).

  * Hybrid MoE support end-to-end (validated on LFM2-8B-A1B and
    Qwen3.6-MoE TP=2).

================================================================================
THIRD-PARTY COMPONENTS AND ATTRIBUTION
================================================================================

This product includes software developed by third parties listed below.
Their licenses apply to those components in addition to the Apache-2.0
License covering this repository as a whole.

--------------------------------------------------------------------------------
TurboQuant codec
--------------------------------------------------------------------------------

The codec layer (Walsh-Hadamard rotation, Lloyd-Max codebook construction,
sign-removed rotation, n_centroids parameterization) is the contribution
of Zandieh et al., "TurboQuant" (ICLR 2026), and underlies turbo_attn's
compression scheme. turbo_attn extends the codec with:

  * bf16 norm storage (was fp32 in initial drafts)
  * Multi-corpus calibration (c4 + chat + code + math, multi-run pooled)
  * Per-layer per-(k,v) bit allocation via logit-KLD distortion
  * Fisher-weighted Lloyd's centroid fitting
  * Asymmetric K/V across all nine {2,4,8}² combinations

See ATTRIBUTION.md §"Codec layer" for the technique-by-technique split.

--------------------------------------------------------------------------------
FlashAttention / FlashAttention-4 (vendored)
--------------------------------------------------------------------------------

Directory: `tkv/kernels/cute/_fa/`

This directory contains an owned hard-fork of the FlashAttention-4
CuTeDSL forward kernel (SM80-class: Ampere / Ada / Blackwell GeForce
only) by Tri Dao and the FlashAttention team, distributed under the
BSD-3-Clause license. The original LICENSE is preserved at
`tkv/kernels/cute/_fa/LICENSE`.

The fork is pinned to upstream commit
`15270e66dc88d70d08f6ba6003eb17e7307149b0` (2026-04-03). It carries
twelve modules, every one of them reachable from — and used by — the
Arbi kernels: `ampere_helpers`, `block_info`, `block_sparsity`,
`cute_dsl_utils`, `fast_math`, `mask`, `named_barrier`, `pack_gqa`,
`seqlen_info`, `softmax`, `tile_scheduler`, `utils`. Nine are verbatim
upstream (module-path rewrite only); three — `pack_gqa.py`,
`softmax.py`, `utils.py` — carry local patches. See
`tkv/kernels/cute/_fa/UPSTREAM.md` for the exact per-file provenance,
the baseline SHA and how it was established, and the full patch ledger.

No FlashAttention forward-pass kernel class is forked. The Arbi
kernels (`ArbiAttentionForward` and its Bypass / Hybrid subclasses)
own their mainloop, epilogue, base class and parameter plumbing; what
they take from this directory is the surrounding substrate — masks,
softmax, seqlen/block info, pack-GQA, named barriers, block sparsity,
tile schedulers and the SM80 gemm helpers. Parts of
`tkv/kernels/cuda/prefill/fa4_tq_base.py` (`_TQAttnBase.__init__`'s
attention-shape plumbing, `_check_type`, `advance_pipeline`) also
derive from this BSD-3-Clause source; that file carries an in-place
attribution notice.

The TQ-specific inline-dequant logic itself (see the ORIGINATED
CONTRIBUTIONS section above) lives entirely in
`tkv/kernels/cuda/prefill/fa4_tq_load.py` and
`tkv/kernels/loaders/tkv.py`, mixed into the OWNED native mainloop
(`ArbiAttentionForward`) rather than a subclass of any
FlashAttention-4 class.

--------------------------------------------------------------------------------
vLLM
--------------------------------------------------------------------------------

turbo_attn registers as a vLLM attention backend via vLLM's plugin
entry-point system. vLLM is © 2023 the vLLM team and is distributed
under the Apache-2.0 license. See https://github.com/vllm-project/vllm.

A small set of patches against vLLM is required while upstream merges
the `CacheDType` Literal relaxation; see `docker/PATCHES.md`.

--------------------------------------------------------------------------------
SGLang
--------------------------------------------------------------------------------

turbo_attn registers as an SGLang attention backend via a runtime
monkey-patch of `_init_pools`. SGLang is © the SGLang Project and is
distributed under the Apache-2.0 license. See
https://github.com/sgl-project/sglang.

================================================================================
CITATION
================================================================================

If you build on turbo_attn — kernels, integration patterns, or the
per-layer bit-allocation pipeline — please cite this work via the
metadata in `CITATION.cff` and credit the originating contribution
listed above. If you build on the underlying TurboQuant codec, please
cite Zandieh et al., ICLR 2026, in addition.

Per the Apache License, Version 2.0, §4(d), this NOTICE file must be
reproduced as part of any redistribution that includes a NOTICE file.
