Metadata-Version: 2.4
Name: sequential-speculative-decoding
Version: 0.1.1
Summary: Speculative decoding methods for efficient LLM inference: standard SD and hierarchical speculative decoding (HSD)
Author-email: Yijin Zeng <yijinzeng98@gmail.com>, Hazem Massoud <hazem_massoud@hotmail.com>
License-Expression: MIT
Project-URL: Repository, https://github.com/Yijin-Zeng/Sequential-Speculative-Decoding
Keywords: speculative-decoding,llm,inference,transformers
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.40
Dynamic: license-file

# Sequential Speculative Decoding

A Python package implementing standard and sequential speculative decoding for accelerating LLM inference
without changing the output distribution.

Speculative decoding uses a small, fast **draft model** to propose multiple tokens, then lets the larger **target model** verify them in a single forward pass. The result is the same text you'd get from the target model alone, but faster.

This package currently implements three speculative decoding methods:
- **Standard SD** — token-level verification (Leviathan et al., 2023)
- **Naive HSD** — sequence-level hierarchical verification (Zhou et al., 2026)
- **Capped HSD** — improved HSD with capped prefix ratios (Zhou et al., 2026)

## Installation

First, install PyTorch for your hardware (see [pytorch.org](https://pytorch.org/get-started/locally/) for the right CUDA). Then run:

```bash
pip install sequential-speculative-decoding
```

or:

```bash
git clone https://github.com/Yijin-Zeng/Sequential-Speculative-Decoding
cd Sequential-Speculative-Decoding
pip install -e .
```

## Quick Start

```python
from seqspecdecod import load_models, speculative_generate

tokenizer, draft_model, target_model = load_models(
    draft_name="Qwen/Qwen3-0.6B",
    target_name="Qwen/Qwen3-4B",
    device="cuda",
)

result = speculative_generate(
    draft_model=draft_model,
    target_model=target_model,
    tokenizer=tokenizer,
    prompt="Explain what machine learning is in one paragraph.",
    max_new_tokens=200,
    gamma=5,
    method="standard",
)

print(result["text"])
print(f"Speed: {result['tokens_per_sec']:.1f} tok/s")
print(f"Acceptance rate: {result['acceptance_rate']:.1%}")
```

## Benchmarking

Use `benchmark` to compare speculative decoding against the target-only baseline across multiple prompts:

```python
from seqspecdecod import load_models, benchmark

tokenizer, draft_model, target_model = load_models()

prompts = [
    "Give me a short introduction to large language models.",
    "Explain the difference between supervised and unsupervised learning.",
    "What is the transformer architecture and why is it so popular?",
    "What is gradient descent and how does it work?",
]

baseline_results, spec_results = benchmark(
    draft_model=draft_model,
    target_model=target_model,
    tokenizer=tokenizer,
    prompts=prompts,
    max_new_tokens=200,
    gamma=5,
    method="standard",
    verbose=True,
)
```

## Benchmark Results

Measured on Qwen3-0.6B (draft) and Qwen3-4B (target), 4 prompts, 200 tokens each, Gamma = 5. 
Hardware: NVIDIA GeForce RTX 3070 (8 GB VRAM).

| Method | Speed (tok/s) | Speedup | Acceptance Rate |
|---|---|---|---|
| Target only (baseline) | 1.9 | 1.0× | — |
| Standard SD | 3.2 | 1.7× | 43% |
| Capped HSD | 3.7 | **1.9×** | 52% |
| Naive HSD | 1.8 | 0.9× | 49% |

**Effect of Gamma (draft length):**

![Speed vs gamma](https://raw.githubusercontent.com/Yijin-Zeng/Sequential-Speculative-Decoding/main/docs/speculative_decoding_speed-1.png)

> **Note:** Naive HSD is slower than the baseline in practice because its resampling step requires multiple additional target model calls, so in practice, it is not recommended.

## Methods

### Standard Speculative Decoding

The draft model proposes Gamma tokens autoregressively. The target model evaluates them all in one forward pass. Tokens are accepted left-to-right with probability min(p/q, 1). The first rejection stops the chain and resamples from a residual distribution. Produces an extra bonus token if all proposed tokens are accepted.

### Naive Hierarchical SD (HSD)

Verifies the entire drafted sequence at once using joint probability ratios. Scans backward to find the longest acceptable prefix, then resamples the rest using branch divergence probabilities. Each resampling step calls both models again, which is why it is slow in practice.

### Capped HSD

A more efficient variant of HSD that caps prefix ratios to avoid the multi-step resampling. Achieves the best speed in our experiments.

## API

```python
# Load models
tokenizer, draft_model, target_model = load_models(
    draft_name="Qwen/Qwen3-0.6B",   # any HF model
    target_name="Qwen/Qwen3-4B",
    device="cuda",                   # or "cpu", "mps"
)

# Generate
result = speculative_generate(
    draft_model, target_model, tokenizer,
    prompt="...",
    max_new_tokens=200,
    gamma=5,                         # draft tokens per round
    method="standard",               # "standard" | "naive_hsd" | "capped_hsd"  
    draft_temperature=0.6,
    target_temperature=0.6,
)
# result keys: text, tokens_generated, tokens_per_sec, acceptance_rate, average_accepted_n

# Benchmark against target-only baseline
baseline_results, spec_results = benchmark(
    draft_model, target_model, tokenizer,
    prompts=[...],
    max_new_tokens=200,
    gamma=5,
    method="standard",
    verbose=True,
)
```

See `examples/` for runnable scripts.

## Project Structure

```
src/seqspecdecod/   # the source code for the package, including generation, sampling, benchmarking, model loading
notebooks/          # toy experiments, including several simple experiments for each method and gamma comparison
examples/           # runnable scripts showing basic usage and method comparison
docs/               # figures generated from the notebooks
```

## References

1. Leviathan, Y., Kalman, M., & Matias, Y. (2023). [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192). *ICML*.
2. Zhou, Y. et al. (2026). [Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding](https://arxiv.org/abs/2601.05724).

## License

MIT
