Metadata-Version: 2.5
Name: inferencekit-cli
Version: 0.1.0
Summary: Make your model serve faster on your GPU, and prove it. Skills for Claude Code, Codex and OpenCode, plus a CLI and MCP server: your agent picks and tunes inference engines, can write GPU kernels, and backs every number with client-timed benchmarks, anti-cheat gates and quality checks.
Project-URL: Homepage, https://github.com/kittenml/inferencekit
Project-URL: Repository, https://github.com/kittenml/inferencekit
Project-URL: Issues, https://github.com/kittenml/inferencekit/issues
Author: Stellon Labs
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: benchmarking,capacity,correctness,cuda,gpu,inference,llama.cpp,llm,mcp,nsight-compute,nsight-systems,nvidia,profiling,quantization,roofline,serving,sglang,tensorrt-llm,vllm
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Requires-Dist: jsonschema>=4.20
Requires-Dist: pyyaml>=6.0
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2.3; extra == 'mcp'
Description-Content-Type: text/markdown

<div align="center">

# inferencekit

**Turn your coding agent into an inference engineer.**

inferencekit made Gemma 4 run 1.33× faster than tuned vLLM. Your model is next.

[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![PyPI: inferencekit-cli](https://img.shields.io/pypi/v/inferencekit-cli.svg?label=inferencekit-cli)](https://pypi.org/project/inferencekit-cli/)
[![Claude Code](https://img.shields.io/badge/Claude%20Code-plugin-d97757.svg)](#install)
[![Codex](https://img.shields.io/badge/Codex-plugin-111111.svg)](#install)
[![OpenCode](https://img.shields.io/badge/OpenCode-plugin-4a7bd0.svg)](#install)

[Install](#install) · [How it works](#how-it-works) · [Results](#results) · [Features](#features) · [Engines](https://github.com/kittenml/inferencekit-engines) · [Roadmap](#roadmap) · [Contributing](CONTRIBUTING.md)

![Bar chart of speedups against engine baselines on the same hardware, including tuned vLLM and MLX-Audio: Gemma 4 26B-A4B on RTX 5090, 1.33× faster than tuned vLLM; LTX-2.5 on RTX PRO 6000, 1.25× faster than ComfyUI; Qwen3-8B on RTX 5090, 1.25× faster than default vLLM; Drama3 on Apple M4, 1.11× faster than tuned MLX-Audio; MOSS-Image-SFT on RX 9070 XT, 2.46× with quality secondary (FP8, 24 vs 50 steps; paired 95% CI 2.453–2.465×, lower bound 2.453×)](assets/hero-speedups.png)

</div>

inferencekit is a plugin for Claude Code, Codex and OpenCode, plus a command-line tool (CLI). Give your agent a model, hardware and a goal. It finds, benchmarks and tunes supported inference engines for your workload. You can stop with the best tested configuration or go further with custom optimizations, including GPU kernels.

## How it works

inferencekit gives your agent the playbook and the tools of a senior inference engineer. On every run it works the problem the same way.

1. **Understand the task.** The agent first asks about your goal, deadline, budget and how much the results may change, reusing answers you already gave. It identifies the model, hardware and real inputs, then estimates memory, compute and launch costs per request. Text, image, audio and video use their own metrics and quality checks.
2. **Find and tune an engine.** It researches current implementations, then tunes and benchmarks supported engines (vLLM, SGLang, llama.cpp, TensorRT-LLM and others) for your workload on your hardware. The fastest tested configuration that meets your quality requirements can be the result, or the tuned baseline for further work.
3. **Find the bottleneck.** It selects the profiler for your hardware: Nsight on NVIDIA, ROCm tools on AMD, or Instruments and Metal on Apple. It measures the critical path and investigates memory, compute and launch costs, checking current vendor docs when support is uncertain.
4. **Investigate and implement, if requested.** Search current code, papers, compiler docs and engineering talks; inspect dispatch, IR and assembly when useful. Choose graph execution, fusion, matrix kernels, packing, state reuse or other techniques from measured bottlenecks. Independent implementations can run in parallel; integration and shared-device measurements stay isolated.
5. **Prove every step.** Every change gets a fast correctness check and a timed screen; milestones get the full proof: interleaved rounds against the tuned baseline, the anti-cheat gates and quality against a reference model. Changes that miss the agreed performance or quality requirements are reverted.
6. **Ship it.** You get the fastest proven configuration, its launch command, and a report with the commands that reproduce every number.

## When it asks you

The agent works on its own and stops only when it needs you:

- **Missing details.** If you didn't say which model, GPU or workload, it asks before it starts.
- **Your preferences.** Choose exactly the same results, small differences with the same quality, or speed with an agreed quality tradeoff. Set your goal, spending, deadline and check-ins; it reuses answers you already gave.
- **Model changes.** Permission to simplify the model is separate from permission to use stronger compression. Plans are optional: ask to see one if you want it.
- **The budget cap.** When the run reaches the cap you set, it stops and asks.
- **Things only you can give.** For example, permission to read the GPU's profiler counters, or a Hugging Face token for a gated model.

Everything else runs without asking. For a hands-off run, approve a budget with a cap up front (`non_interactive: true` and `budget.approved: true` in the intake file); the agent then writes its questions and the choices it made to `QUESTIONS.md` instead of waiting for you.

## Results

**Pushing speed on AMD:** with quality secondary, MOSS-Image-SFT reached **2.46×** faster image generation than the tested tuned vLLM-Omni baseline on a 16 GB RX 9070 XT.‡

Single-user decode for the language models; one clip at a time for video; single-user speech generation for Drama3.

| Engine | GPU | Speedup | Agent | Agent time | Tokens | Cost |
|---|---|---|---|---|---|---|
| [Gemma 4 26B-A4B (NVFP4)](https://github.com/kittenml/inferencekit-engines/tree/main/gemma-4-26b-a4b-nvfp4_rtx-5090) | RTX 5090 | **1.33×** vs tuned vLLM | Claude Opus 5.5 (high) | 19 h | 630M | $199 |
| [LTX-2.5 text-to-video](https://github.com/kittenml/inferencekit-engines/tree/main/ltx-2.5_rtx-pro-6000) | RTX PRO 6000 | **1.25×** vs ComfyUI | Claude Opus 5.5 (high) | 3.6 h | 47M | $18 |
| [Qwen3-8B (FP8)](https://github.com/kittenml/inferencekit-engines/tree/main/qwen3-8b-fp8_rtx-5090)* | RTX 5090 | **1.25×** vs vLLM | Claude Opus 5.5 (xhigh) | about 10 h | about 240M | about $125 |
| [Drama3 speech](https://github.com/haaziq-sl/inferencekit-engines/tree/main/drama-3_apple-m4-16gb/profiles/main5-tower5)† | Apple M4 (16 GB) | **1.112×** vs tuned MLX-Audio | GPT-6.1 Sol (high) | ~70 h elapsed | ~2.07B | ~$475 standard-rate equivalent |
| [MOSS-Image-SFT (FP8)](https://github.com/haaziq-sl/inferencekit-engines/tree/main/moss-image-sft_rx-9070-xt-fp8)‡ | RX 9070 XT (16 GB) | **2.46×** vs tuned vLLM-Omni | GPT-6.1 Terra (high) | ~19.6 h elapsed | ~1.80B | ~$206 standard-rate equivalent |

<sub>* Against vLLM with its default settings; neither side uses speculative decoding. † Drama3 used identical quantized weights and matched PCM audio, codes, chunks and completion in 144 fresh output pairs. Four paired timing rounds gave a 95% CI of 1.059×–1.156×. [Comparison details](https://github.com/haaziq-sl/inferencekit-engines/tree/main/drama-3_apple-m4-16gb/profiles/main5-tower5).</sub>

<sub>‡ Lossy speed-first experiment: FP8/24 steps vs BF16/50 steps; paired 95% CI **2.453–2.465×**, conservative lower bound **2.453×**. [Outputs, timing and accounting](https://github.com/haaziq-sl/inferencekit-engines/tree/main/moss-image-sft_rx-9070-xt-fp8/).</sub>

Time and tokens include setup, benchmarking, optimization, validation and reporting. Costs are API list-price estimates; most tokens are cached conversation reads. Drama3’s estimates cover direct agent work ([accounting](https://github.com/haaziq-sl/inferencekit-engines/blob/main/drama-3_apple-m4-16gb/ACCOUNTING_V65.json)). How each engine was built and measured: [inferencekit-engines](https://github.com/kittenml/inferencekit-engines).

## Speed you can trust

Fast is easy. Fast and still correct is the hard part, and it's the part agents skip.

Agents are good at finding speed. They're also good at finding fake speed: a cached answer, a shorter reply, a different video. inferencekit keeps only the speed that's real.

Without these checks, one agent thought its video engine was 2% faster, but it was rendering a completely different video. Another was sure it had a 1.12× speedup; measured again on a clean machine, its engine was 5% slower than where it started.

A speedup counts only when both engines pass every check that applies.

| A way to game a benchmark | What catches it |
|---|---|
| Answer repeated prompts from a cache | `caching`: a replayed prompt may not be over 10% faster; a prompt changed by one character may not get the old answer |
| Report more tokens than were made | `token_count`: the reported count must match the streamed tokens and the re-tokenized text |
| Stop early or cut answers short | `output_length`: with end-of-text ignored, every answer must be exactly the requested length |
| Drop rare tokens from the vocabulary | `rare_tokens`: random CJK text, emoji, long numbers and code names must come back exactly |
| Ignore sampling settings or fake probabilities | `hidden_probes`: temperature must change answers; probabilities must be valid |
| Use another chat template so prompts get shorter | Render check: every prompt must become the same number of tokens as on the baseline |
| Speculative decoding that changes answers | `spec_exact`: greedy answers must match the same engine with speculation off |
| Lower quality for speed | Quality vs a reference, GSM8K, per-step video check |
| Numbers that only hold on the agent's machine | Every run writes `manifest.json` with each engine's install and launch commands, so anyone can re-run it on a clean machine, on prompts sealed with `inferencekit prompts seal` that the agent never saw |

Each check ends PASS, FAIL, UNPROVEN (not enough evidence) or NOT RUN. A missing check is never a pass.

## Features

- **9 skills, 4 commands, 5 read-only subagents and an MCP server**, the same in every supported agent.
- **Speed-of-light ceilings:** measured memory bandwidth (`ceiling bw`) and exact bytes per token read from the checkpoint's safetensors headers (`roofline`), including NVFP4, FP8 and mixture-of-experts models.
- **Engine triage and tuning:** vLLM, SGLang, llama.cpp, TensorRT-LLM, ComfyUI, Diffusers and others. Only settings that the engine's own logs show in effect are counted.
- **Kernel work:** profiling routed by hardware, CUDA/HIP, Triton and Metal kernels, and torch.compile. The included engine skeleton targets NVIDIA CUDA.
- **One timer for every engine:** interleaved rounds in a balanced order, medians with 95% confidence intervals.
- **Anti-cheat gates:** caching, token counts, output length, rare tokens, sampling probes, prompt rendering, speculative-decoding exactness.
- **Quality checks:** reference quality and GSM8K for language models; latent and final-clip checks for video. Images and speech use client timing and structural checks plus an external semantic quality protocol; see [media benchmarks](docs/media-benchmarks.md).
- **Budget control:** spending authorized up front, with caps in hardware hours, dollars and elapsed time. No repeated plan approval inside the agreed scope.
- **Reproducible output:** a ranked table labelled PASS, FAIL, UNPROVEN or NOT RUN, a launch command, a review page and the commands that reproduce every number.

### Tools behind each step

| Step | Tools |
|---|---|
| Size the problem | `intake`, `doctor`, `ceiling bw`, `roofline` |
| Start from the best engine | `bench rounds`, `gates run`, `quality` |
| Find the bottleneck | `doctor` routes profiling; Nsight commands on NVIDIA, vendor tools on AMD/Apple |
| Pull the right levers, prove every step | `correctness`, `bench rounds`, `gates run`, `quality` |
| Ship it | `ledger`, `review`, `report` |

## Install

Requirements: [uv](https://docs.astral.sh/uv/) and Python 3.10 or newer. Install the engine runtime for your hardware. NVIDIA capture commands require CUDA and Nsight; AMD and Apple profiling uses vendor tools with separate evidence files. See [supported hardware](#supported-hardware-and-agents).

### Claude Code

```bash
claude plugin marketplace add kittenml/inferencekit
claude plugin install inferencekit@inferencekit
```

### Codex

```bash
codex plugin marketplace add kittenml/inferencekit
codex plugin add inferencekit@inferencekit
```

### OpenCode

```bash
git clone --depth 1 https://github.com/kittenml/inferencekit ~/.local/share/inferencekit
python3 ~/.local/share/inferencekit/plugin/opencode/install.py
```

| Agent | Commands | Check the install |
|---|---|---|
| Claude Code | `/inferencekit:optimize-inference`, `/inferencekit:bench`, `/inferencekit:profile`, `/inferencekit:perf-report` | `claude plugin list`; `/mcp` lists the inferencekit server |
| Codex | same stages; subagent files are followed one at a time where Codex has no subagents | `codex plugin list`; `codex mcp list` |
| OpenCode | `/optimize-inference`, `/bench`, `/profile`, `/perf-report`, and each skill as `/<skill-name>` | `opencode mcp list` |

Updating, removing, per-project installs and OpenCode's options: [docs/install.md](docs/install.md).

## Try it

```text
/inferencekit:optimize-inference Qwen/Qwen3-8B-FP8 on 1x RTX 5090, fastest single-user decode, by 6h
```

To tune existing engines without custom kernels:

```text
/inferencekit:optimize-inference Tune vLLM and SGLang for my model and GPU using their built-in settings only; do not write custom kernels.
```

In OpenCode the command is `/optimize-inference`. It asks only for missing preferences and spending authorization; a plan is optional. For a repeatable setup, start from an intake file: `inferencekit intake template --out intake.yaml`, then `/inferencekit:optimize-inference intake.yaml`. For unattended runs, set `non_interactive: true` and `budget.approved: true` with a cap.

More things to ask for: [docs/examples.md](docs/examples.md).

## CLI and MCP server

```bash
uv tool install "inferencekit-cli==0.1.0"
inferencekit doctor                                         # GPU, driver, CUDA, profiler access
inferencekit gates run --base-url http://127.0.0.1:8000     # anti-cheat gates on any OpenAI-compatible server
uvx --from 'inferencekit-cli[mcp]==0.1.0' inferencekit mcp serve   # MCP server (stdio) for any MCP client
```

Every CLI command is also an MCP tool with the same arguments. Text benchmarks, gates and quality checks talk HTTP to OpenAI-compatible servers. Image and streamed speech measurements use the [media adapter contracts](docs/media-benchmarks.md).

## Supported hardware and agents

| | Status | Run end to end |
|---|---|---|
| NVIDIA GPUs (CUDA) | Full tool set | Yes: RTX 5090, RTX PRO 6000 |
| AMD ROCm GPUs, Apple silicon | Hardware discovery and vendor profiling guidance; HTTP benchmarks; vendor bandwidth/captures collected externally | RX 9070 XT and Apple M4 case studies above |
| x86 CPUs | Skills and HTTP benchmarks; platform profiling collected externally | No |
| ARM CPUs | Not covered | No |
| Language models | Benchmarks, gates, GSM8K, quality vs a reference | Yes |
| Text-to-video | Per-clip timing, per-step check, final-clip PSNR | Yes |
| Text-to-image | Client-timed decoded images and structural checks; external quality proof required | No |
| Speech models | Client-timed streamed PCM, first-signal latency, RTF and underruns; external quality proof required | No |
| Claude Code | Supported | Yes |
| Codex, OpenCode | Install paths and CI install checks | No |

Profiles and logs can contain environment variables, host names and paths: scan them before you share them.

## Roadmap

- **More hardware:** H100, H200, B200, RTX 4090, Intel GPUs and Jetson, with the hardware tools (`doctor`, `ceiling bw`, profiling) for each.
- **More models and modalities:** speech (TTS and STT), image, more video models.
- **Multi-user serving:** published results for many users at once (load sweeps), not just one user at a time.
- **OpenCode end to end.**

## Contributing

We welcome bug reports, engine settings and new skills. See [CONTRIBUTING.md](CONTRIBUTING.md). Develop with `uv sync --all-extras`, then `uv run --all-extras pytest -q` (no GPU needed). Report security problems privately: [SECURITY.md](SECURITY.md).

## License

Apache-2.0: [LICENSE](LICENSE) and [NOTICE](NOTICE). Copyright 2026 Stellon Labs. inferencekit is an independent project, not affiliated with NVIDIA, any engine project or any model publisher. Product and model names belong to their owners. Every result holds for the GPU, driver, engine version, model and workload it was measured on; change any of them and the numbers can change. The software comes with no warranty, as the Apache-2.0 licence states, and you are responsible for the cost of any GPU time you use with it.
