Metadata-Version: 2.4
Name: slmify
Version: 0.2.0.dev0
Summary: Point it at your LLM logs. See exactly where you're overpaying — fine-tune a small model, cache, or downgrade.
Project-URL: Homepage, https://github.com/dannyd08/slmify
Project-URL: Repository, https://github.com/dannyd08/slmify
Project-URL: Issues, https://github.com/dannyd08/slmify/issues
Project-URL: Changelog, https://github.com/dannyd08/slmify/blob/main/CHANGELOG.md
Author-email: Daniel D'Souza <223949492+dannyd08@users.noreply.github.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: clustering,cost,fine-tuning,llm,rightsizing,slm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: drain3>=0.9.11
Requires-Dist: httpx>=0.27
Requires-Dist: model2vec>=0.3
Requires-Dist: numpy>=1.26
Requires-Dist: polars>=1.0
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: scikit-learn>=1.3
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: peft>=0.11; extra == 'all'
Requires-Dist: sentence-transformers>=3.0; extra == 'all'
Requires-Dist: trl>=0.9; extra == 'all'
Requires-Dist: unsloth; extra == 'all'
Provides-Extra: st
Requires-Dist: sentence-transformers>=3.0; extra == 'st'
Provides-Extra: train
Requires-Dist: peft>=0.11; extra == 'train'
Requires-Dist: trl>=0.9; extra == 'train'
Requires-Dist: unsloth; extra == 'train'
Description-Content-Type: text/markdown

<div align="center">

### <a href="https://small-lm.com"><img src="docs/assets/smallm-icon.png" width="34" align="absmiddle" alt="SmaLLM icon" /></a>&nbsp;SmaLLM

# slmify

**Find the 40-70% of your LLM traffic a small model could handle.<br/>
Prove it before you train anything. Your data never leaves your machine.**

<sub>Open-sourced and maintained by the team at <a href="https://small-lm.com"><b>SmaLLM</b></a></sub>

<br/>

[![CI](https://github.com/dannyd08/slmify/actions/workflows/ci.yml/badge.svg)](https://github.com/dannyd08/slmify/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/slmify?color=blue)](https://pypi.org/project/slmify/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Telemetry](https://img.shields.io/badge/telemetry-zero-brightgreen)](#-privacy-posture)
[![Data residency](https://img.shields.io/badge/your_data_leaves_your_machine-never-brightgreen)](#-privacy-posture)

<br/>

[**Quickstart**](#-sixty-seconds-to-your-number) · [**How it works**](#-how-it-works) · [**Benchmarks**](#-benchmarks) · [**Roadmap**](#-roadmap)

<br/>

<!-- Regenerate with vhs: see docs/assets/hero.tape -->
<img src="docs/assets/hero.gif" alt="slmify analyzing production LLM logs and printing a ranked savings report" width="720" />

</div>

---

## Why this exists

Somewhere in your traffic, a frontier model is formatting JSON, filling the same template for the ten-thousandth time, and answering the same twelve questions. You are paying reasoning-model prices for work a 4B model could do faster, cheaper, and on hardware you own.

NVIDIA's research estimates that [40-70% of LLM calls in agentic systems are SLM-replaceable](https://arxiv.org/abs/2506.02153). They sketched the conversion pipeline, then left the measurement as an exercise for the reader.

**slmify measures that number on your traffic.** It ingests your logs, groups them into workloads, scores each one for SLM-fit, prices the opportunity in dollars with break-even math, recommends the cheapest safe fix, and exports training-ready datasets for whichever fine-tuning stack you already use.

**Who's "we"?** We're the team behind [SmaLLM](https://small-lm.com), a platform that runs this whole loop end to end: observe production traffic, fine-tune small models automatically, evaluate them, and canary-route them behind an OpenAI-compatible gateway. The analysis layer answers the question every team asks first ("what's actually worth fine-tuning, and what would it save?"), so we carved it out of the platform and open-sourced it. slmify is fully standalone, no strings attached: it works whether or not you ever touch the rest of our stack.

## ⚡ Sixty seconds to your number

If you use **Claude Code**, **Codex CLI**, or **Gemini CLI**, a corpus of real agent traffic is already sitting on your laptop. slmify turns it into a verdict with zero setup:

```bash
uvx slmify        # no install, no config, no API key, no network
```

No agent traces? Run the bundled demo corpus:

```bash
uvx slmify demo   # full report in ~30 seconds, works on a plane
```

Example output (illustrative):

```text
  WORKLOADS   14 discovered · 31 days · 412,806 calls · $9,110/mo observed spend

  BAND  CLUSTER                       CALLS/MO   SPEND/MO   ACTION           EST. SAVINGS/MO
  A     c07 invoice-json-extraction    41,220     $2,410    FINE_TUNE_SLM    $1,700 - $2,100
  A     c03 support-macro-rewrite      88,410     $1,930    FINE_TUNE_SLM    $1,310 - $1,760
  B     c11 order-status-lookup        64,204       $880    EXACT_CACHE        $610 -   $780
  C     c02 code-review-comments        9,410     $1,240    DOWNGRADE_TRIAL    $340 -   $560
  ────────────────────────────────────────────────────────────────────────────────────────
  KEEP  7 clusters                    209,562     $4,980    KEEP             $0  (and that's fine)
```

Yes, it will tell you **KEEP**. A tool that recommends fine-tuning everything is a tool you can't trust.

## 🔍 From verdict to fine-tuned model in five commands

```bash
# 1. Analyze production logs (format auto-detected; CSV needs a --map)
uvx slmify analyze ./logs.jsonl --html report.html
uvx slmify analyze ./logs.csv --map "prompt=question,response=answer,model=model_name"

# 2. Prove feasibility BEFORE spending anything: zero training, k-shot,
#    any OpenAI-compatible endpoint (Ollama shown)
slmify probe --clusters top3 --endpoint http://localhost:11434/v1 --model qwen3:4b-instruct

# 3. Export train/eval JSONL + ready-to-run config stubs
slmify export c07        # Unsloth · Axolotl · TRL · OpenAI FT · vLLM-router

# 4. Or train locally on the same recipe ([train] extra, CUDA/Unsloth)
slmify train c07

# 5. Verify the fine-tune against the untainted holdout
slmify eval c07 --endpoint http://localhost:8000/v1 --model tuned --baseline-model base
```

## 📊 What the report gives you

<div align="center">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/assets/report-dark.png">
  <img src="docs/assets/report-light.png" alt="slmify HTML report: ranked workloads with SLM-fit bands, savings ranges, and recommended actions" width="720">
</picture>
</div>

For every discovered workload:

- **SLM-fit band (A "Prime" → D "Poor fit")** from 10 published, documented metrics: output contraction, schema stability, task-type priors, template mass, semantic cohesion, reasoning depth, and more. Top-3 evidence bullets and a confidence tag on every band. Raw numbers live in `analysis.json`, not the headline.
- **Probe evidence on demand.** `slmify probe` runs a candidate SLM against 30 held-out samples per cluster (k-shot, zero training) through any OpenAI-compatible endpoint and adjusts the band ±1 step. Graders run cheapest-first (exact → JSON-schema → embedding similarity); an LLM judge only runs if you supply a judge endpoint, and flat judge scores automatically switch to ranking mode.
- **Measured spend and savings as ranges**, `{low, point, high}` with every assumption printed, priced from a vendored [LiteLLM pricing snapshot](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json), monthly volumes extrapolated with a bootstrap CI over your observed days. A 3-day log honestly yields wide bars.
- **Dedicated-GPU break-even math.** The report tells you the sustained calls/hour where a dedicated L4/A10G/A100 beats your current spend, and says "use serverless" when you're below it.
- **An action stack**, decided by auditable ordered rules you can override with `--policies`: `FINE_TUNE_SLM`, `DOWNGRADE_TRIAL`, `EXACT_CACHE`, `SEMANTIC_CACHE`, `PROMPT_CACHE / TRIM`, `FIX_QUALITY_FIRST` (with a named remedy), or an honest `KEEP`.
- **Cache economics as floor / realistic / ceiling.** Exact-duplicate rate is the zero-risk floor; semantic near-duplicate mass is the ceiling.
- **Episode reconstruction.** Multi-turn conversations in flat logs are rebuilt with conservative prefix matching, so volumes, duplicate rates, and dataset exports never double-count growing message arrays.

`slmify export <cluster>` writes `train.jsonl` / `eval.jsonl` in OpenAI chat format, plus config stubs. The eval set is a stratified holdout **never** included in training data, so your fine-tune gets verified on untainted samples.

## ⚙️ How it works

```mermaid
flowchart LR
    A["📥 Any logs<br/>Claude Code · Codex · Gemini CLI<br/>Langfuse · OpenAI JSONL · CSV"] --> B["🧵 Episode<br/>reconstruction"]
    B --> C["🧩 Template mining<br/>(Drain3)"]
    C --> D["🗺️ Semantic clustering<br/>(local embeddings)"]
    D --> E["🎯 10-metric<br/>SLM-fit scoring"]
    E --> F["💰 Savings + break-even<br/>+ action stack"]
    F --> G["🔬 Probe<br/>(zero-training proof)"]
    G --> H["🚀 Export / Train / Eval<br/>Unsloth · Axolotl · TRL · OpenAI"]
```

Workloads are grouped on three levels (workflow step → prompt template → semantic cluster), so "the same call with different variables" and "different calls that mean the same thing" both land where they belong.

## 🧭 Built different

- 🔒 **Local-first, zero telemetry, zero required API keys.** Your prompts never leave your machine. The default embedder is a 30 MB static model; the tool runs offline.
- 🧮 **No false precision, ever.** Bands not decimals, ranges not points, every assumption printed. Expect frequent, honest **KEEP** recommendations.
- 🔬 **Evidence over vibes.** Every score ships with the exact metrics behind it. Every formula is documented and user-overridable via `--policies`.
- 🇨🇭 **Neutral.** Reads every vendor's logs, exports to every trainer. No login, no gates, no nags.

## 🔌 Works with your stack

| Reads (inputs) | Feeds (outputs) |
|---|---|
| Claude Code traces (`~/.claude/projects`) | Unsloth config + dataset |
| Codex CLI sessions (`~/.codex/sessions`) | Axolotl config + dataset |
| Gemini CLI sessions (`~/.gemini/tmp`) | TRL training script stub |
| Langfuse observation exports | OpenAI fine-tuning JSONL |
| OpenAI-style JSONL (3 dialects) | vLLM router config stub |
| Generic JSONL/CSV via dotted-path `--map` | HTML + JSON reports |
| LiteLLM (drop-in `SlmifyLogger` callback) | |

Adapters are pluggable via the `slmify.adapters` entry-point group. OTel GenAI lands in v0.3.

## 🥊 How slmify compares

| | slmify | OpenAI Distillation | Braintrust / LangSmith | Kiln |
|---|---|---|---|---|
| Finds *what* to fix in your traffic | ✅ scored workloads | ✗ | clusters, no scoring/savings | task-first |
| Cross-vendor (reads any logs) | ✅ | OpenAI only | their platform | n/a |
| Runs locally, data never leaves | ✅ | ✗ | ✗ | ✅ |
| Proves SLM feasibility pre-training | ✅ probe | ✗ | ✗ | ✗ |
| $ savings with break-even math | ✅ ranges | ✗ | ✗ | ✗ |
| Exports to any trainer | ✅ | ✗ | partial | ✅ |

*As of 2026-07. These tools solve adjacent problems; several pair well with slmify rather than replacing it.*

## 📈 The market is moving

> "I believe the future of agentic AI is many, many, many small language models."
>
> **Andy Markus**, Chief Data Officer, AT&T (Feb 2026)

AT&T rebuilt its stack around fine-tuned SLMs at 27B tokens/day and publicly reported cost reductions up to 90%. NVIDIA's position paper ([arXiv:2506.02153](https://arxiv.org/abs/2506.02153)) argues SLMs are the future of agentic AI and that most agent calls don't need a frontier model.

They built the measurement and conversion by hand, with ML teams. slmify gives you the measurement in one command.

## 🧪 Benchmarks

`benchmarks/` holds reproducible runs:

- **Ground-truth validation** of the whole grouping pipeline on the bundled labeled corpus: **purity 1.00, V-measure 0.94**.
- **One-command public-data benchmark** (WildChat-1M / LMSYS-Chat-1M) measuring the SLM-able share of real traffic against NVIDIA's 40-70% agentic estimate.

## 📦 Install

```bash
uv tool install slmify        # or: pipx install slmify / pip install slmify
```

Core install is light: polars, numpy, scikit-learn, model2vec, drain3, pydantic, typer, rich, httpx, pyyaml. **No torch, no database, no compiled clustering libs.** Extras: `[st]` (sentence-transformers embedders), `[train]` (local LoRA fine-tuning).

Offline degradation ladder: `demo` needs zero network → the first real analysis downloads the 30 MB default embedder once (then cached) → fully offline, use `--embedder hash` (TF-IDF, already installed). Degraded clusters, still useful, never a dead end.

## 🔐 Privacy posture

Report exemplars are masked by default (`<EMAIL>`, `<NUM>`, `<ID>`, …) via the same template miner that names your clusters; `--show-raw` disables. Dataset exports contain raw text (training needs real data) and say so loudly. There is no telemetry of any kind.

## 🗺️ Roadmap

- [x] **v0.1** — analyzer, HTML report, dataset exports
- [x] **v0.2** — zero-training probe, local LoRA trainer, eval harness, LiteLLM callback
- [ ] **v0.3** — `--watch`/CI diff mode · OTel GenAI adapter · in-process adapter eval · open calibration study on public data · plugin authoring docs · `--backend mlx` (Apple Silicon training)

Have an opinion on what comes next? [Open a discussion](../../discussions).

## ⭐ Support the project

If slmify found money in your logs, or told you honestly that there was none to find, **star the repo**. It's how other engineers discover it, and it directly shapes how much time we can spend on v0.3.

<!-- Once past ~500 stars, add a star-history chart here:
<a href="https://star-history.com/#dannyd08/slmify&Date">
  <img src="https://api.star-history.com/svg?repos=dannyd08/slmify&type=Date" width="600">
</a>
-->

## 🤝 Contributing

Bug reports, adapter plugins (`slmify.adapters` entry points), and benchmark submissions are all welcome. Check [`good first issue`](../../labels/good%20first%20issue) to get started, and read [CONTRIBUTING.md](CONTRIBUTING.md) for the dev setup.

## ❓ FAQ

<details>
<summary><b>Does my data ever leave my machine?</b></summary>
<br/>
No. There is no telemetry, no login, and no required API key. The only network calls slmify ever makes are ones you explicitly configure: downloading the default embedder once, <code>slmify pricing --update</code>, and probe/eval calls to endpoints you point it at (which can be localhost).
</details>

<details>
<summary><b>Do I need a GPU?</b></summary>
<br/>
Not for analysis, probing, or exports. Only <code>slmify train</code> (the optional <code>[train]</code> extra) needs CUDA. Everything else runs on a laptop.
</details>

<details>
<summary><b>Why does it keep telling me KEEP?</b></summary>
<br/>
Because most traffic genuinely isn't worth fine-tuning, and pretending otherwise would make the A-band verdicts worthless. KEEP means the math didn't clear the bar: volume too low, cohesion too weak, or savings below break-even. That's the tool working.
</details>

<details>
<summary><b>How accurate are the savings numbers?</b></summary>
<br/>
They're ranges, not promises. Prices come from a vendored LiteLLM pricing snapshot, volumes are extrapolated with a bootstrap CI over your observed days, and every assumption is printed in the report. Short logs produce wide bars on purpose.
</details>

<details>
<summary><b>Can I change how workloads are scored?</b></summary>
<br/>
Yes. Every formula and threshold is documented, and <code>--policies</code> accepts a YAML override for the scoring weights and the action-stack rules.
</details>

<details>
<summary><b>Is this just a funnel for a paid product?</b></summary>
<br/>
It's a real standalone tool first. We (the <a href="https://small-lm.com">SmaLLM</a> team) sell a platform that automates what slmify recommends: training, serving, and canary-routing the small models. slmify never requires it, never phones home, and exports to every trainer, including ones that compete with us. If you never visit our site, slmify still does its entire job.
</details>

## 📄 License

Apache-2.0.

<div align="center">
<sub><b>slmify</b> is built and maintained by the team behind <a href="https://small-lm.com"><b>SmaLLM</b></a>,<br/>
the platform that closes the loop slmify opens: automated fine-tuning, evaluation, and canary routing of the workloads it finds.<br/><br/>
slmify stays free, local, and neutral either way. Built by engineers who got tired of guessing.</sub>
</div>

<!-- Pre-flip manual step: upload docs/assets/social-preview.png in
     Repo settings → Social preview. It's what renders when the repo
     link is shared on X/HN/Slack. Also enable Discussions in repo
     settings and create the "good first issue" label. -->
