Metadata-Version: 2.4
Name: modelbump
Version: 0.3.0
Summary: Find the LLM model IDs in your code and config, and warn before providers retire them.
Keywords: llm,model deprecation,model retirement,static analysis,github action,openai,anthropic,gemini
Author: Hammad Hassan
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Dist: httpx>=0.28.1
Requires-Dist: pydantic>=2.13
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: tree-sitter>=0.26
Requires-Dist: tree-sitter-python>=0.25
Requires-Dist: tree-sitter-typescript>=0.23.2
Requires-Dist: typer>=0.27.3
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/imhammad/modelbump
Project-URL: Documentation, https://imhammad.github.io/modelbump/
Project-URL: Repository, https://github.com/imhammad/modelbump
Project-URL: Issues, https://github.com/imhammad/modelbump/issues
Project-URL: Changelog, https://github.com/imhammad/modelbump/blob/main/CHANGELOG.md
Description-Content-Type: text/markdown

# ModelBump

**Dependabot for LLM models.** ModelBump finds the AI models your code depends on and warns you before a provider retires them. Later versions will also test the old model against its replacement, repair prompts that regress, and open the upgrade pull request for you.

[![PyPI](https://img.shields.io/pypi/v/modelbump)](https://pypi.org/project/modelbump/)
[![CI](https://github.com/imhammad/modelbump/actions/workflows/ci.yml/badge.svg)](https://github.com/imhammad/modelbump/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green)](https://github.com/imhammad/modelbump/blob/main/LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230)](https://github.com/astral-sh/ruff)
[![Checked with mypy](https://img.shields.io/badge/mypy-strict-2a6db2)](https://mypy-lang.org/)

> **Status:** `v0.3.0` ships the first stage, **Scan**, as a command-line tool and a GitHub Action, plus two building blocks for the next stages: a model runner (`modelbump ask`) and a statistics engine (`modelbump stats`) that tells a real regression from normal LLM randomness. The other stages are still being built; see the [roadmap](#roadmap).

## The problem

Apps built on hosted LLMs (OpenAI, Anthropic, Google, ...) depend on specific model versions, and providers retire those versions on a schedule. When a model is retired, an app either breaks outright or keeps running while quietly producing worse answers, because prompts tuned for the old model don't transfer cleanly to the new one.

A 2026 study of 22,555 migration commits across 17,703 open-source repositories found that:

- **82%** of migrations happened *after* the old model was already shut down (our own re-check on a larger dataset finds a smaller share, but still more than half),
- model identifiers were **hard-coded in 94%** of migrating apps, and
- only **4%** of migrations came with any evaluation suite.

*Source: H. L. Kim, "When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications", arXiv:2609.31288, 2026.*

## Install and scan

Requires Python 3.12 or newer.

```bash
pip install modelbump
modelbump scan .
```

Or run it once without installing, with [uv](https://docs.astral.sh/uv/):

```bash
uvx modelbump scan .
```

Each line of the report gives the file, line and column, the model ID, its status (`OK`, `RETIRING_SOON`, `DEPRECATED`, `RETIRED` or `UNKNOWN`), the shutdown date with the suggested replacement, and where the ID sits (a call argument, an assignment, a config value, a comment, ...).

Useful options:

| Option | What it does |
|--------|--------------|
| `--fail-within 30` | Exit with code 1 if a model is retired or shuts down within 30 days (for CI) |
| `--skip-comments` | Leave out IDs that only describe a model: comments, docstrings, sentences, price tables |
| `--skip-tests` | Leave out test files and test folders |
| `--exclude PATTERN` | Skip matching paths, e.g. `--exclude '*.ipynb'` (repeat for more) |
| `--format json` / `--format sarif` | Machine-readable reports; SARIF shows up in GitHub code scanning |

Run `modelbump scan --help` for the rest.

## GitHub Action

```yaml
name: ModelBump
on:
  pull_request:
  schedule:
    - cron: "0 6 * * 1"  # every Monday, to catch new retirement notices

permissions:
  contents: read
  security-events: write  # to upload findings to code scanning

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: imhammad/modelbump@v0.1.0
        with:
          fail-within: "30"
          skip-comments: "true"
```

All inputs are listed in the [scanner documentation](https://imhammad.github.io/modelbump/architecture/scanner/).

## How well the scanner works

We ran the scanner on 100 real projects, each at the commit just before it moved off a retired model, and labelled a sample of 250 findings by hand ([evaluation details](https://imhammad.github.io/modelbump/research/scanner-eval/)):

| Setting | Finds the old model's lines (recall) | Findings that are real dependencies (precision) |
|---------|------|------|
| Default | 97.7% | 47.9% |
| `--skip-comments --skip-tests` | 82.0% | 75.2% |

Replayed on those 100 projects, the scanner flagged 96 before they migrated. The default is tuned not to miss anything; the two flags trade some misses for a much quieter report.

## Is the new model worse, or is it just randomness?

LLM answers change from run to run, so one run of each model proves little. Give the statistics engine how often each model passed a test case, and it says whether the difference is real:

```console
$ modelbump stats item 45/50 30/50
Old model: 45/50 passed (90.0%)
New model: 30/50 passed (60.0%)
Difference (new - old): -30.0 points, 90% interval -43.8 to -14.5
Verdict: REGRESSED (the new model is worse on this test case)
```

It also stops asking as soon as the answer is clear, and controls false alarms across a whole suite. On real answers from two local models (Qwen 2.5 against itself), a single-run diff flagged 2.6 of 60 unchanged test cases as worse on every run; the engine's suite methods flagged almost none ([results](https://imhammad.github.io/modelbump/research/stats-real/), [how it works](https://imhammad.github.io/modelbump/architecture/stats/)).

## The plan

| Stage | What it does | Status |
|-------|--------------|--------|
| **Scan** | Static analysis finds every model ID in your code and config, checks it against a registry of provider retirement dates, and fails CI *before* the shutdown date. | Available (`v0.1.0`) |
| **Test** | Builds an app-specific regression suite from your prompts, tests, and (optionally) logs. | Planned |
| **Compare** | Runs old vs. new model with sequential statistical tests, separating real regressions from normal LLM randomness, within a cost budget. | Engine available (`v0.3.0`); runs on your app in a later version |
| **Repair** | Searches for prompt edits that fix regressions without breaking anything that already worked. | Planned |
| **Bump** | Opens a ready-to-review PR with the new model, repaired prompts, the regression suite, and an evidence report. | Planned |

## Development

Requires [uv](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/imhammad/modelbump.git
cd modelbump
uv sync
uv run modelbump --version
uv run pytest
```

## Roadmap

- [x] **Phase 0:** Foundations: packaging, CI, pre-commit, branch protection
- [x] **Phase 1:** Migration dataset + model retirement registry
- [x] **Phase 2:** Scanner + GitHub Action (`v0.1.0`)
- [x] **Phase 3:** Model runner with caching and cost budgets (`v0.2.0`)
- [x] **Phase 4:** Statistical comparison engine (`v0.3.0`)
- [ ] **Phase 5:** Regression suite synthesis + LLM-judge calibration
- [ ] **Phase 6:** Prompt repair
- [ ] **Phase 7:** End-to-end `modelbump bump` + `v1.0.0`

## Research

ModelBump is also a research project: an empirical study of how open-source LLM apps migrate off retired models (paper in preparation), and later a paper on the tool itself. Research notes, protocols and results are in the [research section of the docs](https://imhammad.github.io/modelbump/research/).

## License

[Apache License 2.0](https://github.com/imhammad/modelbump/blob/main/LICENSE)
