Metadata-Version: 2.4
Name: modelbump
Version: 0.1.0
Summary: Find the LLM model IDs in your code and config, and warn before providers retire them.
Keywords: llm,model deprecation,model retirement,static analysis,github action,openai,anthropic,gemini
Author: Hammad Hassan
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Dist: pydantic>=2.13
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: tree-sitter>=0.26
Requires-Dist: tree-sitter-python>=0.25
Requires-Dist: tree-sitter-typescript>=0.23.2
Requires-Dist: typer>=0.27.3
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/imhammad/modelbump
Project-URL: Documentation, https://imhammad.github.io/modelbump/
Project-URL: Repository, https://github.com/imhammad/modelbump
Project-URL: Issues, https://github.com/imhammad/modelbump/issues
Project-URL: Changelog, https://github.com/imhammad/modelbump/blob/main/CHANGELOG.md
Description-Content-Type: text/markdown

# ModelBump

**Dependabot for LLM models.** ModelBump finds the AI models your code depends on and warns you before a provider retires them. Later versions will also test the old model against its replacement, repair prompts that regress, and open the upgrade pull request for you.

[![PyPI](https://img.shields.io/pypi/v/modelbump)](https://pypi.org/project/modelbump/)
[![CI](https://github.com/imhammad/modelbump/actions/workflows/ci.yml/badge.svg)](https://github.com/imhammad/modelbump/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green)](https://github.com/imhammad/modelbump/blob/main/LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230)](https://github.com/astral-sh/ruff)
[![Checked with mypy](https://img.shields.io/badge/mypy-strict-2a6db2)](https://mypy-lang.org/)

> **Status:** `v0.1.0` ships the first stage, **Scan**, as a command-line tool and a GitHub Action. The other stages are still being built; see the [roadmap](#roadmap).

## The problem

Apps built on hosted LLMs (OpenAI, Anthropic, Google, ...) depend on specific model versions, and providers retire those versions on a schedule. When a model is retired, an app either breaks outright or keeps running while quietly producing worse answers, because prompts tuned for the old model don't transfer cleanly to the new one.

A 2026 study of 22,555 migration commits across 17,703 open-source repositories found that:

- **82%** of migrations happened *after* the old model was already shut down (our own re-check on a larger dataset finds a smaller share, but still more than half),
- model identifiers were **hard-coded in 94%** of migrating apps, and
- only **4%** of migrations came with any evaluation suite.

*Source: H. L. Kim, "When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications", arXiv:2609.31288, 2026.*

## Install and scan

Requires Python 3.12 or newer.

```bash
pip install modelbump
modelbump scan .
```

Or run it once without installing, with [uv](https://docs.astral.sh/uv/):

```bash
uvx modelbump scan .
```

Each line of the report gives the file, line and column, the model ID, its status (`OK`, `RETIRING_SOON`, `DEPRECATED`, `RETIRED` or `UNKNOWN`), the shutdown date with the suggested replacement, and where the ID sits (a call argument, an assignment, a config value, a comment, ...).

Useful options:

| Option | What it does |
|--------|--------------|
| `--fail-within 30` | Exit with code 1 if a model is retired or shuts down within 30 days (for CI) |
| `--skip-comments` | Leave out IDs that only describe a model: comments, docstrings, sentences, price tables |
| `--skip-tests` | Leave out test files and test folders |
| `--exclude PATTERN` | Skip matching paths, e.g. `--exclude '*.ipynb'` (repeat for more) |
| `--format json` / `--format sarif` | Machine-readable reports; SARIF shows up in GitHub code scanning |

Run `modelbump scan --help` for the rest.

## GitHub Action

```yaml
name: ModelBump
on:
  pull_request:
  schedule:
    - cron: "0 6 * * 1"  # every Monday, to catch new retirement notices

permissions:
  contents: read
  security-events: write  # to upload findings to code scanning

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: imhammad/modelbump@v0.1.0
        with:
          fail-within: "30"
          skip-comments: "true"
```

All inputs are listed in the [scanner documentation](https://imhammad.github.io/modelbump/architecture/scanner/).

## How well the scanner works

We ran the scanner on 100 real projects, each at the commit just before it moved off a retired model, and labelled a sample of 250 findings by hand ([evaluation details](https://imhammad.github.io/modelbump/research/scanner-eval/)):

| Setting | Finds the old model's lines (recall) | Findings that are real dependencies (precision) |
|---------|------|------|
| Default | 97.7% | 47.9% |
| `--skip-comments --skip-tests` | 82.0% | 75.2% |

Replayed on those 100 projects, the scanner flagged 96 before they migrated. The default is tuned not to miss anything; the two flags trade some misses for a much quieter report.

## The plan

| Stage | What it does | Status |
|-------|--------------|--------|
| **Scan** | Static analysis finds every model ID in your code and config, checks it against a registry of provider retirement dates, and fails CI *before* the shutdown date. | Available (`v0.1.0`) |
| **Test** | Builds an app-specific regression suite from your prompts, tests, and (optionally) logs. | Planned |
| **Compare** | Runs old vs. new model with sequential statistical tests, separating real regressions from normal LLM randomness, within a cost budget. | Planned |
| **Repair** | Searches for prompt edits that fix regressions without breaking anything that already worked. | Planned |
| **Bump** | Opens a ready-to-review PR with the new model, repaired prompts, the regression suite, and an evidence report. | Planned |

## Development

Requires [uv](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/imhammad/modelbump.git
cd modelbump
uv sync
uv run modelbump --version
uv run pytest
```

## Roadmap

- [x] **Phase 0:** Foundations: packaging, CI, pre-commit, branch protection
- [x] **Phase 1:** Migration dataset + model retirement registry
- [x] **Phase 2:** Scanner + GitHub Action (`v0.1.0`)
- [ ] **Phase 3:** Model runner with caching and cost budgets
- [ ] **Phase 4:** Statistical comparison engine
- [ ] **Phase 5:** Regression suite synthesis + LLM-judge calibration
- [ ] **Phase 6:** Prompt repair
- [ ] **Phase 7:** End-to-end `modelbump bump` + `v1.0.0`

## Research

ModelBump is also a research project: an empirical study of how open-source LLM apps migrate off retired models (paper in preparation), and later a paper on the tool itself. Research notes, protocols and results are in the [research section of the docs](https://imhammad.github.io/modelbump/research/).

## License

[Apache License 2.0](https://github.com/imhammad/modelbump/blob/main/LICENSE)
