Metadata-Version: 2.5
Name: llm-handover
Version: 0.1.0
Summary: Find the AI models your code uses, see when each one retires, and test the replacement before you switch.
Project-URL: Homepage, https://www.volansaquilae.com/llm-handover/
Project-URL: Documentation, https://www.volansaquilae.com/llm-handover/
Project-URL: Dates, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/
Project-URL: Data, https://www.volansaquilae.com/downloads/llm-model-deprecation-dates.json
Author: Volans Aquilae
License-Expression: Apache-2.0 AND CC-BY-4.0
License-File: LICENSE
License-File: NOTICE
Keywords: anthropic,evals,gemini,llm,model-deprecation,model-migration,model-retirement,openai,regression-testing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# llm-handover

Find the AI models your code uses, see when each one retires, and test the replacement
before you switch.

Guide and a 79-second demo: https://www.volansaquilae.com/llm-handover/

`llm-handover` is a command-line tool with no dependencies beyond Python 3.10 or newer. It
does three things, separately or in one run:

1. **Finds every model ID in your code**: every ID in its retirement dataset, plus anything
   shaped like an OpenAI, Anthropic or Google model ID, so newer models show up too.
2. **Checks each one against retirement dates**: on the providers' own APIs and on Amazon
   Bedrock, Google Cloud and Microsoft Foundry, from a dataset that ships with the tool.
   No API keys and no network are needed for this.
3. **Replays your real prompts on the old and the new model** and shows what changed:
   broken JSON, dropped tool calls, new refusals, missing keywords, cut-off answers.

```console
$ llm-handover scan
llm-handover 0.1.0 | dataset 2026-11-18 (bundled) | as of 2026-11-20
Scanned 3 files in . (listed with git).
Found 5 model ID(s) in 5 place(s): 3 with dataset records, 2 without.

RETIRED  claude-3-haiku-20240307  [fails]
  Claude API: retired on 2026-04-20 (214 days ago) -> claude-haiku-4-5-20251001
  Used in: app/classify.ts:5

RETIRED  gemini-2.0-flash  [fails]
  Gemini API: retired on 2026-06-01 (172 days ago) -> gemini-3.6-flash
  Vertex AI (Gemini Enterprise Agent Platform): retired on 2026-06-01 (172 days ago) -> gemini-3.5-flash-lite
  Used in: app/config.yaml:2

WITHIN 30 DAYS  gpt-5  [fails]
  OpenAI API: retires on 2026-12-11 (21 days left) -> gpt-6.1-sol
  Azure OpenAI (Microsoft Foundry) [gpt-5 (version 2025-08-07)]: retires on 2027-02-09 (81 days left)
  Used in: app/support.py:4

No dates on the chosen platforms:
  claude-instant-1.2 (not in the dataset): app/classify.ts:4 (low confidence)
  gpt-6.1-sol (not in the dataset): app/config.yaml:4

Result: FAILED. 3 model ID(s) at or above --fail-on 30d: claude-3-haiku-20240307, gemini-2.0-flash, gpt-5
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).
```

## Contents

- [Install](#install)
- [Commands](#commands)
- [Finding models: `scan`](#finding-models-scan)
- [Looking up dates: `models` and `update`](#looking-up-dates-models-and-update)
- [Testing a replacement: `replay`](#testing-a-replacement-replay)
- [All three in one go: `run`](#all-three-in-one-go-run)
- [The test set format](#the-test-set-format)
- [What the checks look for](#what-the-checks-look-for)
- [Providers and API keys](#providers-and-api-keys)
- [Output files](#output-files)
- [Exit codes](#exit-codes)
- [Continuous integration](#continuous-integration)
- [States](#states)
- [The dataset](#the-dataset)
- [Limits](#limits)
- [How it is tested](#how-it-is-tested)
- [Licence](#licence)

## Install

```console
pipx install llm-handover
```

Or `pip install llm-handover`, or `uvx llm-handover` to run it without installing. From a
copy of this repository: `pip install .`

## Commands

| Command | What it does | Needs keys |
| --- | --- | --- |
| `llm-handover scan [PATH]` | Steps 1 and 2: find model IDs and check their dates | No |
| `llm-handover models [QUERY]` | Look up models in the dataset | No |
| `llm-handover update` | Download the latest dataset | No |
| `llm-handover replay TESTSET --candidate P:M` | Step 3: compare a new model with an old one | Yes |
| `llm-handover run [PATH] --tests TESTSET` | Steps 1, 2 and 3 in one go | Yes, for models it tests |

Every command has `--help`. Models are written as `provider:model`, where the provider is
`openai`, `anthropic`, `gemini` or `openai-compatible`: for example `openai:gpt-6.1-sol` or
`anthropic:claude-sonnet-5-5`. Only the first colon separates the two, so Bedrock-style IDs
such as `openai-compatible:anthropic.claude-3-haiku-20240307-v1:0` work.

## Finding models: `scan`

```console
llm-handover scan                      # the current folder
llm-handover scan src --fail-on 90d    # fail if anything retires within 90 days
llm-handover scan --platform openai-api,bedrock --format json
```

**Which files.** In a git work tree, files come from `git ls-files --cached --others
--exclude-standard`, so `.gitignore` is respected; otherwise the folder is walked. The output
says which was used. Skipped: folders such as `.git`, `node_modules`, `.venv`, `__pycache__`
and `handover-runs`; lock files; minified files and source maps; binary files; files over
2 MB (`--max-file-size`); symbolic links; and documentation files (`.md`, `.mdx`, `.rst`,
`.txt`, `.adoc`) unless you pass `--include-docs`. `--exclude GLOB` skips more.

**What matches.** A model ID only matches as a whole token: `gpt-5` does not match inside
`gpt-5.1` or `gpt-5-mini`, but it does match inside `"openai/gpt-5"`, `"gpt-5:online"` or at
the end of a sentence. Bedrock's region prefixes are understood, so `us.anthropic.claude-...`
finds `anthropic.claude-...`.

**Confidence.** A match is *high* confidence when it is quoted (`"gpt-5"`) or assigned to
something named like a model setting (`model: gpt-5`, `OPENAI_MODEL=gpt-5`, `--model gpt-5`),
and *low* otherwise, as in a comment. Only high-confidence matches can fail a scan. Very short
IDs such as `o1` and `o3` are only reported with high confidence. `--min-confidence high`
hides low-confidence matches.

**Urgency.** Each record gets exactly one label, as of today (UTC) or `--as-of`:

| Label | Meaning | In `--fail-on` |
| --- | --- | --- |
| RETIRED | Retired, or the retirement date is today or earlier | `retired` and up |
| WITHIN 30 DAYS | Retires within 30 days | `30d` and up |
| WITHIN 90 DAYS | Retires within 31 to 90 days | `90d` and up |
| MAY RETIRE ANY TIME | Only an earliest date was announced, and it has passed | `deprecated` and up |
| DEPRECATED, NO DATE | Deprecated or legacy, with no date | `deprecated` and up |
| SCHEDULED | Retires in more than 90 days | `any` |
| NO DATE | Listed, with no retirement planned | never |

`--fail-on` takes `never`, `retired`, `30d` (the default), `90d`, `deprecated` or `any`. A
model's most urgent record across the chosen platforms decides; `--platform` limits the
platforms to the ones you use (`openai-api`, `claude-api`, `gemini-api`, `bedrock`,
`vertex-ai`, `google-cloud`, `azure-openai`, `foundry`).

Model IDs with no record on the chosen platforms are listed under "No dates on the chosen
platforms" and never fail a scan.

**Formats.** `--format text` (the default), `json` (stable, with a `schema` field),
`markdown` (for pull request comments) or `github` (annotations on the lines, for GitHub
Actions).

## Looking up dates: `models` and `update`

```console
$ llm-handover models gpt-5-mini
2 record(s), dataset 2026-11-18, as of 2026-11-20:
  gpt-5-mini-2025-08-07  [openai-api]  WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
  gpt-5-mini  [openai-api]  WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).
```

`llm-handover update` downloads the latest dataset from
`https://www.volansaquilae.com/downloads/llm-model-deprecation-dates.json` into your cache
folder. The download is validated in full before anything is saved, the save is atomic, and an
older dataset never replaces a newer one unless you pass `--allow-older`. Nothing else in
llm-handover downloads the dataset; `scan` and `models` work offline. See
[The dataset](#the-dataset) for which copy is used.

## Testing a replacement: `replay`

```console
export OPENAI_API_KEY=...
llm-handover replay cases.jsonl --baseline openai:gpt-5 --candidate openai:gpt-6.1-sol
```

For each case in the test set, `replay` gets the old model's answer, asks the new model the
same thing, and compares the two. The old answer comes from:

- the case's recorded `baseline`, when it has one (no call is made);
- otherwise a live call to `--baseline`;
- `--live-baseline` calls `--baseline` even for cases with a recorded answer.

Without `--baseline`, every case needs a recorded answer, and the command refuses to start
if one is missing. Every problem in the test set, and every missing key, is reported before
the first call is made.

Results go to `--out` (by default `handover-runs/<UTC time>`), which must be empty unless you
pass `--force`; with `--force` only llm-handover's own result files are replaced. Cases run
4 at a time (`--concurrency`, 1 to 32). Rate limits and temporary server errors (HTTP 408,
425, 429, 500, 502, 503, 504 and 529), timeouts and network errors are retried 3 times
(`--retries`) with backoff, and a `Retry-After` header is honoured up to 60 seconds.

The run passes when no case reaches `--fail-on` (`regression`, the default, or `warning`, or
`never`) and no case failed. A case *fails* when it could not be judged, for example because
a call returned an error; that counts against the run, because nothing was proven about it.
A *skipped* case does not.

**Stopping.** The first Ctrl-C stops new cases from starting, lets the ones in flight finish,
and writes the reports (exit code 130). A second Ctrl-C quits at once; `results.jsonl` still
holds every case that finished.

## All three in one go: `run`

```console
llm-handover run --tests cases.jsonl
```

`run` scans the code, checks the dates, and tests the replacement for every model that is
urgent enough, using the replacement the provider names. Every case in the test set must say
which model it was written for, in `"target"`. Cases are grouped by target, and each group
gets exactly one outcome:

| Outcome | When |
| --- | --- |
| tested | The model's most urgent record is at or above `--risk` (default `90d`), and the provider names one replacement model, or you chose one |
| not at risk | Its most urgent record is below `--risk` |
| not in the dataset | The dataset has no record for it on its provider's own platform |
| needs a choice | It is at risk, but the provider names no single replacement (for example "gemini-3.8-flash or gemini-3.5-flash"), or it is behind an OpenAI-compatible endpoint |
| stopped | A Ctrl-C came before the group started |

Nothing is guessed. A group that needs a choice makes the run fail until you make one:

```console
llm-handover run --tests cases.jsonl \
  --candidate gemini:gemini-2.5-flash=gemini:gemini-3.8-flash
```

`--candidate OLD=NEW` also lets you test a model that is not at risk, or not in the dataset.

Some details:

- For the `openai`, `anthropic` and `gemini` providers, only records for that provider's own
  API count. For `openai-compatible`, records on every platform count, or only those you
  choose with `--platform`.
- When the provider writes a note after the replacement, such as `gpt-6.1-sol (reasoning.mode:
  pro)`, the model ID before the bracket is tested and the report says the note was not
  applied.
- When the replacement is itself retired, retiring within 90 days or deprecated, the report
  says so, so you do not migrate twice.
- When the old model is already retired, it cannot be called, so only cases with a recorded
  `baseline` are compared; the others are skipped. A group where every case was skipped is
  reported as NOTHING TESTED.
- Models found in the code that are at risk but have no test cases are listed. With
  `--fail-on-untested`, they and NOTHING TESTED groups fail the run.
- `--fail-on never` makes the run always pass (Ctrl-C still exits 130).
- The run's own results folder and the test set file are never scanned.

Example output:

```text
openai:gpt-5 -> openai:gpt-6.1-sol: DID NOT PASS
  4 ok, 1 changed, 1 regressed, 0 failed, 0 skipped
anthropic:claude-sonnet-4-5-20250929 -> anthropic:claude-sonnet-5-5: passed
  6 ok, 0 changed, 0 regressed, 0 failed, 0 skipped
openai:gpt-5.1: not at risk
  retires on 2027-04-01 (132 days left)
DID NOT PASS. Details: handover-runs/20261120-091500/report.md
```

## The test set format

A test set is a UTF-8 text file with one JSON object per line. `examples/testset.jsonl`, in the
source package, has one case of each kind.

```json
{"id": "refund-tool-1", "target": "openai:gpt-5", "messages": [{"role": "system", "content": "You are a support agent."}, {"role": "user", "content": "Please refund order A-1042."}], "tools": [{"name": "refund_order", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}}}], "expect": {"tool": "refund_order"}, "baseline": {"text": "", "tool_calls": [{"name": "refund_order", "arguments": {"order_id": "A-1042"}}]}}
```

| Field | Required | Meaning |
| --- | --- | --- |
| `id` | yes | 1 to 100 letters, digits, `.`, `_` or `-`, unique in the file |
| `messages` | yes | An optional `system` message first, then `user` and `assistant` messages taking turns, starting and ending with `user` |
| `target` | for `run` | The model the case was written for, as `provider:model` |
| `tools` | no | Tools the model may call: `name`, optional `description`, and `parameters` as a JSON Schema object |
| `tool_choice` | no | `auto`, `none` or `required`; needs `tools` |
| `max_tokens` | no | 1 to 200000, default 1024 |
| `temperature` | no | 0 to 2; left out, the provider's default is used |
| `expect` | no | Rules both answers are checked against: `json` (true or false), `contains` and `not_contains` (lists of text), `regex`, `tool` (a tool name) |
| `baseline` | no | The old model's recorded answer: `text`, and optional `tool_calls` as `{"name", "arguments"}` objects |
| `notes` | no | Free text, ignored |

Unknown fields are errors, so a typo such as `"expected"` cannot quietly turn a check off.
Every problem in the file is reported at once, with line numbers.

## What the checks look for

Each check gives a level: **regression** (likely to break something), **warning** (worth a
look), **info** or **pass**. A case is *regressed* if any check is a regression, *changed* if
any is a warning, and *ok* otherwise.

| Check | Regression | Warning | Info |
| --- | --- | --- | --- |
| `empty` | The new answer is empty and the old one was not | | |
| `expect.*` | The old answer met a rule and the new one does not | Both miss it | The new one now meets it |
| `json` | The old answer was JSON (an object or a list, optionally in a code fence) and the new one is not, or changed between object and list | | |
| `json.keys` | Top-level keys disappeared | New top-level keys | |
| `tool_calls` | The new model called no tool, or different tools in a different order | The new model called a tool where the old one did not | |
| `tool_args` | Argument names disappeared, or the arguments are not valid JSON | New argument names | Same names, different values |
| `refusal` | The new model refused where the old one answered | | The old one refused, the new one answered |
| `cut_off` | | The new answer stopped at `max_tokens` | |
| `length` | | The answer got more than 50% longer or shorter (`--length-threshold`) | |
| `similarity` | | | The wording changed a lot (`--similarity-threshold`, default 0.6) |

A refusal is detected when the provider flags one (OpenAI's `refusal` field, Anthropic's
`refusal` stop reason, Gemini's safety finish reasons and blocked prompts), or when the
answer opens with refusal wording such as "I can't help with". The wording check, the length
check and the similarity check are heuristics, and the report says so. Similarity compares
characters, not meaning. Read the regressed cases before you decide.

## Providers and API keys

Keys are read from environment variables only. They are never written to any file, and they
are removed from any error message before it is shown or saved.

| Provider | Key | Base address (optional) | Request |
| --- | --- | --- | --- |
| `openai` | `OPENAI_API_KEY` | `OPENAI_BASE_URL`, default `https://api.openai.com/v1` | `POST /chat/completions` |
| `anthropic` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL`, default `https://api.anthropic.com/v1` | `POST /messages` |
| `gemini` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | `GEMINI_BASE_URL`, default `https://generativelanguage.googleapis.com/v1beta` | `POST /models/MODEL:generateContent` |
| `openai-compatible` | `LLM_HANDOVER_COMPAT_API_KEY` (optional) | `LLM_HANDOVER_COMPAT_BASE_URL` (required) | `POST /chat/completions` |

For `openai-compatible`, the key is sent as `Authorization: Bearer KEY`, or in another header
named by `LLM_HANDOVER_COMPAT_AUTH_HEADER` (for example `api-key`). That covers gateways and
services that speak the OpenAI Chat Completions format: for example a LiteLLM proxy
(`http://localhost:4000/v1`), OpenRouter (`https://openrouter.ai/api/v1`) or Azure OpenAI's
v1 API (`https://YOUR-RESOURCE.openai.azure.com/openai/v1` with the `api-key` header). Check
your provider's documentation for its exact base address. Amazon Bedrock and Google Vertex AI
are not called directly in this version; use them through such a gateway.

Addresses must use https. Plain http is accepted for `localhost`, `127.0.0.1` and `::1`, or
anywhere if you set `LLM_HANDOVER_ALLOW_HTTP=1`. Redirects are never followed, because
following one would send your key to another address; use the final address instead.

Live calls cost money. A run makes one call to the new model per case, plus one to the old
model for each case without a recorded baseline.

## Output files

`replay` writes three files to its results folder:

- `results.jsonl`: one line per case, with both answers, every check, and the case's full
  state history. Lines are written as cases finish, so a crash keeps what was done, and the
  file is rewritten in test-set order at the end.
- `summary.json`: counts, check totals, tokens, median latency and the run's state history
  (`"schema": "llm-handover/replay-summary/1"`).
- `report.md`: what regressed, changed, failed and was skipped.

`run` writes `summary.json` (`"schema": "llm-handover/run-summary/1"`), `report.md` and
`scan.json` (the scan it started from, `"schema": "llm-handover/scan/1"`), plus one folder per
tested group with the three `replay` files.

The results hold the models' answers, which may contain whatever your prompts contain.
Treat them like the prompts.

## Exit codes

| Code | Meaning |
| --- | --- |
| 0 | Passed |
| 1 | Did not pass: a scan finding at or above `--fail-on`, a replay regression or failed case, or a run that did not pass |
| 2 | Could not run as asked: a bad option, an invalid file, a missing key, or a file that cannot be read or written |
| 3 | An internal error in llm-handover (please report it; `--debug` shows the details) |
| 130 | Stopped with Ctrl-C |

## Continuous integration

The scan needs no keys, so it can run on every pull request and once a week, because dates
change even when your code doesn't. In GitHub Actions:

```yaml
name: Model retirements
on:
  pull_request:
  schedule:
    - cron: "0 7 * * 1"
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install llm-handover
      - run: llm-handover update || echo "Using the bundled dates"
      - run: llm-handover scan --format github --fail-on 30d
```

With `--format github`, findings at or above `--fail-on` become error annotations, and other
models that are retired, retiring within 90 days or deprecated become warnings, each on its
file and line.

The source package also has `action.yml`, a composite GitHub Action that runs the same scan
and also writes a table to the job summary. Its inputs: `path` (default `.`), `fail-on`
(default `30d`), `platform`, `min-confidence` (default `low`), `update-dataset` (default
`true`) and `python-version` (default `3.12`).

Any other CI works the same way: `pipx run llm-handover scan --fail-on 30d`. To test
replacements in CI, run `llm-handover run --tests cases.jsonl` with the keys as secrets, and
keep the results folder as a build artefact.

## States

Every run, test case and results file moves through a fixed table of states, enforced at
run time; any move not in its table raises an error instead of happening. The history of
each one is saved in the results, so every result shows how it got there.

| Machine | States and moves |
| --- | --- |
| A run (`replay`, `run`, and each tested group inside `run`) | created → validated → running → completed; created, validated or running → aborted |
| A test case | pending → baseline → candidate → comparing → done; baseline, candidate or comparing → failed; pending → skipped |
| A results file | open → finished (rewritten in order) or closed (the run did not finish normally) |

The only signal shared between threads is the stop request: set once by the first Ctrl-C,
never cleared.

## The dataset

The dates come from the *LLM model deprecation and retirement dates* dataset by Volans
Aquilae: one record per model ID and platform, with status, dates, the named replacement and
the source page each record was checked against. It is published at
<https://www.volansaquilae.com/guides/llm-model-deprecation-dates/> under the Creative Commons
Attribution 4.0 licence (CC BY 4.0), and a copy ships inside the package.

Which copy is used, in this order:

1. a file you pass with `--data`;
2. the copy saved by `llm-handover update`, if it is valid and not older than the bundled one;
3. the bundled copy.

The cache folder is `~/.cache/llm-handover` on Linux (or under `$XDG_CACHE_HOME`),
`~/Library/Caches/llm-handover` on macOS and `%LOCALAPPDATA%\llm-handover\Cache` on Windows;
`LLM_HANDOVER_CACHE_DIR` overrides it. A warning is shown when the dataset in use is more
than 14 days old.

Providers change dates and replacements without much notice. Treat the dataset as a strong
hint and check the provider's page (each record's `source_url`) before acting on a date.

## Limits

- The dataset covers text and reasoning models from OpenAI, Anthropic and Google, on their
  own APIs, Amazon Bedrock, Google Cloud and Microsoft Foundry. Models outside it are found
  by their shape but have no dates.
- Finding models by their shape can list strings that are not model IDs, such as an HTML
  anchor named like a model. Those are only listed, never failed on.
- A model ID built at run time, for example from two strings joined together, cannot be found.
- Test cases are text conversations with optional tool definitions. Tool results, images,
  audio, files and streaming are not supported in this version.
- Some models reject some settings: reasoning models may refuse a `temperature`, for example.
  The call then fails and the case shows the provider's error, which is often exactly what you
  need to know before switching.
- Models that think before answering count those tokens against `max_tokens` on some
  providers. If new answers come back cut off or empty, raise `max_tokens` in the cases.
- GitHub shows a limited number of annotations per step, so the action writes at most 10 per
  model and 50 in all; the text report lists every place.

## How it is tested

The test suite covers every command end to end against a local server that imitates the
OpenAI, Anthropic and Gemini APIs, through the same HTTP code that talks to the real ones,
including authentication headers, retries, rate limits, redirects, oversized responses,
timeouts and Ctrl-C. It runs on Python 3.10 to 3.14. Calls to the real provider APIs are not
part of the automated tests, because they need paid keys; the request and response formats
follow each provider's public API documentation. If something differs on a live API, please
open an issue with the error message.

```console
pip install -e . pytest ruff mypy
ruff check . && ruff format --check . && mypy && pytest
```

## Licence

The code is licensed under the Apache License 2.0 (the `LICENSE` file). The bundled dataset
is © Volans Aquilae, licensed under CC BY 4.0; the `NOTICE` file has the attribution to keep
if you redistribute it.
