Metadata-Version: 2.5
Name: mlx-model-doctor
Version: 0.8.0
Summary: Validate MLX model repositories before loading them.
Project-URL: Homepage, https://github.com/IonDen/mlx-model-doctor
Project-URL: Source, https://github.com/IonDen/mlx-model-doctor
Project-URL: Issues, https://github.com/IonDen/mlx-model-doctor/issues
Project-URL: Changelog, https://github.com/IonDen/mlx-model-doctor/blob/master/CHANGELOG.md
Project-URL: Roadmap, https://github.com/IonDen/mlx-model-doctor/blob/master/ROADMAP.md
Author-email: Denis Ineshin <denis.ineshin@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: apple-silicon,cli,diagnostics,huggingface,mlx,mlx-lm,model-validation,quantization,safetensors,tokenizer,validation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: huggingface-hub>=1.0
Provides-Extra: mlx-lm
Requires-Dist: mlx-lm>=0.31.3; extra == 'mlx-lm'
Provides-Extra: mlx-vlm
Requires-Dist: mlx-vlm<0.7,>=0.6.12; extra == 'mlx-vlm'
Description-Content-Type: text/markdown

# mlx-model-doctor

<p align="center">
  <img src="https://raw.githubusercontent.com/IonDen/mlx-model-doctor/master/docs/assets/mlx-model-doctor.jpeg" alt="mlx-model-doctor" width="100%">
</p>

[![PyPI version](https://img.shields.io/pypi/v/mlx-model-doctor.svg)](https://pypi.org/project/mlx-model-doctor/)
[![Python versions](https://img.shields.io/pypi/pyversions/mlx-model-doctor.svg)](https://pypi.org/project/mlx-model-doctor/)
[![License: Apache 2.0](https://img.shields.io/pypi/l/mlx-model-doctor.svg)](https://github.com/IonDen/mlx-model-doctor/blob/master/LICENSE)

Validate an MLX / Hugging Face model repository before you load it.

A model repo can be broken in ways you only discover halfway through `load()`: a `config.json` that's missing or internally inconsistent, a missing tokenizer file, a `model.safetensors.index.json` that points at shards that aren't there, quantization metadata that uses a mode or group size MLX rejects, a chat template that's absent or whose stop token has a typo, a corrupt safetensors header, a quantized layer whose packed weight and scales shapes disagree, or a model that simply won't fit in the memory you have. `mlx-model-doctor` checks those up front and prints a report, so a bad repo fails fast with a clear reason instead of a confusing crash.

The checks read repository metadata and the safetensors *header* — `config.json`, the tokenizer files, the safetensors index, quantization fields, and the tensor map (dtypes, shapes, byte-offsets) parsed from the header alone. They need no GPU or MLX and never download the weights; on the Hub the header arrives over a small HTTP range request, so the checks stay cheap to run anywhere. For the default `text` profile, an optional `--smoke` check loads the model through `mlx-lm` (Apple Silicon) under a memory cap, to confirm it loads and generates. VLM smoke through `mlx-vlm` is future work.

```bash
mlx-model-doctor check local ./my-model
mlx-model-doctor check hf mlx-community/Llama-3.2-3B-Instruct-4bit
```

See [EXAMPLES.md](EXAMPLES.md) for real, dated transcripts plus newer command examples.

## Install

```bash
pip install mlx-model-doctor
# With the optional mlx-lm smoke check (Apple Silicon):
pip install "mlx-model-doctor[mlx-lm]"
```

Or with `uv`:

```bash
uv add mlx-model-doctor
uv add "mlx-model-doctor[mlx-lm]"
```

Verify the install:

```bash
mlx-model-doctor version
mlx-model-doctor --help
```

Requires Python ≥ 3.11. The static checks are pure Python and need only `huggingface-hub`; the optional `--smoke` runtime check currently applies to the default `text` profile and needs `mlx-lm` plus Apple Silicon.

## What it checks

The built-in `text` plugin runs these against a model repository, broadly in this order:

`text` is the default plugin. For vision-language repositories, pass `--plugin vlm`
to run the explicit non-runtime VLM profile. The VLM plugin keeps the same metadata
and safetensors-header posture: no weights are downloaded and no model runtime is
loaded.

- **Required files** — `config.json` is present and readable.
- **Config consistency** — `config.json` parses, and its `model_type` is set.
- **Tokenizer** — the tokenizer files a text model needs are present, and the special-token configuration is coherent.
- **Chat template** — a chat/instruct model declares a chat template (in `tokenizer_config.json` or a `chat_template.jinja`), and the end-of-turn token its template emits is a registered special token. A typo'd stop token loads fine and then never stops generating.
- **Safetensors index** — when the weights are sharded, `model.safetensors.index.json` is valid and every shard it references exists.
- **Tensor header** — read the safetensors header itself (the tensor map: dtypes, shapes, byte-offsets; no weight download) to catch a corrupt header (overlapping or out-of-bounds tensor offsets), a weight map that points at tensors no shard contains, a declared tied embedding that contradicts the stored weights, and an MLX-quantized layer whose packed-weight and scales shapes don't agree. These run by default; pass `--skip-weights` to skip them for a faster config-only pass.
- **Quantization metadata** — quantization fields are present and use a valid MLX mode with a valid group size and bit width (`affine`, `mxfp4`, `mxfp8`, `nvfp4`). This reads the metadata, not the tensors.
- **Generation tokens** — the `eos` / `pad` / `bos` token IDs are present and agree across `config.json`, `generation_config.json`, and `tokenizer_config.json`.
- **Memory budget** — an estimate of the memory the model needs at your context length, compared against a budget you pass with `--max-memory`.

Each check returns a result with a status (`pass` / `warn` / `fail` / `skip`), a message, and — when something is wrong — a remediation hint. The report aggregates them, and the process exit code reflects the worst result under your fail policy.

## Python API

```python
from mlx_model_doctor import check_local_model, check_hf_model

report = check_local_model("./my-model")
print(report.summary)            # {"pass": 9, "warn": 1, "fail": 0, "skip": 2}
for result in report.results:
    print(result.status, result.check_id, result.message)

# Hugging Face repos (hits the Hub):
report = check_hf_model("mlx-community/Llama-3.2-3B-Instruct-4bit")
```

`DoctorReport` renders to text, JSON, or Markdown (`render_text` / `render_json` / `render_markdown`), and the result objects are frozen dataclasses, so the output is stable to diff in CI.

## Commands

| Command | What it does |
|---|---|
| `version` | Print the version plus the active Python, virtualenv, and dependency status. |
| `man` | Print usage examples and the exit-code table. |
| `plugins` | List registered check plugins (`text`, `vlm`). |
| `check local <path>` | Validate a model directory on disk. |
| `check hf <repo_id>` | Validate a model repository on the Hugging Face Hub (network). |
| `sample hf` | Survey likely-MLX repos for an author and validate a deterministic sample. |

`check` accepts `--format {text,json,markdown,github}`, `--output <file>`, `--max-memory <e.g. 32gb>`, `--context-length <n>`, `--fail-on {error,warn,never}`, `--skip-weights` (skip the tensor-header checks for a faster config-only pass), and `--smoke` for the default text smoke backend. Use `--plugin vlm` for vision-language repositories; omit it for the default `text` profile. The `github` format prints GitHub Actions annotations (see [Use it in CI](#use-it-in-ci)).

Exit codes: `0` checks passed (under the fail policy), `1` checks found failures, `2` tool error — a bad target, a missing dependency, or zero checks run.

## The Hugging Face path

`check hf` and `sample hf` talk to the Hub through `huggingface-hub`. They read repository metadata (the file list, sizes, the small text files, and the safetensors header over a range request) rather than downloading the weights, but they do need network access, and an auth or rate-limit problem surfaces as a clear tool error rather than a stack trace. `sample hf` is a survey: it lists an author's repos, keeps the ones that look like MLX models, validates a deterministic sample of them, and reports each as its own batch item — a per-model failure is recorded and the run continues.

## Validate before uploading to Hugging Face

For a model you build or convert locally, run the static checks before upload and treat warnings as release blockers unless you have reviewed them:

```bash
mlx-model-doctor check local ./dist/my-mlx-model --fail-on warn
hf upload my-org/my-mlx-model ./dist/my-mlx-model --repo-type model
mlx-model-doctor check hf my-org/my-mlx-model --fail-on warn
```

If your publisher is Python-based, keep the same order:

```python
from huggingface_hub import upload_folder

# Run `mlx-model-doctor check local ./dist/my-mlx-model --fail-on warn` first.
upload_folder(
    repo_id="my-org/my-mlx-model",
    folder_path="./dist/my-mlx-model",
    repo_type="model",
)
```

Use `--fail-on warn` before upload when you want a clean producer release gate. Use the default `--fail-on error` when warnings are acceptable but hard failures should still block. After upload, `check hf` verifies that the Hub repository exposes the same files and metadata the local directory did.

## Use it in CI

Gate a pull request on a model repository with the [GitHub Marketplace Action](https://github.com/marketplace/actions/mlx-model-doctor). It runs the static checks (no weights downloaded, no GPU), writes the report to the job summary, and fails the job under your fail policy:

```yaml
- uses: IonDen/mlx-model-doctor@v0
  with:
    source: hf
    target: mlx-community/Llama-3.2-3B-Instruct-4bit
    plugin: text
    fail-on: warn
```

For a vision-language model, opt into the VLM profile:

```yaml
- uses: IonDen/mlx-model-doctor@v0
  with:
    source: hf
    target: mlx-community/InternVL3-2B-4bit
    plugin: vlm
```

Add `version: "==0.7.0"` to pin the tool to a release; without it the action installs the latest published version.

For a model directory you keep in git, validate it on every commit with the pre-commit hook:

```yaml
repos:
  - repo: https://github.com/IonDen/mlx-model-doctor
    rev: v0.7.0
    hooks:
      - id: mlx-model-doctor
        args: ["path/to/model"]
```

## Output contract

`--format json` prints a stable, versioned payload. The top-level fields are:

| Field | Type | Description |
|---|---|---|
| `schema_version` | string | Schema major.minor version (e.g. `"1.0"`), independent of the package version. |
| `target` | string | The model path or repo ID that was checked. |
| `source` | `"local"` or `"hf"` | Where the model came from. |
| `plugin` | string | The check plugin that ran (e.g. `"text"`). |
| `summary` | object | Check counts: `pass`, `warn`, `fail`, `skip` (integers). |
| `environment` | object | Reserved and currently always empty (`{}`); kept empty so JSON output stays stable to diff across environments. |
| `zero_check_reason` | string or null | When a run produced no checks (a zero-check run, which exits `2`), a message naming the responsible plugin; null on a normal run. |
| `results` | array | One entry per check; see below. |

Each result in `results[]` has:

| Field | Type | Description |
|---|---|---|
| `check_id` | string | Namespaced identifier, e.g. `"text/files.required"`. |
| `title` | string | Short human-readable check name. |
| `status` | string | `"pass"`, `"warn"`, `"fail"`, or `"skip"`. |
| `severity` | string | `"info"`, `"low"`, `"medium"`, or `"high"`. |
| `message` | string | What was found. |
| `remediation` | string or null | What to do if the check fired. |
| `details` | object | Open object with check-specific key/value pairs. |
| `duration_s` | number or null | Reserved; currently always `null`. Per-check timing is not emitted so JSON output stays stable to diff run-to-run. |

The machine-readable schema ships with the package at `mlx_model_doctor/schema/report.v1.schema.json` and is validated against real output in CI.

Exit codes: `0` checks passed under the fail policy, `1` failures found, `2` a tool error (a bad target, a missing dependency, or zero checks run). `--format github` reports the same results as GitHub Actions annotations; inside a workflow it also writes the Markdown report to the job summary and the `pass` / `warn` / `fail` / `skip` / `exit-code` / `schema-version` values to the step outputs.

## Stability policy

### Public API

The names you can depend on — only change on a major release:

`check_local_model`, `check_hf_model`, `CheckOptions`, `DoctorReport`, `CheckResult`, `render_json`, `render_text`, `render_markdown`, `render_github`, `exit_code_for`, `FailOn`, and the error types `ModelDoctorError`, `TargetError`, `DependencyError`, `MemorySafetyError`. `exit_code_for` raises `ValueError` on an unrecognized `fail-on` value.

### Internal layer

The check, plugin, and target Protocols; `CheckContext`; the plugin registry; and the `hub=` parameter on `check_hf_model` (a test injection seam whose type may change) are internal and not stable across releases.

### Schema versioning

`schema_version` is `MAJOR.MINOR`, versioned independently of the package. A minor bump adds new optional fields or new values to open fields (such as the memory check's `estimate_source` values). A major bump means a documented field was removed, renamed, or retyped, or a closed enum (`status`, `severity`, `source`) changed.

The top-level object, `summary`, and each entry in `results[]` are closed (`additionalProperties: false`), so a new field there is a coordinated schema edit plus a minor version bump. Validate against the schema that matches the payload's `schema_version`, not a pinned older copy — otherwise a newer payload's added field will fail your validator.

### Promoted `details` keys

`details` is otherwise free-form, but three keys from the memory check are stable across the 1.x schema line: `lower_bound_bytes`, `estimate_source`, and `memory_lower_bound_kind`. `lower_bound_bytes` is a structural lower bound — it counts attention, MLP, and embedding parameters at ≤16-bit weights (or quantized-equivalent) plus KV cache, but excludes norms, biases, and an untied `lm_head`. It sits below real runtime use and is not a fit guarantee.

### Batch output

The `sample hf --format json` survey has its own published schema, at `mlx_model_doctor/schema/sample-batch.v1.schema.json` and validated against real output in CI. It carries a `schema_version` of `sample-batch/MAJOR.MINOR` (same bump rules as above), and each checked item embeds a full single-`check` report that conforms to `report.v1.schema.json`.

## Version sensitivity

The checks encode behavior from specific upstream versions (MLX, transformers,
safetensors). When a value falls outside the known set — a quantization mode
added in a newer MLX, an unfamiliar safetensors dtype — the check **warns**
rather than failing. Only structurally invalid metadata (wrong type, missing
required fields) produces a failure. Each version-bound table carries a comment
naming the upstream version it was verified against.

## Status

**Beta (0.8.0).** The static `check local` path and the report/CLI surface are solid and well tested; this release moves the project from Alpha to Beta. Version-bound check tables (MLX quantization modes, safetensors dtypes) now warn rather than fail on a recognized-but-unlisted value, so a repository built against a newer upstream release is flagged as unverified instead of rejected outright — see [Version sensitivity](#version-sensitivity). The optional runtime smoke check now covers vision-language repositories too: `--smoke --plugin vlm` loads a model through `mlx-vlm` and generates from a dummy image under the same advisory memory caps as the text path, with remote code execution refused by default. `sample hf` gained a configurable scan depth (`--max-candidates`), a listing-signal filter (`--signal-filter`), and a local listing cache (`--no-cache` / `--cache-ttl`) for surveying more of an author's catalog without repeating Hub calls. The safetensors header (read without downloading weights) backs four tensor-level checks — offset corruption, weight-map parameter sanity, tied-embedding consistency, and MLX quantized-layer shape consistency — which run by default (`--skip-weights` opts out). A single `check` reports whether a repository looks like an MLX model and why; the VLM profile adds image-processor and image-token wiring checks for vision-language repositories. The quantized-shape and quantization-mode checks read each layer's own `bits`/`group_size`/`mode`, so a mixed-precision model (4-bit experts with 8-bit dense and router layers) is validated per layer rather than reported as broken. The memory estimate handles mixed precision the same way: when a model mixes bit widths it takes the weight figure from the stored file sizes instead of the model-level setting. The Hugging Face path (`check hf`, `sample hf`) is implemented and tested offline against fakes; its live behavior is exercised by opt-in network tests. It also ships a GitHub Action and a pre-commit hook. The public API and JSON output now have a documented, versioned stability contract — see [Output contract](#output-contract) and [Stability policy](#stability-policy). Pin a version if you depend on the schema or the API.

## License

Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE). Validating a model repository does not touch the model's own weights or license; those belong to their respective authors.

## Acknowledgements

- [Apple ML Explore](https://github.com/ml-explore/mlx) for MLX and [`mlx-lm`](https://github.com/ml-explore/mlx-lm).
- [Hugging Face](https://github.com/huggingface/huggingface_hub) for the Hub client and `safetensors`.

## Sister projects

Other MLX libraries for Apple Silicon:

- [mlx-taef](https://github.com/IonDen/mlx-taef) — tiny autoencoders for fast diffusion-latent previews and low-memory decode (FLUX / SD).
- [mlx-teacache](https://github.com/IonDen/mlx-teacache) — TeaCache residual caching to skip redundant FLUX denoising steps.
- [mlx-quant-fidelity](https://github.com/IonDen/mlx-quant-fidelity) — measure how much quality a quantization costs (KL divergence, top-token flips, perplexity delta).

---

By Denis Ineshin · [ineshin.space](https://ineshin.space)
