Metadata-Version: 2.4
Name: justfit-llm
Version: 2.3.1
Summary: Apple silicon inference with coordinated CPU, GPU and ANE work
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/YuhuaBillChen/justfit-llm
Project-URL: Repository, https://github.com/YuhuaBillChen/justfit-llm
Project-URL: Issues, https://github.com/YuhuaBillChen/justfit-llm/issues
Classifier: Development Status :: 5 - Production/Stable
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Programming Language :: Python :: 3
Requires-Python: <3.15,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
License-File: source/THIRD_PARTY_NOTICES
Requires-Dist: annotated-doc==0.0.5
Requires-Dist: anyio==4.14.2
Requires-Dist: attrs==26.1.0
Requires-Dist: certifi==2026.10.19
Requires-Dist: cffi==2.1.1
Requires-Dist: click==8.4.2
Requires-Dist: cryptography==50.0.1
Requires-Dist: filelock==3.32.3
Requires-Dist: fsspec==2026.7.0
Requires-Dist: h11==0.16.0
Requires-Dist: hf-xet==1.6.0
Requires-Dist: httpcore==1.0.9
Requires-Dist: httpx==0.28.1
Requires-Dist: huggingface_hub==1.28.0
Requires-Dist: idna==3.19
Requires-Dist: Jinja2==3.1.6
Requires-Dist: jsonschema==4.26.0
Requires-Dist: jsonschema-specifications==2025.9.1
Requires-Dist: llguidance==1.8.0
Requires-Dist: markdown-it-py==4.2.0
Requires-Dist: MarkupSafe==3.0.3
Requires-Dist: mdurl==0.1.2
Requires-Dist: numpy==2.5.2
Requires-Dist: packaging==26.3
Requires-Dist: Pillow==13.0.0
Requires-Dist: Pygments==2.21.0
Requires-Dist: pycparser==3.0
Requires-Dist: pypdfium2==5.13.0
Requires-Dist: PyYAML==6.0.3
Requires-Dist: referencing==0.37.0
Requires-Dist: regex==2026.7.19
Requires-Dist: rich==15.0.0
Requires-Dist: rpds-py==2026.6.3
Requires-Dist: safetensors==0.8.0
Requires-Dist: shellingham==1.5.4
Requires-Dist: tokenizers==0.22.2
Requires-Dist: tqdm==4.70.0
Requires-Dist: transformers==5.15.1
Requires-Dist: typer==0.27.1
Requires-Dist: typing_extensions==4.16.0
Requires-Dist: fastapi==0.141.1
Requires-Dist: uvicorn==0.52.1
Requires-Dist: annotated-types==0.8.0
Requires-Dist: pydantic==2.14.1
Requires-Dist: pydantic_core==2.50.1
Requires-Dist: starlette==1.9.0
Requires-Dist: typing-inspection==0.4.4
Dynamic: license-file

<div align="center">
  <picture>
    <source media="(max-width: 600px)" srcset="docs/assets/justfit-banner-mobile.svg">
    <img src="docs/assets/justfit-banner.svg" alt="JustFit — long-context LLM serving on Apple silicon" width="100%">
  </picture>

  [![Version](https://img.shields.io/badge/version-2.3.1-087f78)](CHANGELOG.md)
  [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-246a96)](LICENSE)
  [![Apple silicon](https://img.shields.io/badge/platform-Apple%20silicon-526b81)](#supported-setup)

  **[Get started](#get-started) · [Results](#measured-results) · [Documentation](#documentation) · [简体中文](README.zh-CN.md)**
</div>

JustFit is a local inference system for serving a **27B language-and-vision model on memory-constrained Macs**. It coordinates compressed context, a resident speculative draft, concurrent requests and CPU/GPU/ANE work under one unified-memory budget.

**[2.3.1 is available on GitHub Releases](https://github.com/YuhuaBillChen/justfit-llm/releases/tag/v2.3.1)** with source, Python packages and an Apple-notarized native runtime; the six ANE bundles of [2.3.0](https://github.com/YuhuaBillChen/justfit-llm/releases/tag/v2.3.0) apply unchanged. Install from [PyPI](https://pypi.org/project/justfit-llm/2.3.1/) using the instructions below.

## Why JustFit

- **Keep long histories alongside the draft.** Warm-prefix tests reach a 320K aggregate page capacity across two live requests while retaining the speculative draft. This is a tested B2 bound, not a 320K single-request or cold-prefill guarantee.
- **Make concurrent work useful.** Bound transient workspaces, share compressed pages and preserve committed state across scheduling and cold restarts.
- **Put idle accelerators to work.** Use CPU-assisted prefill on M4 Pro, GPU A8 paths on M5 Pro/M6, and qualified ANE work when execution phase and memory allow it. Vision offload aims to leave GPU capacity for text; it does not guarantee lower image latency.
- **Serve through a familiar API.** OpenAI-compatible chat, images, streaming, structured output and native-enforced thinking budgets.

## Measured results

<a href="docs/assets/system-comparison.png"><img src="docs/assets/system-comparison.png" alt="Four-system comparison on M4 Pro, M5 Pro and M6: tested retained context, 16K prefill and 16K decode rates" width="100%"></a>

**Capacity and speed are separate experiments.** Solid bars show tested B1 bounds; hatching extends them to the B2 total. Both JustFit generations reach a 320K page capacity using their respective warm-prefix protocols; v2 retains its draft. Speed uses eight fixed SPEED-Bench 16K tasks with 512 forced output tokens. Weight formats, decoders and profiles differ. M4 Pro/M5 Pro have physical 24 GiB; M6 has physical 32 GiB, with 8 GiB locked only for the v1/v2 B2 capacity tests. M6 speed runs use physical 32 GiB.

At 16K, v2's median decode rate is **2.0–2.5× the v1 rate** across these configurations. Relative to upstream Splash 1.3.0, median prefill improves **9–46%**, while singleton decode declines **2–9%**. These whole-system results are not isolated mechanism gains or a chip-generation ranking.

[Protocols, exact rates and limitations →](docs/RESULTS.md)

## Get started

### Supported setup

- **Apple silicon and macOS 27.0+.** Reference configurations: M4 Pro, M5 Pro and M6, with at least 24 GiB. Other chip variants are not individually qualified.
- **Python 3.12 recommended**; the package declares support for Python 3.12–3.14.
- The prebuilt installation below requires no local compilation. Building from source separately requires a **compatible Xcode/CoreAI SDK and Metal compiler toolchain**.
- Internet access and storage for the pinned model pack: approximately **17.4 GB**, plus dependencies, build outputs and caches. The model repository is publicly accessible without a Hugging Face token at the checked revision.

Installing the Python package on Linux does not enable native inference there. The standalone GPU engine's macOS 26.4 source target is separate from this complete macOS 27/CoreAI setup.

### Install the reviewed release

Create a fresh environment and install the pinned release from PyPI:

```bash
python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install justfit-llm==2.3.1
justfit verify
justfit doctor
justfit serve \
  --runtime-manifest https://github.com/YuhuaBillChen/justfit-llm/releases/download/v2.3.1/runtime-manifest.json \
  --runtime-sha256 <runtime-manifest.json SHA256 from the release SHA256SUMS> \
  --ane-manifest https://github.com/YuhuaBillChen/justfit-llm/releases/download/v2.3.0/ane-index.json \
  --ane-manifest-sha256 fd1b71cfa9d4e785741dc2e0ba4d43f2a4dd556a62c0a61f37838b7f9e3ec99d
```

First run with these options downloads and verifies the prebuilt runtime, matching ANE assets and pinned model, then serves on **`http://127.0.0.1:8000`**. ANE bundles require an exact chip and macOS build `26A434` match; on other builds, omit the two ANE options to use GPU vision. Later runs of `justfit serve` reuse installed components. Ctrl+C stops the foreground server.

A first run without runtime options or an installed runtime uses the source-build path and requires developer tools. [Quickstart](QUICKSTART.md) covers source builds, ports and models.

### Send a request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "incoai/Qwen3.8-27B-Splash",
    "messages": [{"role": "user", "content": "What is 17 multiplied by 19?"}],
    "max_tokens": 64,
    "reasoning_effort": "none"
  }'
```

The model ID retains the upstream runtime pack's name; the engine and CLI are JustFit. The default model is pinned to revision `9d27070b71f7142c6b6025f03ac011d70a73cb48`.

## How it works

| Mechanism | Responsibility |
| --- | --- |
| **KVExec** | Consume compressed history with bounded reconstruction and rotated-domain attention. |
| **PhaseSwap** | Control component and workspace lifetimes while the speculative draft stays resident. |
| **StateTrans** | Preserve committed KV, recurrent and draft state across scheduling and persistence. |
| **CrossFire** | Coordinate CPU, GPU and ANE work using phase ownership and memory admission. |

### ANE behavior

| Reference preset | Language prefill | ANE vision eligibility |
| --- | --- | --- |
| M4 Pro / `m4` | CPU-assisted GPU | Text prefill or decode |
| M5 Pro / `m5` | GPU A8 | Text prefill or decode |
| M6 / `m6` | GPU A8 + ANE FFN | Decode with no active text prefill |

Vision eligibility requires qualified 4K/8K assets, a matching chip and OS build, and memory admission. **4K/8K count image patches, not text tokens or screen pixels.** Idle requests, unsupported inputs and unavailable resources use GPU. M6 reserves ANE for language prefill before admitting vision work. [ANE setup and download interface](docs/ANE_DOWNLOADS.md).

## Documentation

| I want to… | Start here |
| --- | --- |
| Install, build or choose a port/model | [Quickstart](QUICKSTART.md) · [Python installation](docs/PYPI.md) |
| Understand the measurements | [Results and protocols](docs/RESULTS.md) · [Preview validation](docs/PREVIEW_VALIDATION.json) |
| Change device settings | [Configuration](docs/CONFIGURATION.md) · [ANE policy](ANE_POLICY.md) |
| Prepare ANE assets or a prebuilt runtime | [ANE downloads](docs/ANE_DOWNLOADS.md) · [Offline provisioning](docs/ANE_PROVISIONING.md) · [Runtime distribution](docs/RUNTIME_DISTRIBUTION.md) |
| Deploy a managed service | [Deployment and rollback](docs/DEPLOYMENT.md) · [Broker integration](service/broker-integration/README.md) |
| Control reasoning output | [Thinking budget](docs/THINKING_BUDGET.md) |
| Report a problem or contribute | [Issues](https://github.com/YuhuaBillChen/justfit-llm/issues) · [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) |

The default endpoint is local. Network deployment needs appropriate authentication and TLS; see [security scope](SECURITY.md). Native runtime sources are in `source/`, ANE tooling in `vision-tools/`, and optional managed-service components in `service/`.

## Paper and citation

**JustFit v2: Long-Context Local LLM Serving with Just-in-Time State Management** — Yuhua Chen, 2026. **[Read the preprint (PDF)](https://github.com/YuhuaBillChen/justfit-llm/blob/main/paper/JustFit-v2.pdf)** · [Paper and citation files](paper/README.md). This link always points to the current manuscript; an arXiv link will be added when public.

**Full-set AIME 2026:** on M5 Pro, the same four-bit target with TQ4 KV scores **86/90 (95.6%) with A8 off** and **85/90 (94.4%) with A8 on**, covering all 30 questions at three seeds. Each arm has three capped generations included in the denominator. This tests A8 within the quantized configuration.

If JustFit supports your research, please cite:

```bibtex
@unpublished{chen2026justfitv2,
  author = {Chen, Yuhua},
  title = {{JustFit v2}: Long-Context Local LLM Serving with Just-in-Time State Management},
  year = {2026},
  month = oct,
  note = {Preprint},
  url = {https://github.com/YuhuaBillChen/justfit-llm/blob/main/paper/JustFit-v2.pdf}
}
```

[Machine-readable citation](CITATION.cff) · [JustFit v1 paper](https://arxiv.org/abs/2609.17475)

## Acknowledgments and license

JustFit's native runtime is derived from **[Splash 1.3.0](https://github.com/incoai/splash)**. Splash already provides language ANE FFN, paged KV tiering and composite-state persistence; JustFit extends their integration, retention and resource coordination. Upstream authors, licenses and implementation credits are preserved in [NOTICE](NOTICE) and [third-party notices](source/THIRD_PARTY_NOTICES). Model weights and derived assets retain their own licenses.

Code is available under [Apache 2.0](LICENSE). JustFit maintainers handle this project's support; please direct JustFit issues here rather than to upstream maintainers.
