Metadata-Version: 2.4
Name: justfit-llm
Version: 2.3.0
Summary: Apple silicon inference with coordinated CPU, GPU and ANE work
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/YuhuaBillChen/justfit-llm
Project-URL: Repository, https://github.com/YuhuaBillChen/justfit-llm
Project-URL: Issues, https://github.com/YuhuaBillChen/justfit-llm/issues
Classifier: Development Status :: 5 - Production/Stable
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Programming Language :: Python :: 3
Requires-Python: <3.15,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
License-File: source/THIRD_PARTY_NOTICES
Requires-Dist: annotated-doc==0.0.5
Requires-Dist: anyio==4.14.2
Requires-Dist: attrs==26.1.0
Requires-Dist: certifi==2026.10.19
Requires-Dist: cffi==2.1.1
Requires-Dist: click==8.4.2
Requires-Dist: cryptography==50.0.1
Requires-Dist: filelock==3.32.3
Requires-Dist: fsspec==2026.7.0
Requires-Dist: h11==0.16.0
Requires-Dist: hf-xet==1.6.0
Requires-Dist: httpcore==1.0.9
Requires-Dist: httpx==0.28.1
Requires-Dist: huggingface_hub==1.28.0
Requires-Dist: idna==3.19
Requires-Dist: Jinja2==3.1.6
Requires-Dist: jsonschema==4.26.0
Requires-Dist: jsonschema-specifications==2025.9.1
Requires-Dist: llguidance==1.8.0
Requires-Dist: markdown-it-py==4.2.0
Requires-Dist: MarkupSafe==3.0.3
Requires-Dist: mdurl==0.1.2
Requires-Dist: numpy==2.5.2
Requires-Dist: packaging==26.3
Requires-Dist: Pillow==13.0.0
Requires-Dist: Pygments==2.21.0
Requires-Dist: pycparser==3.0
Requires-Dist: pypdfium2==5.13.0
Requires-Dist: PyYAML==6.0.3
Requires-Dist: referencing==0.37.0
Requires-Dist: regex==2026.7.19
Requires-Dist: rich==15.0.0
Requires-Dist: rpds-py==2026.6.3
Requires-Dist: safetensors==0.8.0
Requires-Dist: shellingham==1.5.4
Requires-Dist: tokenizers==0.22.2
Requires-Dist: tqdm==4.70.0
Requires-Dist: transformers==5.15.1
Requires-Dist: typer==0.27.1
Requires-Dist: typing_extensions==4.16.0
Requires-Dist: fastapi==0.141.1
Requires-Dist: uvicorn==0.52.1
Requires-Dist: annotated-types==0.8.0
Requires-Dist: pydantic==2.14.1
Requires-Dist: pydantic_core==2.50.1
Requires-Dist: starlette==1.9.0
Requires-Dist: typing-inspection==0.4.4
Dynamic: license-file

<div align="center">
  <picture>
    <source media="(max-width: 600px)" srcset="docs/assets/justfit-banner-mobile.svg">
    <img src="docs/assets/justfit-banner.svg" alt="JustFit — long-context LLM serving on Apple silicon" width="100%">
  </picture>

  [![Version](https://img.shields.io/badge/version-2.3.0-087f78)](CHANGELOG.md)
  [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-246a96)](LICENSE)
  [![Apple silicon](https://img.shields.io/badge/platform-Apple%20silicon-526b81)](#supported-setup)

  **[Get started](#get-started) · [Results](#measured-results) · [Documentation](#documentation) · [简体中文](README.zh-CN.md)**
</div>

JustFit is a local inference system for serving a **27B language-and-vision model on memory-constrained Macs**. It coordinates compressed context, a resident speculative draft, concurrent requests and CPU/GPU/ANE work under one unified-memory budget.

**Stable version: `2.3.0`.** This source tree prepares the stable release. Package-index and release-asset publication are pending; use the reviewed wheel and native runtime supplied with this release candidate. The immutable [RC24 preview](https://test.pypi.org/project/justfit-llm/2.3.0rc24/) remains available separately.

## Why JustFit

- **Keep long histories alongside the draft.** Warm-prefix tests reach a 320K aggregate page capacity across two live requests while retaining the speculative draft. This is a tested B2 bound, not a 320K single-request or cold-prefill guarantee.
- **Make concurrent work useful.** Bound transient workspaces, share compressed pages and preserve committed state across scheduling and cold restarts.
- **Put idle accelerators to work.** Use CPU-assisted prefill on M4 Pro, GPU A8 paths on M5 Pro/M6, and qualified ANE work when execution phase and memory allow it. Vision offload aims to leave GPU capacity for text; it does not guarantee lower image latency.
- **Serve through a familiar API.** OpenAI-compatible chat, images, streaming, structured output and native-enforced thinking budgets.

## Measured results

<a href="docs/assets/system-comparison.png"><img src="docs/assets/system-comparison.png" alt="Four-system comparison on M4 Pro, M5 Pro and M6: tested retained context, 16K prefill and 16K decode rates" width="100%"></a>

**Capacity and speed are separate experiments.** Solid bars show tested B1 bounds; hatching extends them to the B2 total. Both JustFit generations reach a 320K page capacity using their respective warm-prefix protocols; v2 retains its draft. Speed uses eight fixed SPEED-Bench 16K tasks with 512 forced output tokens. Weight formats, decoders and profiles differ. M4 Pro/M5 Pro have physical 24 GiB; M6 has physical 32 GiB, with 8 GiB locked only for the v1/v2 B2 capacity tests. M6 speed runs use physical 32 GiB.

At 16K, v2's median decode rate is **2.0–2.5× the v1 rate** across these configurations. Relative to upstream Splash 1.3.0, median prefill improves **9–46%**, while singleton decode declines **2–9%**. These whole-system results are not isolated mechanism gains or a chip-generation ranking.

[Protocols, exact rates and limitations →](docs/RESULTS.md)

## Get started

### Supported setup

- **Apple silicon and macOS 27.0+.** Reference configurations: M4 Pro, M5 Pro and M6, with at least 24 GiB. Other chip variants are not individually qualified.
- **Python 3.12 recommended**; the package declares support for Python 3.12–3.14.
- **Compatible Xcode/CoreAI SDK and Metal compiler toolchain** for this source preview. `justfit serve` finds installed tools; it does not install Xcode or the SDK.
- Internet access and storage for the pinned model pack: approximately **17.4 GB**, plus dependencies, build outputs and caches. The model repository is publicly accessible without a Hugging Face token at the checked revision.

Installing the Python package on Linux does not enable native inference there. The standalone GPU engine's macOS 26.4 source target is separate from this complete macOS 27/CoreAI setup.

### Install the reviewed release

Create a fresh environment. Install the reviewed wheel downloaded alongside this source tree:

```bash
python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install --index-url https://pypi.org/simple/ -r requirements-runtime.txt
python -m pip install ./justfit_llm-2.3.0-py3-none-any.whl
justfit verify
justfit doctor
justfit serve
```

First run: create a versioned workspace, build the native runtime, download and verify the pinned model, choose the device preset, then serve on **`http://127.0.0.1:8000`**. Later runs reuse the build and model. Ctrl+C stops the foreground server. GPU vision works without a compiled ANE vision profile.

If the Metal compiler is missing, install the component with `xcodebuild -downloadComponent MetalToolchain` after installing compatible Xcode. [Step-by-step setup and overrides](QUICKSTART.md) cover custom ports, local models and prebuilt-runtime inputs. Use the matching 2.3.0 runtime and published SHA256 to skip local compilation; see the linked instructions.

### Send a request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "incoai/Qwen3.8-27B-Splash",
    "messages": [{"role": "user", "content": "What is 17 multiplied by 19?"}],
    "max_tokens": 64,
    "reasoning_effort": "none"
  }'
```

The model ID retains the upstream runtime pack's name; the engine and CLI are JustFit. The default model is pinned to revision `9d27070b71f7142c6b6025f03ac011d70a73cb48`.

## How it works

| Mechanism | Responsibility |
| --- | --- |
| **KVExec** | Consume compressed history with bounded reconstruction and rotated-domain attention. |
| **PhaseSwap** | Control component and workspace lifetimes while the speculative draft stays resident. |
| **StateTrans** | Preserve committed KV, recurrent and draft state across scheduling and persistence. |
| **CrossFire** | Coordinate CPU, GPU and ANE work using phase ownership and memory admission. |

### ANE behavior

| Reference preset | Language prefill | ANE vision eligibility |
| --- | --- | --- |
| M4 Pro / `m4` | CPU-assisted GPU | Text prefill or decode |
| M5 Pro / `m5` | GPU A8 | Text prefill or decode |
| M6 / `m6` | GPU A8 + ANE FFN | Decode with no active text prefill |

Vision eligibility requires qualified 4K/8K assets, a matching chip and OS build, and memory admission. **4K/8K count image patches, not text tokens or screen pixels.** Idle requests, unsupported inputs and unavailable resources use GPU. M6 reserves ANE for language prefill before admitting vision work. [ANE setup and download interface](docs/ANE_DOWNLOADS.md).

## Documentation

| I want to… | Start here |
| --- | --- |
| Install, build or choose a port/model | [Quickstart](QUICKSTART.md) · [Python installation](docs/PYPI.md) |
| Understand the measurements | [Results and protocols](docs/RESULTS.md) · [Preview validation](docs/PREVIEW_VALIDATION.json) |
| Change device settings | [Configuration](docs/CONFIGURATION.md) · [ANE policy](ANE_POLICY.md) |
| Prepare ANE assets or a prebuilt runtime | [ANE downloads](docs/ANE_DOWNLOADS.md) · [Offline provisioning](docs/ANE_PROVISIONING.md) · [Runtime distribution](docs/RUNTIME_DISTRIBUTION.md) |
| Deploy a managed service | [Deployment and rollback](docs/DEPLOYMENT.md) · [Broker integration](service/broker-integration/README.md) |
| Control reasoning output | [Thinking budget](docs/THINKING_BUDGET.md) |
| Report a problem or contribute | [Issues](https://github.com/YuhuaBillChen/justfit-llm/issues) · [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) |

The default endpoint is local. Network deployment needs appropriate authentication and TLS; see [security scope](SECURITY.md). Native runtime sources are in `source/`, ANE tooling in `vision-tools/`, and optional managed-service components in `service/`.

## Paper and citation

**JustFit v2: Long-Context Local LLM Serving with Just-in-Time State Management** — Yuhua Chen, 2026. **[Read the preprint (PDF)](https://github.com/YuhuaBillChen/justfit-llm/blob/paper-v2-2026-10-23-r2/paper/2026-10-23/JustFit-v2.pdf)** · [Paper and citation files](paper/README.md). This dated version remains available as the manuscript evolves; an arXiv link will be added when public.

If JustFit supports your research, please cite:

```bibtex
@unpublished{chen2026justfitv2,
  author = {Chen, Yuhua},
  title = {{JustFit v2}: Long-Context Local LLM Serving with Just-in-Time State Management},
  year = {2026},
  month = oct,
  note = {Preprint, version 2026-10-23 revision 2},
  url = {https://github.com/YuhuaBillChen/justfit-llm/blob/paper-v2-2026-10-23-r2/paper/2026-10-23/JustFit-v2.pdf}
}
```

[Machine-readable citation](CITATION.cff) · [JustFit v1 paper](https://arxiv.org/abs/2609.17475)

## Acknowledgments and license

JustFit's native runtime is derived from **[Splash 1.3.0](https://github.com/incoai/splash)**. Upstream authors, licenses and implementation credits are preserved in [NOTICE](NOTICE) and [third-party notices](source/THIRD_PARTY_NOTICES). Model weights and derived assets retain their own licenses.

Code is available under [Apache 2.0](LICENSE). JustFit maintainers handle this project's support; please direct JustFit issues here rather than to upstream maintainers.
