Metadata-Version: 2.4
Name: fermion-research
Version: 0.1.12
Summary: Five-value sub-2-bit LLMs: chat with a ~2 GB 8B container at native-runtime speed, load it as a Transformers model, or serve it on an OpenAI-compatible endpoint
Author: Fermion Research
Maintainer: Fermion Research
License-Expression: Apache-2.0
Project-URL: Homepage, https://fermionresearch.com
Project-URL: Documentation, https://fermionresearch.com/learn
Project-URL: Repository, https://github.com/fermionresearch
Project-URL: Models, https://huggingface.co/fermionresearch/Neutrino-8B
Project-URL: Receipts, https://github.com/fermionresearch/kit
Project-URL: Issue Tracker, https://github.com/fermionresearch/kit/issues
Keywords: llm,quantization,ternary,sub-2-bit,bitnet,inference,transformers,openai-api,cpu-inference,edge-ai
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.3
Requires-Dist: transformers>=5.0
Requires-Dist: numpy>=1.26
Requires-Dist: huggingface_hub>=0.23
Dynamic: license-file

# fermion

Five-value sub-2-bit language models by Fermion Research. One CLI chats with
a ~2 GB 8B container (TRTC v4) or a 2.5 KB QR-code model — and serves either
one on an **OpenAI-compatible HTTP endpoint** (`fermion serve` — with native
OpenAI tool calling, token streaming, and a persistent session runtime that
reuses the KV cache across turns),
so it drops into Open WebUI, Continue, LangChain, LlamaIndex or any agent
harness that speaks `/v1/chat/completions`.

> **Tool calling is an 8B capability.** `serve` accepts and injects tool
> schemas for every SKU, but only `Neutrino-8B` reliably emits `tool_calls`.
> The 0.6B models answer in prose instead — measured, and consistent with our
> banked finding that function calling has a parameter-count floor. Point
> agent frameworks at the 8B.

> NAME FINAL 2026-07-24: `fermion` is the final pip/CLI name (rename chain
> from the pre-launch working slug documented in RELEASE_RUNBOOK.md §5).
> HF org `fermionresearch`, GitHub org `fermionresearch`. Model family =
> **Neutrino** (2026-07-24 launch-shape addendum): the CLI's default model
> is the one published SKU `fermionresearch/Neutrino-8B`, whose repo also
> bundles the prebuilt native `fermion-run` binaries under `bin/`.

## Quickstart (2 commands)

> **Recent releases in one breath** — 0.1.6: batched prefill (4-5x) and
> download progress. 0.1.7: OpenAI tool calling on `serve`, persistent
> session runtime (agent turns up to 72x faster), tolerant tool-call
> parsing. 0.1.8: multi-conversation KV cache (agent frameworks no longer
> evict your session) and tool-name aliasing for MCP-style clients.
> 0.1.9: the 8B now downloads as a 2.56 GB coded transport and unpacks
> locally (sha-verified against the manifest), and `serve` streams prose
> even on tool-calling requests. 0.1.10: fp16 KV cache by default (halves
> KV memory; `--kv-dtype`) and experimental YaRN long context
> (`--yarn-factor`) on the native runtime. 0.1.11: free-disk preflight
> before every model download, full-disk errors that name the directory
> that filled, `FERMION_CACHE_DIR` to relocate the model cache, and the
> transport is deleted after its sha-verified decode (8B cache footprint
> ~6.4 GB → ~3.9 GB). 0.1.12: `fermion inspect` reports the int8 KV column
> (it shipped in 0.1.10 but the table only showed fp16/fp32), plus honest
> scoping on `--yarn-factor` and on which SKU emits tool calls.

> **`--yarn-factor`: leave it off unless you need >40,960 tokens.** YaRN
> extends *addressable positions*; it does not improve recall. It also
> applies a global attention-temperature term (`0.1·ln(factor)+1`) that is
> not gated by position, so it changes output at **every** length, short
> prompts included. Our needle probe measures retrieval falling off well
> inside the native window with YaRN off entirely, so the ceiling is the
> model, not the flag.

```
pip install fermion-research
fermion chat                # downloads fermionresearch/Neutrino-8B, opens REPL
```

That first run fetches **3.89 GB** — the container, the tokenizer, the config
and the native runner — and nothing else. The model repo also carries a GGUF
build and a coded transport that this CLI never opens; they are filtered out.
(`FERMION_DOWNLOAD_ALL=1` fetches the whole repo if you want them.)

### Disk space: what the quickstart needs, and *where*

"No space left on device" with hundreds of GB free almost always means the
free space is on a **different volume** than the one being written. Everything
the quickstart writes goes to directories on your root/home volume unless you
say otherwise:

- **Linux:** `pip install fermion-research` pulls torch's CUDA wheel stack —
  **~8 GB** for the install, and pip stages **~3 GB of it in `/tmp`** (often a
  RAM-backed tmpfs). The 8B model itself is **~4 GB** in
  `~/.cache/huggingface`. CPU-only box? Install torch from the CPU index
  first — ~1 GB instead of ~7.5 GB, and the `/tmp` trap disappears:
  `pip install torch --index-url https://download.pytorch.org/whl/cpu`
- **Apple Silicon:** the pip install is small, but the first 8B fetch needs
  **~6.5 GB** free in the cache volume during the one-time transport decode
  (2.6 GB download + 3.9 GB decoded container, both on disk for a moment).
  Once the decoded container passes its sha256 check the transport is
  deleted, so the steady-state footprint is **~3.9 GB**
  (`FERMION_KEEP_TRANSPORT=1` keeps the transport instead).

Point the caches at the big disk and both traps disappear:

```bash
export HF_HOME=/big/disk/hf              # the model cache (the standard HF knob)
export FERMION_CACHE_DIR=/big/disk/fermion   # or: this CLI's model downloads only
export TMPDIR=/big/disk/tmp              # pip's staging area, if `pip install` is what failed
```

Since 0.1.11 the CLI **checks free space in the cache directory before
downloading** and prints the directory, the bytes needed and the bytes free
when the model will not fit; a disk that fills mid-download or mid-decode gets
the same named-path message instead of a traceback.

With a local file (no download):

```
fermion chat --model /path/to/neutrino-8b_v4.bin         # 8B container
fermion chat --model /path/to/qr_chatmax_artifact.bin    # the QR-code model
fermion generate "hello" --model ... --max-new 32        # one-shot
fermion info --model ...                                 # header + integrity check
fermion serve --model ...                                # OpenAI-compatible API
```

### Sampling defaults: chat and serve are sampled, generate is deterministic

`fermion chat` and `fermion serve` default to the **graded shipping config** —
temperature 0.01, top-p 1.0, repetition penalty 1.05 over a 256-token window —
because that is the configuration the conversational surfaces were graded at.
`fermion generate` defaults to **deterministic greedy** (temperature 0, no
penalty) because it is the scriptable, pipeable path that our receipts,
`fermion verify` and the token-identity gates depend on reproducing. Every
knob is a flag on all three, so either default is one argument away.

### `fermion info` — did my download actually work?

`info` prints the container header **and proves the file is the whole file**:
it checks the length against the record structure the container itself
describes, and against the `sha256` the model repo's `MANIFEST.json` pins.
It exits non-zero on a truncated or altered container, so it is safe to use
in a script:

```
$ fermion info --model ./Neutrino-8B
./Neutrino-8B/neutrino-8b_v4.bin: TRTC v4 arch=3 layers=36 hidden=4096 vocab=151936 (3.61 GiB)
[fermion] integrity OK: 3,875,404,812 bytes · length matches its own record
          structure · length matches MANIFEST.json · sha256 matches MANIFEST.json

$ fermion info --model ./half-downloaded.bin ; echo $?
./half-downloaded.bin: INTEGRITY CHECK FAILED
  container is truncated: input embedding weights needs bytes up to 622,329,932
  but the file is only 67,108,864 bytes
1
```

`--no-checksum` skips the hash (the length checks always run).

## `fermion serve` — OpenAI-compatible endpoint

```
fermion serve                                  # 127.0.0.1:8000, default model
fermion serve --model /path/to/neutrino-8b_v4.bin --port 8000
```

```
POST /v1/chat/completions    messages, temperature, top_p, max_tokens, stop, stream
POST /v1/completions         plain-text completion for older clients
GET  /v1/models              the loaded model id
GET  /health                 ok + the loaded container's sha256
```

Streaming is real SSE (`data: {...}\n\n` chunks with `choices[].delta`,
terminated by `data: [DONE]`); non-streaming returns `choices[].message`
plus a `usage` block. Errors come back OpenAI-shaped (`{"error": {...}}`).

**Point any OpenAI client at it.** Python:

```python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
    model="neutrino-8b_v4.bin",
    messages=[{"role": "user", "content": "What is 2+2?"}],
).choices[0].message.content)
```

**Open WebUI** — Settings → Connections → OpenAI API:

```
Base URL: http://127.0.0.1:8000/v1
API key:  not-needed          # any non-empty string
```

(Open WebUI in Docker: use `http://host.docker.internal:8000/v1`.)

**Continue** (`~/.continue/config.json`):

```json
{"models": [{"title": "Neutrino 8B", "provider": "openai",
             "model": "neutrino-8b_v4.bin", "apiKey": "not-needed",
             "apiBase": "http://127.0.0.1:8000/v1"}]}
```

**curl:**

```
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is 2+2?"}],"max_tokens":32}'
```

Notes:

- **Localhost only by default.** The server is unauthenticated; pass
  `--api-key SECRET` to require `Authorization: Bearer SECRET`, and expect a
  printed warning if you bind a non-loopback `--host`. `--cors` adds
  `Access-Control-Allow-Origin: *` for browser-side clients (off by default).
- Sampler defaults are the **graded shipping config** (same as `fermion chat`,
  and unlike `fermion generate`, which stays greedy — see above): temperature
  0.01, top-p 1.0, repetition penalty 1.05 over a 256-token window. A client that
  asks for `temperature: 0` gets a deterministic argmax **with the repetition
  penalty still applied** on either backend — the C runtime no-ops its sampler
  flags at temp 0, so the native path emulates argmax on top of a live
  sampler rather than dropping the penalty. Send the vLLM-style
  `"repetition_penalty": 1.0` extension to turn the penalty off.
- One request at a time (single resident model, serialised behind a lock);
  `n > 1`, embeddings and function-calling are not implemented and say so.
- Chat templating is the same code path as `fermion chat`, and served output
  is gated token-identical to `fermion generate` at matched settings.
- `--draft PATH` attaches a second container as a speculative-decoding draft
  model (assisted generation; the draft must share the tokenizer).

## What backs it

### Two decode backends, and the CLI tells you which one you got

Since **0.1.4** the CLI runs the prebuilt native runtime by default.

| backend | what it is | when it is used |
|---|---|---|
| `native` | `bin/fermion-run-<platform>`, the C runtime downloaded with the model — the path every published tokens/s number was measured on | automatically, whenever a binary exists for your platform and `--device cpu` |
| `torch` | the `hf_ternary` reference loader + `transformers.generate` | everywhere else, and whenever you ask for it |

```
fermion info --model ...       # prints the active backend and why
fermion generate --backend torch ...   # force the reference path
fermion generate --backend native ...  # fail loudly instead of running slow
FERMION_THREADS=8 fermion chat ...     # override the thread count
```

`fermion serve` reports the same thing in `GET /health` as `"backend"`.

**Native runtimes exist for macOS arm64 and Linux x86-64 only.** On Windows,
Linux arm64, or with `--device cuda/mps`, you get the torch path: correct, and
one to two orders of magnitude slower. The published speed figures are native
figures and do not describe the torch path.

Two known differences on the native path, both deliberate and documented in
`fermion/native.py`:

- The C runtime disables its whole sampler at `--temp 0`, so a greedy request
  that also carries a repetition penalty is mapped to
  `--temp 0.01 --min-p 0.999 --seed 0` — argmax by construction, penalty
  intact, still byte-reproducible.
- Greedy output is token-identical to the torch reference **at float32**, not
  at the CLI's bfloat16 default, and identity is a near-tie property rather
  than a guarantee: two independent implementations pick different tokens when
  the top-2 logits are within measurement noise. Measured agreement and the
  divergence analysis are in `handoffs/results_pip_native/RESULTS.md`.

## What backs it

- **TRTC v4 containers** load through the `hf_ternary` integration (vendored
  verbatim, sha-recorded): native `Qwen3ForCausalLM` etc. with packed
  five-value planes resident (98.9% memory honesty), correctness-gated at
  0 greedy mismatches over 768 tokens vs the expander reference.
- The torch path is the honest **reference**: correct everywhere torch runs,
  fast nowhere. It is what `fermion verify` and `--draft` (speculative
  decoding is a torch-graph feature) use, and what the native path is gated
  against. Measured numbers live in the model card and eval-receipts, each
  with venue+version+date.
- CLI activation dtype defaults to **bfloat16** (`--dtype` to override):
  fp16 NaN-overflows at 8B scale (measured; receipts in
  `packaging/acceptance/`), while `--dtype float16` reproduces the
  fp16-identity receipts on the small twin containers gated that way.
- **Tiny models** (QR 2.5 KB GRU / GIF 170 KB transformer) run through the
  float64 reference decoder the browser demo is bit-exactness-gated against.

```python
import fermion
from transformers import AutoModelForCausalLM
fermion.write_transformers_config("neutrino-8b_v4.bin", "cfg-dir")
model = AutoModelForCausalLM.from_pretrained("cfg-dir")   # native Qwen3
```

## Loading in plain `transformers`

```bash
hf download fermionresearch/Neutrino-8B --local-dir Neutrino-8B \
    --exclude "gguf/*" --exclude "*.tv4z"
```

```python
import fermion          # <-- REQUIRED, and it must come first
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Neutrino-8B")
tokenizer = AutoTokenizer.from_pretrained("Neutrino-8B")
```

`import fermion` is what registers the `trtc_v4` model type with the
Transformers Auto classes. Pass a **local directory**, not a Hub id.

### If you see this error, you forgot the import

```
ValueError: The checkpoint you are trying to load has model type `trtc_v4`
but Transformers does not recognize this architecture. ... You can update
Transformers with the command `pip install --upgrade transformers`.
```

**Ignore that advice.** Upgrading Transformers will never help and neither
will installing it from source — `trtc_v4` is our model type, registered at
import time by this package, so no Transformers release will ever know it.
`trust_remote_code=True` does not help either: these repos carry no
`auto_map`. Add `import fermion` above the Transformers import and it loads.

## Dev

```
pip wheel --no-deps -w dist .     # build the wheel
python -m venv /tmp/v && /tmp/v/bin/pip install dist/*.whl
/tmp/v/bin/fermion --version
```

License: Apache-2.0 (flagship is a Qwen3-8B derivative, Apache-2.0 upstream).
