Metadata-Version: 2.4
Name: nenya-mac
Version: 1.0.0
Summary: Metal engine for Nenya-2.5B-A0.5B on Apple silicon
License-Expression: Apache-2.0
Project-URL: Homepage, https://huggingface.co/ArsalanZ/Nenya-2.5B-A0.5B-Base-Metal-8bit
Project-URL: Model, https://huggingface.co/ArsalanZ/Nenya-2.5B-A0.5B-Base
Project-URL: Technical report, https://www.arsalanzafar.ai/nenya-technical-report
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy
Requires-Dist: huggingface_hub
Provides-Extra: convert
Requires-Dist: torch; extra == "convert"
Requires-Dist: safetensors; extra == "convert"
Provides-Extra: test
Requires-Dist: torch; extra == "test"
Requires-Dist: transformers; extra == "test"
Requires-Dist: safetensors; extra == "test"
Dynamic: license-file

# nenya-mac

<div align="center">
  <a href="https://www.arsalanzafar.ai/nenya-technical-report"><img alt="Technical report" src="https://img.shields.io/badge/%F0%9F%93%84%20Technical%20Report-Nenya-1783ff"/></a>
  <a href="https://huggingface.co/ArsalanZ/Nenya-2.5B-A0.5B-Base"><img alt="Hugging Face: the model" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Nenya--2.5B--A0.5B--Base-ffc107"/></a>
  <a href="https://huggingface.co/ArsalanZ/Nenya-2.5B-A0.5B-Base-Metal-8bit"><img alt="Hugging Face: the engine's weights" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Metal%208--bit-ffc107"/></a>
</div>
<div align="center">
  <a href="https://huggingface.co/ArsalanZ/Nenya-2.5B-A0.5B-Base-Metal-8bit/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-f5de53"/></a>
</div>

## Intro

Nenya is a general purpose 0.5B active, 2.5B total parameter mixture-of-experts (MoE) base model.
Nenya rivals Qwen3.5 2B on pretraining benchmarks with 500× lower training compute and 3.4× fewer active FLOPs/token.

This package is Nenya's Metal engine for Apple silicon. With it installed, `nenya.generate()` runs the model on the
Mac's GPU, from 8-bit weights in a single 3.47 GB file hosted in
[ArsalanZ/Nenya-2.5B-A0.5B-Base-Metal-8bit](https://huggingface.co/ArsalanZ/Nenya-2.5B-A0.5B-Base-Metal-8bit).

## Quick start

```bash
pip install "nenya[mac]"
```

```python
import nenya
print(nenya.generate("The mitochondrion is the organelle responsible for", max_new_tokens=40))
print(nenya.generate("Question: What is the capital of Australia?\nAnswer:", max_new_tokens=8, temperature=0))
```

`pip install "nenya[mac]"` installs two packages: [nenya](https://pypi.org/project/nenya/), which provides
`nenya.generate()`, and this engine, which it runs on. Installing `nenya-mac` on its own gives you only the engine
(see "Direct use" below).

## Benchmarks

![Benchmark accuracy of Nenya-2.5B-A0.5B and five small base models on 12 lm-eval tasks](https://huggingface.co/ArsalanZ/Nenya-2.5B-A0.5B-Base-Metal-8bit/resolve/main/accuracy_bars.png)

## Speed and memory on a Mac

One sequence at a time, in the Metal engine.

|**Decoding speed (tokens/s)** | Apple M4 base | Apple M1 base |
|---|---|---|
| Empty context | **127.1** | **74.3** |
| 1K tokens | 123.9 | 72.5 |
| 4K tokens | 114.1 | 67.6 |
| 16K tokens | 92.5 | 55.5 |

| **Peak memory while decoding** | Apple M4 base | Apple M1 base |
|---|---|---|
| At 1K tokens | 4.10 GB | 4.07 GB |
| At 16K tokens | 4.30 GB | 4.26 GB |
| Without the weight file (1K / 16K) | 0.67 / 0.86 GB | 0.54 / 0.73 GB |


**Prompt processing and time to first token (Apple M4).** Prompts run through every layer at once, in chunks of 512
tokens:

| Prompt length (tokens) | 16 | 64 | 256 | 1,024 | 4,096 | 16,384 |
|---|---|---|---|---|---|---|
| Time to first token | 0.04 s | 0.06 s | 0.15 s | 0.57 s | 2.7 s | 18.6 s |
| Prompt processing (tokens/s) | 392 | 1,129 | 1,720 | 1,801 | 1,510 | 909 |

## Usage notes

- **Download:** the first call downloads the weight file (3.47 GB) from
  [ArsalanZ/Nenya-2.5B-A0.5B-Base-Metal-8bit](https://huggingface.co/ArsalanZ/Nenya-2.5B-A0.5B-Base-Metal-8bit) and
  caches it with `huggingface_hub`. The engine stays loaded between calls; `nenya.unload()` frees it.
- **Sampling:** the default is temperature 0.9 with top-k 20, the model's recommended setting for open-ended text.
  Use `temperature=0` (greedy) for maths, code and short factual answers. `top_p` and `min_p` are supported too.
- **Backend:** `nenya.generate()` uses the engine automatically. A call that passes transformers' own `generate()`
  arguments (e.g. `repetition_penalty`) runs on transformers instead, with a warning. `backend="mac"` or
  `backend="transformers"` picks one.
- **Requirements:** a Mac with Apple silicon (M1 or later) and Python 3.12. macOS 11+. The wheel ships the compiled
  engine, so no Xcode tools are needed.

Direct use, without the text wrapper:

```python
import nenya_mac
eng = nenya_mac.Engine(nenya_mac.default_weights())   # one decoding session (up to 16,384 tokens)
eng.append(prompt_ids)                                  # the prompt: batched through every layer
logits = eng.logits()                                   # next-token logits, numpy float32 [vocab]
new_ids = eng.generate([], 40, temperature=0.9, top_k=20, seed=1, stop=[100257])
eng.reset()
```

**Your own fine-tune**, saved in the release's format, runs on the engine after a conversion:
`pip install "nenya-mac[convert]"`, then `python -m nenya_mac.convert <folder or repo id> my.oe`, and set
`NENYA_MAC_WEIGHTS=my.oe`. Generating from another repo without converted weights falls back to transformers.

## The weight file

The file holds 8-bit matrices in groups of 32, a 4-bit screening copy of the output head with exact fp16
re-scoring of the candidates, the mHC boundaries folded into the output projections, and fp16 Engram tables.

## License

Apache 2.0, like the model (`LICENSE`, `NOTICE`).
