Metadata-Version: 2.4
Name: qorva
Version: 0.1.0
Summary: Hybrid Mamba-2 + Attention + MoE language models, easy to load and run.
Author-email: Your Name <you@example.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/yourusername/qorva
Project-URL: Repository, https://github.com/yourusername/qorva
Project-URL: Issues, https://github.com/yourusername/qorva/issues
Keywords: language-model,mamba,attention,mixture-of-experts,transformer,pytorch
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.1.0
Requires-Dist: tiktoken>=0.5.0
Requires-Dist: huggingface_hub>=0.20.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: numpy>=1.24.0
Provides-Extra: train
Requires-Dist: datasets>=2.16.0; extra == "train"
Requires-Dist: tqdm>=4.66.0; extra == "train"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Dynamic: license-file

# qorva

Hybrid **Mamba-2 + Attention + Mixture-of-Experts** language models — easy to load, run, and fine-tune.

```bash
pip install qorva
```

## Quick start

```python
from qorva import load_pretrained, generate

model = load_pretrained("your_username/qorva-model-fast")
print(generate(model, "Once upon a time in a small village,", max_new_tokens=100))
```

Or, using the bound method directly:

```python
from qorva import load_pretrained

model = load_pretrained("your_username/qorva-model-fast")
print(model.generate("Once upon a time", max_new_tokens=100, temperature=0.8))
```

## Command line

```bash
qorva-generate --model your_username/qorva-model-fast --prompt "Once upon a time"
```

## Build from scratch

```python
from qorva import QorvaModel, QorvaConfig

cfg   = QorvaConfig.nano_full()      # ~370M, Mamba + Attention + MoE
model = QorvaModel(cfg)
print(f"{model.num_parameters()/1e6:.1f}M parameters")
```

Available presets:
- `QorvaConfig.micro()` — ~32M, for fast experimentation
- `QorvaConfig.nano_full()` — ~370M, Mamba + Attention + MoE
- `QorvaConfig.nano_baseline()` — ~370M, Attention-only baseline

## Push your trained model to HuggingFace

```python
from qorva import push_to_hub

push_to_hub(model, "your_username/qorva-nano-370m", token="hf_...")
```

## Perplexity evaluation

```python
from qorva.utils.generation import compute_perplexity

ppl = compute_perplexity(model, "Some evaluation text here...")
print(f"PPL: {ppl:.2f}")
```

## Architecture

Each layer mixes two branches with learnable weights:

```
x' = x + alpha * Mamba2(LN(x)) + beta * GQAttention(LN(x))
x  = x' + LatentMoE(LN(x'))      # stage="full" only
```

| Stage | Branches |
|---|---|
| `attn_only` | Attention + FFN |
| `mamba_only` | Mamba + FFN |
| `hybrid` | Mamba + Attention + FFN |
| `full` | Mamba + Attention + LatentMoE |

## License

Apache-2.0
