Metadata-Version: 2.4
Name: wiola
Version: 0.1.1
Summary: Wiola: a novel decoder-only Small Language Model with Spiral Rotary Positional Encoding, Gated Cross-Layer Attention, Adaptive Token Merging, Dual-Stream Feed-Forward, and WiolaRMSNorm.
Author: OSCOWL-AI
License: Apache-2.0
Project-URL: Homepage, https://github.com/Wiola-OSCOWL-ai/wiola
Project-URL: Repository, https://github.com/Wiola-OSCOWL-ai/wiola
Project-URL: Model Hub, https://huggingface.co/oscowlai
Keywords: small-language-model,transformer,language-model,spiral-rotary,attention,pytorch,huggingface
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: torch>=2.1.0
Requires-Dist: transformers<5,>=4.46
Requires-Dist: tokenizers>=0.19.0
Requires-Dist: safetensors>=0.4.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: tqdm>=4.66.0
Provides-Extra: data
Requires-Dist: datasets>=2.18.0; extra == "data"
Provides-Extra: train
Requires-Dist: datasets>=2.18.0; extra == "train"
Requires-Dist: tensorboard>=2.15.0; extra == "train"
Provides-Extra: wandb
Requires-Dist: wandb>=0.16.0; extra == "wandb"
Provides-Extra: eval
Requires-Dist: lm-eval>=0.4.0; extra == "eval"
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: ruff>=0.4.0; extra == "dev"
Dynamic: license-file

<div align="center">

# Wiola

**A novel decoder-only Small Language Model built from first principles.**
<img width="3584" height="1184" alt="ai" src="https://github.com/user-attachments/assets/26bf4329-4a5d-4de8-a06e-f4188849de40" />

[![CI](https://github.com/Wiola-OSCOWL-ai/wiola/actions/workflows/ci.yml/badge.svg)](https://github.com/Wiola-OSCOWL-ai/wiola/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![Models on HF](https://img.shields.io/badge/%F0%9F%A4%97%20Hub-oscowlai-yellow)](https://huggingface.co/oscowlai)

</div>

Wiola is an experimental, fully open-source language model architecture that
introduces five independently novel components. It is implemented in PyTorch,
integrates natively with 🤗 Transformers (`AutoModelForCausalLM` via
`trust_remote_code=True`), ships in four sizes (120M / 360M / 700M / 1.5B), and
is covered by 22 unit tests.

> **Status — experimental research prototype.** The architecture is implemented,
> tested, and ready to train. Pre-trained weights and benchmark numbers are
> future work; any perplexity/scaling figures in the paper are *projections*,
> not measurements. See [Limitations](#limitations).

---

## Table of contents

- [The five novel components](#the-five-novel-components)
- [Model family](#model-family)
- [Installation](#installation)
- [Quickstart](#quickstart)
- [Training](#training)
- [Evaluation](#evaluation)
- [Inference examples](#inference-examples)
- [Publishing to the Hugging Face Hub](#publishing-to-the-hugging-face-hub)
- [Repository layout](#repository-layout)
- [Testing](#testing)
- [Limitations](#limitations)
- [Contributing](#contributing)
- [License & citation](#license--citation)

---

## The five novel components

| Component | What it does |
|-----------|--------------|
| **SRPE** — Spiral Rotary Positional Encoding | Places token positions on a 3D helical manifold with dual winding angles and a sinusoidal radial term, encoding multiple positional scales analytically (no extra parameters). |
| **GCLA** — Gated Cross-Layer Attention | GQA self-attention + SRPE, plus soft cross-attention to compressed summaries of preceding layers, blended by a learned scalar gate and a per-position sigmoid output gate. |
| **ATM** — Adaptive Token Merging | During *training only*, greedily merges semantically redundant adjacent tokens (cosine similarity > τ) in the middle third of layers, then exactly restores length. Disabled at inference. |
| **DSFF** — Dual-Stream Feed-Forward | Two parallel dense streams — a narrow SwiGLU stream and a wide GELU stream — fused by a learned per-dimension sigmoid gate. |
| **WiolaRMSNorm** | RMSNorm with a learned per-dimension offset `δ` added *before* normalisation; `δ = 0` recovers standard RMSNorm exactly. |

A full mathematical treatment is in [`docs/architecture.md`](docs/architecture.md)
and the accompanying paper.

---

## Model family

| Variant | `d` | Layers | Heads (`H`/`H_kv`) | Params | Config |
|---------|----:|-------:|:------------------:|-------:|--------|
| `wiola-120m` | 768  | 12 | 12 / 4 | ~120M | [`configs/wiola-120m.yaml`](configs/wiola-120m.yaml) |
| `wiola-360m` | 1024 | 16 | 16 / 4 | ~360M | [`configs/wiola-360m.yaml`](configs/wiola-360m.yaml) |
| `wiola-700m` | 1536 | 24 | 16 / 8 | ~700M | [`configs/wiola-700m.yaml`](configs/wiola-700m.yaml) |
| `wiola-1.5b` | 2048 | 28 | 16 / 8 | ~1.5B | [`configs/wiola-1.5b.yaml`](configs/wiola-1.5b.yaml) |

All variants use a 32,000-token byte-level BPE vocabulary and a 2,048-token
context window.

---

## Installation

Requires **Python 3.9+** and a recent PyTorch. On Windows, install PyTorch from
the [official selector](https://pytorch.org/get-started/locally/) first if you
need a specific CUDA build.

```bash
git clone https://github.com/Wiola-OSCOWL-ai/wiola.git
cd wiola

python -m venv .venv
# Windows (Git Bash):   source .venv/Scripts/activate
# Linux / macOS:        source .venv/bin/activate

pip install -e ".[train]"      # editable install + training extras
```

Verify the install:

```bash
python -c "from wiola import WiolaForCausalLM, WiolaConfig; \
print(WiolaForCausalLM(WiolaConfig.from_preset('wiola-120m')).num_parameters())"
```

---

## Quickstart

Build a model from a preset and run a forward pass — no download required:

```python
import torch
from wiola import WiolaConfig, WiolaForCausalLM

config = WiolaConfig.from_preset("wiola-120m")
model = WiolaForCausalLM(config).eval()

input_ids = torch.randint(0, config.vocab_size, (1, 16))
out = model(input_ids=input_ids)
print(out.logits.shape)          # torch.Size([1, 16, 32000])

generated = model.generate(input_ids, max_new_tokens=20, do_sample=False)
print(generated.shape)           # torch.Size([1, 36])
```

---

## Training

The training pipeline has **one source of truth for text**: either a local
corpus *or* a Hugging Face dataset (set exactly one in the config). If a
tokenizer is missing, it is **built automatically** instead of crashing.

### 1. (Optional) Build the tokenizer ahead of time

```bash
python scripts/prepare_tokenizer.py --config configs/wiola-120m.yaml
```

This step is optional — `train.py` will build it on demand if it is absent.

### 2. (Optional) Tokenize and pack the corpus

```bash
python scripts/prepare_data.py --config configs/wiola-120m.yaml
```

### 3. Train

```bash
python scripts/train.py --config configs/wiola-120m.yaml
```

Resume from a checkpoint:

```bash
python scripts/train.py --config configs/wiola-120m.yaml --resume runs/wiola-120m/latest
```

### Using your own text

Edit the `data` block of a config to point at local files and clear the HF
fields:

```yaml
data:
  local_path: ./data/my_corpus     # a file, a directory, or a glob
  hf_dataset: null
  hf_config: null
  block_size: 1024
  val_fraction: 0.01
```

### What gets logged

Each run tracks training loss, validation loss, perplexity, learning rate,
gradient norm, throughput (tokens/sec) and GPU memory. Set
`logging.backend` to `tensorboard` or `wandb` to stream them, or leave it as
`none` for stdout only. Checkpoints follow a `latest / best / step-N / final`
lifecycle, and each one stores the run config and random seed for
reproducibility. See [`docs/training.md`](docs/training.md) for the full
reference.

---

## Evaluation

Compute perplexity on a held-out split:

```bash
python scripts/evaluate.py --model runs/wiola-120m/best --split test
```

The script also prints the exact `lm-eval` harness commands for HellaSwag,
PIQA and ARC-Easy so you can reproduce standard benchmarks once you have
trained weights. See [`docs/evaluation.md`](docs/evaluation.md).

---

## Inference examples

```bash
python examples/generate.py --model runs/wiola-120m/best --prompt "The history of"
python examples/chat.py     --model runs/wiola-120m/best
python examples/batch.py    --model runs/wiola-120m/best
```

Loading a published checkpoint from the Hub (note `trust_remote_code=True`,
which is required because Wiola ships custom modeling code):

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "oscowlai/wiola-120m"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
```

---

## Publishing to the Hugging Face Hub

After training, push weights + tokenizer + custom code + model card in one step:

```bash
huggingface-cli login
python scripts/push_to_hub.py \
    --model runs/wiola-120m/best \
    --repo oscowlai/wiola-120m \
    --card model_cards/wiola-120m.md
```

A complete, copy-paste GitHub + Hugging Face walkthrough for Windows 10 /
VS Code / Git Bash lives in [`docs/publishing.md`](docs/publishing.md).

---

## Repository layout

```
wiola/
├── src/wiola/
│   ├── configuration_wiola.py      # WiolaConfig (+ from_preset)
│   ├── modeling_wiola.py           # WiolaModel, WiolaForCausalLM
│   ├── components/                 # SRPE, GCLA, ATM, DSFF, WiolaRMSNorm
│   ├── data/                       # one-source-of-truth data pipeline
│   └── utils.py
├── configs/                        # _base.yaml + 4 variant configs
├── scripts/                        # prepare_tokenizer/data, train, evaluate, push_to_hub
├── examples/                       # generate, chat, batch
├── tests/                          # 22 unit tests
├── docs/                           # architecture, training, evaluation, publishing
├── model_cards/                    # one card per variant (for the Hub)
└── .github/workflows/ci.yml        # lint + tests
```

---

## Testing

```bash
pip install -e ".[dev]"
pytest -q
```

All 22 tests should pass. They cover every component plus the assembled model,
including the incremental-decoding correctness check (full forward vs. cached
two-chunk forward agree to `l_inf < 1e-4`).

---

## Limitations

- ATM is disabled at inference to keep the KV-cache consistent.
- GCLA's layer-to-layer dependency complicates pipeline parallelism.
- SRPE's radial term may exhibit phase interference for very long contexts.
- **No pre-trained weights are released yet.** Projected perplexity and
  scaling figures are hypotheses until models are trained and evaluated.

---

## Contributing

Contributions are welcome — see [`CONTRIBUTING.md`](CONTRIBUTING.md) and the
issue / pull-request templates under [`.github/`](.github). Please run
`ruff check .` and `pytest` before opening a PR.

---

## License & citation

Released under the [Apache License 2.0](LICENSE).

```bibtex
@misc{wiola2025,
  title  = {The Wiola Architecture for Efficient Small Language Models},
  author = {Aryuemaan Chowdhury},
  year   = {2026},
  howpublished = {\url{https://github.com/Wiola-OSCOWL-ai/wiola}}
}
```
