Metadata-Version: 2.4
Name: apeiron_lm
Version: 0.0.3.2
Summary: Byte-native parallel language model with unified latent reasoning
Author: Ömür Bera Işık
License: Apache-2.0
Project-URL: Homepage, https://github.com/yourusername/apeiron_lm
Project-URL: Repository, https://github.com/yourusername/apeiron_lm
Project-URL: Issues, https://github.com/yourusername/apeiron_lm/issues
Keywords: nlp,language-model,byte-level,parallel-decoding,deep-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0.0
Requires-Dist: numpy>=1.24.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Provides-Extra: hf
Requires-Dist: transformers>=4.35.0; extra == "hf"
Requires-Dist: datasets>=2.14.0; extra == "hf"
Requires-Dist: huggingface_hub>=0.20.0; extra == "hf"
Provides-Extra: training
Requires-Dist: tensorboard>=2.14; extra == "training"
Requires-Dist: tqdm>=4.66; extra == "training"
Provides-Extra: wandb
Requires-Dist: wandb>=0.16; extra == "wandb"
Provides-Extra: all
Requires-Dist: transformers>=4.35.0; extra == "all"
Requires-Dist: datasets>=2.14.0; extra == "all"
Requires-Dist: huggingface_hub>=0.20.0; extra == "all"
Requires-Dist: tensorboard>=2.14; extra == "all"
Requires-Dist: tqdm>=4.66; extra == "all"
Requires-Dist: wandb>=0.16; extra == "all"
Dynamic: license-file

# Apeiron LM
**v0.0.3.2** — Byte-native parallel language model. Gradient-free SDPS training, Fourier global context, LoRA, multi-GPU, Flash Attention.

> No tokenizer. No autoregressive loop. No vocabulary.

---

## What is Apeiron LM?

Apeiron operates entirely at the byte level. Raw bytes (text, image, audio, video) are encoded into a continuous latent vector and decoded in a single **parallel** forward pass — no left-to-right loop, no tokenizer, no vocabulary.

### Architecture

```
Input bytes  [0…255]
  → ByteEncoder          Transformer + mean-pool  →  v ∈ ℝᶜ
  → NoveltyEstimator     MLP gate                 →  novelty ∈ [0,1]
  → LatentExplorer       Prototype bank + proj    →  c* ∈ ℝᶜ
  → UnifiedLatentReasoner
        ├─ LengthPredictor    MLP → N_pred
        ├─ ContextBlock       (one of):
        │     none        CausalDWConv1D           O(N·k·C)
        │     replace     FourierSpectralConv1D    O(N·log(N)·C)
        │     parallel    Fourier + DWConv, gated  O(N·log(N)·C)
        │     adaptive    AdaptiveFourierFilter    O(N·log(N)·C) + O(C²)
        │     multiscale  MultiScaleFourierBlock   O(3N·log(N)·C)
        └─ ByteProjector  → N×256 byte logits
  → Output bytes
```

**Hard invariants (never broken):**
- Full parallelism — no sequential loop in forward or generation
- Byte-native — no tokenizer anywhere
- Output vocab = exactly 256
- Infinite effective context (Fourier modes)

---

## Installation

```bash
pip install -e .

# Optional extras
pip install transformers          # model conversion from GPT-2, BERT, etc.
pip install Pillow                # image I/O
pip install flash-attn --no-build-isolation  # Flash Attention 2
pip install tensorboard wandb     # experiment tracking
```

---

## Quick Start

```python
from apeiron_lm import ApeironLM, ApeironConfig

model = ApeironLM(ApeironConfig.small())
print(model.generate_text("Hello", max_generate=64))
```

---

## Training

### Option A — SDPS (gradient-free, seconds)

```python
from apeiron_lm import ApeironLM, ApeironConfig
from apeiron_lm.sdps import sdps_fit

model  = ApeironLM(ApeironConfig.small())
result = sdps_fit(model, data_tensor)   # (N, seq_len) LongTensor
print(result)
# SDPSResult(
#   elapsed_seconds=1.247,
#   n_samples=1024,
#   n_layers=4,
#   covariance_rank=128,
#   per_layer_residual_trace=['5.2134', '3.8901', '2.1244', '0.9912'],
#   per_head_reconstruction_error=
#     layer 0: ['0.142', '0.138', '0.151', '0.139']
#     ...
#   refine_steps_done=0,
# )
```

**SDPS with gradient refinement** (best of both worlds):

```python
result = sdps_fit(
    model, data_tensor,
    refine_steps = 200,    # gradient steps on byte_proj + length_mlp only
    refine_lr    = 1e-4,   # encoder stays frozen
)
```

**Stream from DataLoader:**

```python
result = sdps_fit(model, train_loader, max_batches=500)
```

**Supervised output head (closed-form):**

```python
sdps   = SDPSTrainer(model)
sdps.fit(data_tensor)
W_out  = sdps.fit_output_head(X, y_labels, n_classes=10)
```

**What SDPS does (v0.0.3.2 — all 8 quality gaps closed):**
1. Captures encoder activations via forward hooks
2. Per-head covariance Σ_h for each attention head
3. Solves W_Q/K/V/O analytically per head (exact, not tiled)
4. FFN weights via target-propagation + ridge regression (not just eigenvectors)
5. Byte embedding seeded with sinusoidal encoding of byte values
6. byte_proj fitted via ridge regression (latent → byte histogram)
7. LatentExplorer prototypes seeded via k-means++ on training latents
8. length_mlp final layer fitted via ridge regression

### Option B — Gradient training

```python
from apeiron_lm import ApeironTrainer, TrainingArguments

args    = TrainingArguments(output_dir="./runs", num_epochs=10, batch_size=32)
trainer = ApeironTrainer(model, dataset, args)
trainer.train()
```

### Option C — SDPS warm-start → gradient fine-tune

```python
sdps_fit(model, data_tensor)      # analytical warm-start
trainer = ApeironTrainer(model, dataset, args)
trainer.train()                   # gradient refinement from SDPS weights
```

---

## Fine-Tuning

### Full fine-tuning

```python
from apeiron_lm.finetune import finetune

result = finetune(
    model          = ApeironLM.load("./pretrained"),
    train_data     = dataset,
    output_dir     = "./finetuned",
    num_epochs     = 3,
    learning_rate  = 1e-5,
    layer_lr_decay = 0.9,      # encoder layers get progressively smaller LR
    freeze_encoder = False,
)
```

### LoRA fine-tuning (parameter-efficient)

```python
from apeiron_lm.finetune import ApeironFinetuner, FinetuneConfig

cfg = FinetuneConfig(
    output_dir          = "./lora_finetuned",
    num_epochs          = 3,
    learning_rate       = 2e-4,
    use_lora            = True,
    lora_r              = 8,
    lora_alpha          = 16.0,
    lora_dropout        = 0.05,
    lora_target_modules = ["in_proj", "out_proj", "linear1", "linear2"],
    lora_merge_after    = False,   # True = merge LoRA into weights after training
)
finetuner = ApeironFinetuner(model, dataset, cfg)
result    = finetuner.train()
```

### LoRA manual API

```python
from apeiron_lm.lora import (
    LoRAConfig, apply_lora, merge_lora, unmerge_lora,
    save_lora, load_lora, lora_parameter_count,
)

# Apply LoRA
lora_cfg = LoRAConfig(r=8, alpha=16, target_modules=["linear1", "linear2"])
apply_lora(model, lora_cfg)
trainable, total = lora_parameter_count(model)
print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.2f}%)")

# Train (only LoRA A, B matrices update)
trainer = ApeironTrainer(model, dataset, args)
trainer.train()

# Save only LoRA weights (~KB file)
save_lora(model, "./lora.pt")

# Load into a fresh model
model2 = ApeironLM.load("./pretrained")
apply_lora(model2, lora_cfg)
load_lora(model2, "./lora.pt")

# Merge for inference (no LoRA overhead)
merge_lora(model)
```

---

## Fourier Global Context

```python
from apeiron_lm import ApeironConfig, ApeironLM

cfg = ApeironConfig.base()

# Standard Fourier (infinite context, O(N log N C))
cfg.fourier_mode = "replace"

# Fourier + local DWConv in parallel (recommended)
cfg.fourier_mode = "parallel"

# Input-conditioned frequency filter (v0.0.3.2)
cfg.fourier_mode = "adaptive"

# Multi-resolution: coarse/mid/fine (v0.0.3.2, default for large)
cfg.fourier_mode = "multiscale"

# Channel-mixing (full C×C complex matrix per mode, v0.0.3.2)
cfg.fourier_mix_channels = True   # more expressive, use for small C

cfg.fourier_n_modes = 64   # number of frequency modes

model = ApeironLM(cfg)
```

| Preset | fourier_mode  | n_modes |
|--------|--------------|---------|
| tiny   | none         | 16      |
| small  | none         | 32      |
| base   | parallel     | 64      |
| large  | multiscale   | 128     |

---

## Attention Backends

```python
from apeiron_lm.attention import (
    set_attention_backend,
    current_attention_backend,
    attention_diagnostics,
)

# Auto-select best available (Flash > SDPA > naive)
set_attention_backend("auto")

# Force a specific backend
set_attention_backend("flash")   # requires: pip install flash-attn
set_attention_backend("sdpa")    # PyTorch built-in
set_attention_backend("naive")   # always works

print(current_attention_backend())  # "naive" / "sdpa" / "flash"
print(attention_diagnostics())

# Or via config
cfg = ApeironConfig.base()
cfg.attention_backend = "sdpa"
model = ApeironLM(cfg)   # wires backend on init
```

---

## Multi-GPU Training

### DataParallel (single machine, simplest)

```python
from apeiron_lm.parallel import wrap_data_parallel
from apeiron_lm import ApeironTrainer, TrainingArguments

model   = ApeironLM(cfg)
model   = wrap_data_parallel(model)   # wraps if >1 GPU
trainer = ApeironTrainer(model, dataset, args)
trainer.train()
```

### DDP (recommended for multi-GPU)

```bash
torchrun --nproc_per_node=4 train.py
```

```python
# train.py
from apeiron_lm.parallel import DDPContext, ApeironDDPTrainer

ctx     = DDPContext.detect()
model   = ApeironLM(cfg).to(ctx.device)
trainer = ApeironDDPTrainer(model, dataset, args, ctx=ctx)
trainer.train()
trainer.save("./output")   # rank 0 only
```

### FSDP (large models)

```python
from apeiron_lm.parallel import wrap_fsdp

model = ApeironLM(cfg)
model = wrap_fsdp(model, mixed_precision=True)
```

---

## Model Conversion from Transformers

```bash
# Convert GPT-2 → Apeiron
apeiron-convert --from gpt2 --output ./my_model

# More options
apeiron-convert \
  --from bert-base-uncased \
  --output ./bert_model \
  --preset base \
  --steps 2000 \
  --verbose
```

```python
from apeiron_lm.convert import ModelConverter, ConversionConfig

cfg    = ConversionConfig(
    teacher_name    = "gpt2",
    output_dir      = "./converted",
    preset          = "small",
    distill_steps   = 1000,
    teacher_layer   = -1,        # which teacher layer to distil (-1 = last)
    use_sdps_warmup = True,      # SDPS warm-start after distillation
)
result = ModelConverter(cfg).convert()
model  = ApeironLM.load(result["saved_to"])
```

Requires: `pip install transformers`

---

## Multimodal

```python
from apeiron_lm.multimodal import ApeironMultimodal, build_multimodal_input
from PIL import Image

mm = ApeironMultimodal(model)

# Text + image → text
result = mm.generate(
    text            = "Describe this image:",
    image           = Image.open("cat.png"),
    output_modality = "text",
    max_generate    = 128,
)
print(result["text"])

# Build raw multimodal byte tensor
inp = build_multimodal_input(
    text   = "Caption:",
    image  = Image.open("photo.jpg"),
)
```

Requires: `pip install Pillow`

---

## Save / Load

```python
model.save("./my_model")
model = ApeironLM.load("./my_model")
```

---

## Configuration Reference

```python
cfg = ApeironConfig(
    # Core
    latent_dim           = 128,
    encoder_layers       = 4,
    encoder_heads        = 4,
    encoder_ff_dim       = 512,
    n_max                = 512,

    # Context
    conv_kernel          = 5,
    fourier_mode         = "parallel",   # none/replace/parallel/adaptive/multiscale
    fourier_n_modes      = 64,
    fourier_mix_channels = False,        # True = full C×C complex per mode

    # Attention
    attention_backend    = "auto",       # auto/flash/sdpa/naive

    # Explorer
    n_prototypes         = 64,
    explorer_threshold   = 0.5,
    explorer_warmup_steps = 500,

    # Training
    len_hidden           = 64,
    dropout              = 0.1,
)
cfg.validate()
```

---

## Model Presets

| Preset | ~Params | latent_dim | enc_layers | fourier_mode |
|--------|---------|-----------|-----------|-------------|
| tiny   | ~1M     | 64        | 2         | none        |
| small  | ~6M     | 128       | 4         | none        |
| base   | ~25M    | 256       | 6         | parallel    |
| large  | ~100M   | 512       | 8         | multiscale  |

---

## Known Limitations

**Causal Fourier:** `FourierSpectralConv1D` uses full (non-causal) FFT. The
model is designed for parallel decoding so strict causality is not required
during training. If you need strict AR inference, use `fourier_mode="none"`
(CausalDWConv1D is truly causal).

**Flash Attention integration:** Flash Attention is available as a backend for
`efficient_attention()` but is not yet wired directly into the ByteEncoder's
`MultiheadAttention` module (PyTorch MHA does not expose a custom attention
kernel interface). The `set_attention_backend()` API is fully functional for
custom attention calls; full MHA integration is planned for v0.0.4.

**SDPS sparse data:** SDPS requires n_samples >> latent_dim for a
well-conditioned covariance. Minimum recommended: n_samples ≥ 4 × latent_dim.

**Multi-GPU / activation capture:** `SDPSTrainer._capture_activations()` does
not yet support DDP (model must be on a single device during SDPS). Run SDPS
before wrapping with DDP.

**Video decoding:** `bytes_to_video()` is not yet implemented. Only
`video_frames_to_bytes()` (encode) is available.

**FSDP + LoRA:** FSDP wrapping after `apply_lora()` may encounter issues
with parameter flattening. Recommended order: apply LoRA → merge weights →
wrap FSDP. Or use DDP instead.

---

## Version History

| Version | Highlights |
|---------|-----------|
| v0.0.3.2 | SDPS 8-gap fix, Fourier channel-mix/adaptive/multiscale, LoRA, multi-GPU, Flash Attention, 299 tests |
| v0.0.3  | SDPS gradient-free training, Fourier global context, model conversion, fine-tuning API, multimodal |
| v0.0.2  | CausalDWConv1D, LatentExplorer, HuggingFace integration, 78 tests |
| v0.0.1  | Initial byte-native architecture, ByteEncoder, parallel decoding |

---

## License

Apache 2.0 — see `LICENSE`.

## Author

Ömür Bera Işık, 2026.
