Metadata-Version: 2.4
Name: organic-clone-machines
Version: 0.0.2
Summary: OCM: Hodgkin-Huxley spiking neural networks as a Transformer-shaped language model architecture
Author: Ömür Bera Işık
License-Expression: MIT
Keywords: hodgkin-huxley,spiking neural networks,language model,neuroscience,transformers,GPU,biologically realistic,OCM
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.40
Requires-Dist: datasets>=2.18
Requires-Dist: numpy>=1.24
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Provides-Extra: examples
Requires-Dist: datasets>=2.18; extra == "examples"
Dynamic: license-file

# OCM — Organic Clone Machines

**v0.0.2** · Author: Ömür Bera Işık

A Transformer-shaped architecture where every sublayer is a simulated population of biologically realistic **Hodgkin-Huxley (HH) spiking neurons** — run in parallel on GPU as a single batch tensor.

> **First language model based on Hodgkin-Huxley spiking neuron dynamics.**

---

## Install

```bash
pip install organic-clone-machines
```

Or from source:

```bash
git clone <your-repo>
cd ocm_project
pip install -e .
```

---

## Quick Start

```python
from ocm import OCMConfig, OCMForCausalLM
import torch

cfg = OCMConfig(
    num_neurons=256,           # HH population per layer
    synapses_per_neuron=16,    # sparse out-degree per neuron
    quality="standard",        # integration fidelity preset
    hidden_size=256,
    num_layers=4,
    vocab_size=50257,
)

model = OCMForCausalLM(cfg)
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# Forward pass
ids = torch.randint(0, 50257, (1, 32))
out = model(ids, labels=ids)
print(f"Loss: {out.loss.item():.4f}")

# Text generation
tokens = model.generate(ids[:, :8], max_new_tokens=20, temperature=0.8)
```

### Train on any HuggingFace dataset

```python
from ocm import OCMConfig
from ocm.trainer import OCMTrainer

trainer = OCMTrainer.for_causal_lm(
    config=OCMConfig(num_neurons=512, synapses_per_neuron=32, quality="standard"),
    dataset_name="wikitext",
    dataset_config="wikitext-2-raw-v1",
    tokenizer_name="gpt2",
    output_dir="./ocm-gpt",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    learning_rate=3e-4,
)
trainer.train()
```

### CLI

```bash
# Text generation (causal LM)
ocm-train causal_lm \
    --num_neurons 512 --synapses_per_neuron 32 --quality standard \
    --dataset_name wikitext --dataset_config wikitext-2-raw-v1 \
    --tokenizer_name gpt2 --output_dir ./ocm-gpt \
    --num_train_epochs 3 --per_device_train_batch_size 8

# Masked LM (BERT-style)
ocm-train masked_lm \
    --num_neurons 256 --synapses_per_neuron 16 \
    --dataset_name wikitext --dataset_config wikitext-2-raw-v1 \
    --tokenizer_name bert-base-uncased --output_dir ./ocm-bert

# Sequence classification
ocm-train seq_clf \
    --num_neurons 128 --synapses_per_neuron 8 --num_labels 2 \
    --dataset_name stanfordnlp/imdb \
    --tokenizer_name gpt2 --output_dir ./ocm-imdb

# Token classification (NER)
ocm-train tok_clf \
    --num_neurons 128 --synapses_per_neuron 8 --num_labels 9 \
    --dataset_name conll2003 \
    --tokenizer_name bert-base-cased --output_dir ./ocm-ner

# See all options
ocm-train --help
```

---

## The Five Model Types

| # | Class | Task | HF Analogy |
|---|-------|------|-----------|
| 1 | `OCMModel` | Feature extraction backbone | `GPT2Model` |
| 2 | **`OCMForCausalLM`** | **Text generation** ← flagship | `GPT2LMHeadModel` |
| 3 | `OCMForMaskedLM` | BERT-style masked LM | `BertForMaskedLM` |
| 4 | `OCMForSequenceClassification` | Sentence classification / regression | `BertForSequenceClassification` |
| 5 | `OCMForTokenClassification` | NER, POS tagging | `BertForTokenClassification` |

```python
from ocm import (
    OCMModel,
    OCMForCausalLM,
    OCMForMaskedLM,
    OCMForSequenceClassification,
    OCMForTokenClassification,
)
```

All five are full `transformers.PreTrainedModel` subclasses — `save_pretrained`, `from_pretrained`, `push_to_hub` all work.

---

## Architecture

### The Hodgkin-Huxley Core

Each OCM block maintains a population of **N neurons**, each described by four ODEs (Hodgkin & Huxley, 1952):

```
C_m dV/dt = I_ext - g_Na·m³·h·(V-E_Na) - g_K·n⁴·(V-E_K) - g_L·(V-E_L)
dm/dt = αm(V)(1-m) - βm(V)·m
dh/dt = αh(V)(1-h) - βh(V)·h
dn/dt = αn(V)(1-n) - βn(V)·n
```

All N neurons are stored as a single `(batch, N, 4)` tensor for `(V, m, h, n)`. The derivatives for the entire population are computed in one vectorized PyTorch call — one kernel launch regardless of N. This is the core insight from the source paper: **GPU SIMD maps perfectly onto neuron-parallel HH integration**.

### Cross-Token Mixing (the Attention Analogue)

Each OCMBlock keeps a **spike ring buffer** of shape `(max_delay, batch, N)`. At token position `t`, neuron `j` receives:

```
I_syn[j](t) = Σᵢ  W_ij · S_i(t − δij)
```

where `δij` is a fixed integer axonal delay drawn from `[1, max_delay]` and `W_ij` is a learnable synapse weight. This delayed-spike readout is OCM's causal cross-token mixing primitive — the role attention plays in a Transformer.

### Recurrence vs. Parallelism

OCM processes token positions **sequentially** (RNN/SSM-shaped). Parallelism is along the **neuron dimension** (N neurons per step, all integrated in one batched call). The `past_ocm_state` cache lets you extend a sequence one token at a time in O(1) work per layer during generation.

| Dimension | Transformer | OCM |
|-----------|-------------|-----|
| Token positions | Parallel | Sequential (RNN-like) |
| Features per position | Sequential through layers | Parallel: N neurons/layer |
| Cross-position mixing | Attention (all-to-all) | Delayed spike buffer (causal, sparse) |

---

## Configuration Reference

```python
OCMConfig(
    # ── Biology ──────────────────────────────────────
    num_neurons=256,            # N neurons per layer
    synapses_per_neuron=16,     # out-degree k (must be < num_neurons)
    max_delay=8,                # spike ring buffer depth (token steps)
    v_thresh=0.0,               # spike threshold (mV)
    surrogate_beta=2.0,         # surrogate gradient steepness
    heterogeneous_neurons=False, # per-neuron learnable (g_Na,g_K,g_L)

    # ── Integration ───────────────────────────────────
    quality="standard",         # "draft"|"standard"|"high"|"research"

    # ── Architecture ─────────────────────────────────
    hidden_size=256,            # d_model
    num_layers=4,               # OCM block stack depth
    max_position_embeddings=1024,
    dropout=0.1,

    # ── Standard HF ──────────────────────────────────
    vocab_size=50257,
    pad_token_id=0,
    num_labels=2,               # for classification heads
)
```

### Quality Presets

All presets use `dt = 0.01 ms` (paper's stability floor — smaller dt → divergence). Only substep count and integrator vary:

| Preset | Substeps | Integrator | Cost | Use when |
|--------|----------|------------|------|----------|
| `draft` | 1 | Euler | 1× | Shape/pipeline debug |
| `standard` | 4 | Euler | 4× | Normal training ← default |
| `high` | 8 | Euler | 8× | Long sequences |
| `research` | 8 | RK4 | 32× | Max fidelity |

### Sizing Guide

| Scale | `num_neurons` | `synapses_per_neuron` | ~Params (4L, d=256) |
|-------|--------------|----------------------|---------------------|
| Micro | 64 | 8 | ~500 K |
| Small | 256 | 16 | ~3 M |
| Medium | 512 | 32 | ~10 M |
| Large | 1024 | 64 | ~35 M |

---

## HuggingFace Integration

```python
# Save / load
model.save_pretrained("./my-ocm")
model = OCMForCausalLM.from_pretrained("./my-ocm")

# Push to Hub
model.push_to_hub("username/ocm-wikitext")

# Stream large datasets
trainer = OCMTrainer.for_causal_lm(
    config=cfg,
    dataset_name="c4",
    dataset_config="en",
    streaming=True,
    max_train_samples=100_000,
    tokenizer_name="gpt2",
    output_dir="./ocm-c4",
)
```

---

## Training Dynamics / Cold Start

Early in training, neurons receiving weak stimulus may not fire (`s = 0`). Synapse weight gradients are `d(W·s)/d(W) = s`, so un-fired synapses get zero gradient from the synaptic path.

Mitigation strategies:
- **Lower `surrogate_beta`** (e.g. `1.0`) early on — widens the gradient region around threshold
- **Start with `quality="draft"`** for the first few hundred steps, then switch to `"standard"`
- `stim_scale` (per-block learnable scalar) is initialized to `10.0`; a warm-start with a slightly larger value pushes neurons closer to threshold faster

---

## The Biology

Standard squid-axon parameters (Hodgkin & Huxley, 1952; Dayan & Abbott 2001):

```
C_m = 1.0 μF/cm²    g_Na = 120 mS/cm²   E_Na = +50 mV
                     g_K  =  36 mS/cm²   E_K  = −77 mV
                     g_L  = 0.3 mS/cm²   E_L  = −54.387 mV
V_rest = −65 mV      m_∞ = 0.0529   h_∞ = 0.5961   n_∞ = 0.3177
```

With `heterogeneous_neurons=True`, each neuron gets its own learnable `(g_Na, g_K, g_L)` initialized near standard values with small noise, allowing population diversity to emerge via gradient descent.

---

## PyPI

```bash
# Install
pip install organic-clone-machines

# Upgrade
pip install --upgrade organic-clone-machines

# Publish a new version (maintainer only)
python -m build
python -m twine upload dist/*
```

---

## Citation

```bibtex
@software{isik2026ocm,
  title   = {OCM: Organic Clone Machines — Hodgkin-Huxley Spiking Neurons as a Language Model Architecture},
  author  = {Işık, Ömür Bera},
  year    = {2026},
  version = {0.0.2},
}

@article{isik2026hh,
  title  = {Massively Parallel Hodgkin-Huxley Neuron Simulation via GPU Batch Parallelism: One Neuron Cost, Billion Neuron Scale},
  author = {Işık, Ömür Bera},
  year   = {2026},
}
```

---

## License

MIT License — Copyright (c) 2026 Ömür Bera Işık
