Metadata-Version: 2.4
Name: physis-lm
Version: 0.1.2
Summary: Reference PyTorch implementation of the Physis-LM preprint (byte-native, non-autoregressive, hierarchical-hourglass language model). Made with AI assistance.
Author: Omur Bera Isik
License: Apache-2.0
Keywords: language-model,non-autoregressive,byte-level,hierarchical,pytorch,research
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy>=1.24
Requires-Dist: torch>=2.1
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: hypothesis>=6.0; extra == "test"
Provides-Extra: hf
Requires-Dist: transformers>=4.40; extra == "hf"
Provides-Extra: jax
Requires-Dist: jax>=0.4; extra == "jax"
Requires-Dist: jaxlib>=0.4; extra == "jax"
Dynamic: license-file

# physis-lm

Reference PyTorch implementation of the preprint **"Physis-LM: A Parallel
Non-Autoregressive Language Model with Hierarchical Latent Compression and
Toward Guaranteed Output Consistency"** (Omur Bera Isik, 2026) — a byte-native,
non-autoregressive language model built around a Dynamic-Depth Hierarchical
Hourglass (DDHH), a Physis Long-Range Bridge (PLRB), Length Parameters (LP), and
a Soft Consensus Module (SConM).

Made with AI assistance. Version 0.1.2.

## What's new in 0.1.2

- **The model now demonstrably generalizes on a task that requires routing a
  transformation through the bottleneck — after fixing a bug that made it look
  like it couldn't.** `AdaptiveSlotPool` (Section 10.1) was the only attention in
  the model that did not RMSNorm its input before the K/Q projections; combined
  with the shared small init, its pre-softmax scores were std ~1e-7, softmax was
  uniform for all M slot queries, and the deep bottleneck latent Z collapsed to M
  copies of one vector (measured: cross-slot cosine 1.0000). A *copy* task hid
  this (the output->input cross-attention bypasses Z); a Caesar-*shift* task
  exposed it — training sat at chance. **Fix (bug S2):** RMSNorm before the K/V
  projections + a 1/sqrt(C) init on WQs/WKs. After it, a 3.7M and a 14.7M model
  both learn shift+3 to **100% held-out byte AND sequence accuracy** on a
  seed-disjoint test set (26^16 prompt space, so this is rule-learning, not
  memorization). See `docs/SCALE_FINDINGS.md` study S8.
- **On real text, it does NOT yet produce meaningful language — and the reason is
  compute, not the architecture.** Trained on 1.1M bytes of Shakespeare
  (next-chunk prediction, an 11M model on CPU), the model's loss falls to ~4.8
  bits/byte within ~150 steps and then plateaus: it learns the marginal character
  distribution (its samples are a single repeated character) and little context
  structure. This reproduced across three chunk lengths and two batch sizes. A
  byte model needs many epochs over even this small corpus to learn language, and
  at ~0.5-2s/step on a 4GB CPU box the model sees well under one epoch; plain
  cross-entropy (without the designed auxiliary losses) and the parallel-chunk
  objective compound it. Whether the architecture models language at real scale
  is untested here — gated by hardware, not answered. Study S9.
- **Section 3b — Hugging Face wrapper (`physis_lm.hf`).** `PhysisLMConfig` +
  `PhysisLMForByteModeling` round-trip through `save_pretrained`/`from_pretrained`
  and register with `AutoModel`. It is non-autoregressive, so `generate()` RAISES
  and points at `generate_cpd` rather than silently running HF's AR decoder.
  Optional import; `transformers` is not a hard dependency.
- **Section 3a — JAX parity backend (`physis_lm.jax_backend`).** The primitive
  layers and a full `PhysisBlock`, ported as pure functions over torch-copied
  weights and parity-tested to ~1e-10 in x64 (12 tests). Scope is stated plainly:
  the full end-to-end hourglass/PLRB/SConM forward is **not** ported (note J1).
- **Appendix D — power-iteration spectral norm (`spectral_norm_method=
  "power_iteration"`).** The fused CUDA kernels remain out of scope (no GPU), but
  the numerical method they would accelerate is implemented and selectable;
  converges to the exact SVD sigma_max. Its autograd surrogate makes gradcheck
  fail *by design* — documented, not hidden (note D1).
- **Honest scale ceiling.** On the 4GB CPU sandbox, models above ~15M are killed
  by transient out-of-memory spikes mid-training (PLRB's content-dependent tensor
  shapes fragment the allocator) even at ~1GB steady RSS; `malloc_trim` + bs=4
  made 14.7M stable, and beyond that needs more RAM or a GPU. This is an
  environment limit, not an architecture result (study S8b).

## What's new in 0.1.1

Four of the reference's documented findings now have fixes. Each was
re-reproduced against 0.1.0 before being touched, is covered by a new test, and
— except F5, which only changes an initialization and is a strict improvement —
is **off by default**, so every 0.1.0 result still reproduces exactly.

- **F5 fix (ON by default):** the attention-pool downsampler is now
  norm-preserving (W_V init std 1/sqrt(C) + a learnable gain), so the residual
  stream no longer collapses below RMSNorm's eps floor at init. Strict
  improvement; `pool_norm_preserving=False` restores the old init.
- **R2 fix (opt-in prototype):** `plrb_soft_routing=True` gives PLRB a
  differentiable soft-routing path so `W_route` actually trains — the gradient
  route the paper claims but the literal argmax doesn't provide.
- **F3 fix (opt-in prototype):** `generate_cpd(streaming=True)` now runs, via
  periodic PLRB re-hashing (Remark 15.1). Exact (bit-identical to naive CPD) at
  `rehash_period=1`. It does **not** reach the paper's O(L_out) cost — the DDHH
  still recomputes each chunk, because its coarse levels genuinely can't be
  cached (finding F3 stands) — and that is stated, not hidden.
- **F4 option (opt-in):** `fixed_depth_padding=True` pins ell/n0 to a constant
  graph. A true logit-level padding-length invariance was investigated and shown
  structurally impossible for this architecture; see "Batching and
  train/inference consistency" below.
- **Thinker-mode (opt-in prototype):** `thinker_steps > 1` (and optional
  `thinker_scratchpad_slots`) iterates the deep bottleneck over the latent as a
  parallelism-preserving deliberation loop (no autoregressive fallback). The
  mechanism is live and trainable; it is **not** a demonstrated reasoning gain —
  the extra steps must be trained to do useful work. See "Thinker-mode" below.
- License is now **Apache-2.0** (was MIT).

Full detail and honest limits: `docs/IMPLEMENTERS_NOTES.md` (notes F3, F4, F5,
R2) and `docs/SCALE_FINDINGS.md` (studies S3, S6, S7).

## Read this first (honesty statement)

**Physis-LM is currently in the demo stage; we cannot guarantee that everything
works. If you train the model and get poor results, please be aware that this
could be one of the contributing factors.**

The paper's own title page states that **no version of this architecture has
ever been assembled, trained, or evaluated end-to-end**, and that every quality
claim in its Section 20 is an untested prediction. This package changes the
first half of that sentence — the architecture is now assembled, correct at the
paper's own toy scale, and heavily tested — and none of the second half:

- **There are no pretrained weights.** Anywhere. `physis-lm train` really
  trains (and toy-scale tests verify learning happens), but training a
  competitive model needs the paper's 50-500B-byte budgets and GPUs.
- Everything implemented is tested against the paper's own numbered claims;
  everything not implemented raises a clear error or is listed below.
- Three paper claims are **demonstrably false as written** (findings F3, F4,
  F5) and one is self-contradictory (R2); 0.1.1 adds an honest fix or mitigation
  for each (see "What's new" above) while the underlying findings are still
  documented in full, not papered over. See `docs/IMPLEMENTERS_NOTES.md` and
  `docs/SCALE_FINDINGS.md`.

## Install

```bash
pip install physis-lm            # from a built wheel/PyPI
# or, from a source checkout:
pip install -e ".[test]" && pytest
```

Dependencies: `numpy`, `torch>=2.1` (CPU is fine — this package is developed
and tested entirely on one CPU core). Python >= 3.10.

## Quickstart — library

```python
import torch
from physis_lm import PhysisCoreConfig, PhysisLM

cfg = PhysisCoreConfig(C=32, M=8, H=4, W=8, Nmax=32, nref=2048,
                       NB=2, K_sconm=2, droute=8, R=2, bucket_target=8)
model = PhysisLM(cfg)

ids = torch.tensor([list(b"hello physis, this is a byte-native model")])
out = model(ids)                       # single parallel forward pass
print(out.logits.shape)                # (1, Nmax, 256) — all positions at once
print(out.sigma_score)                 # LP completion score (Section 12.6)

text, info = model.generate_cpd(ids, Lmax=128)   # Chunked Parallel Decoding
print(bytes(text.tolist()))
```

Training (the full Section 18.4 three-stage schedule, homoscedastic loss
weighting, curriculum, checkpointing):

```python
from physis_lm import data as D
from physis_lm.torch_backend import train as TR

ds = D.dataset_from_paths(["my_corpus/"], context_len=128,
                          Nmax=cfg.Nmax, pad_byte=cfg.pad_byte)
provider = D.batch_provider_from_dataset(ds, batch_size=4, pad_byte=cfg.pad_byte)
history = TR.train(model, provider, TR.TrainSettings(total_steps=1000,
                                                     ckpt_dir="ckpt"))
```

The model pads internally to exactly `M * r**ell` (the CV 15.3 hard
requirement), so **you never need to understand DDHH depth arithmetic**: any
input length up to `cfg.max_supported_raw_length() - Nmax` just works, and
anything else raises an error telling you to increase `nref`.

## Quickstart — CLI

```bash
physis-lm selftest                     # end-to-end pipeline check (~5 s)
physis-lm info --size base             # configs + parameter-count oracles
physis-lm train --data my_files/ --steps 500 --ckpt-dir ckpt
physis-lm generate --ckpt ckpt/final.pt --prompt "hello" --max-bytes 200
physis-lm scale-study                  # regenerate docs/SCALE_FINDINGS.md
```

`train` accepts `.txt`, `.jsonl` (`--text-field`), `.csv`/`.tsv`, `.html`
(`--html-mode strip|raw`), and raw binary files; malformed inputs fail with
messages naming the file, line, and problem.

## Status table

Per-module status; each module's docstring carries the same note, and
`docs/IMPLEMENTERS_NOTES.md` holds the full list of paper
ambiguities/contradictions and the decision taken for each (tags like "note B7"
below resolve there).

| Module | Paper | Status | Tested by | Limitations / notes |
|---|---|---|---|---|
| `core` geometry, padding, output slice | 7.2, Rem. 7.1, CV 15.3, Rem. 15.4 | implemented | test_core (incl. the literal CV 15.3 sweep, hypothesis sweeps) | ell >= 1 clamp (G1) |
| `core` complexity + parameter oracles | 7.7, C.1/C.6, Prop 9.1, 17.5, Sec. 1, 12.2, 18.7, App. A | implemented | test_core reproduces every printed table/figure | paper's 130,808 misprint recorded (P1) |
| `core` LSH oracle + config | 9.3, Prop 9.2, CV 9.3, App. A.1 | implemented | test_core (collision/recall sims; config rejections) | Omega = f(seed, round, B) (R1) |
| RMSNorm / RoPE / SwiGLU / spectral / local attention | 6.2-6.6, 28.1-28.4, CV 8.3, Prop 8.2 | implemented | test_torch_layers (banded==O(n^2) oracle, CV 8.3 both halves, circulant Jacobian, gradchecks) | banded impl is O(nW) memory; no CUDA kernels (App. D out of scope, no GPU) |
| PhysisBlock, pooling, upsample, bottleneck, LP | 6.5, 7.3-7.4, 10, 12, CV 8.5, CV 12.6 | implemented | test_torch_blocks (CV 8.5 both cases, all three LP propositions vs autograd) | B3, B8-B10 readings; 0.1.2 fixes AdaptiveSlotPool bottleneck-collapse bug S2 (RMSNorm + 1/sqrt(C) init) |
| PLRB | 8-9, 19.5, C.6 | implemented (fixed hashing; 0.1.1 opt-in soft routing) | test_torch_plrb (segmented == mask oracle, sentinel, zero W_route grad by default, soft-routing gives W_route a real gradient) | default trains everything except W_route as literally specified (R2); 0.1.1 `plrb_soft_routing` fixes this (prototype); learnable-Omega variant a disclosed stub (R3); dense hashing = C.6 trap at large n (study S5) |
| SConM + streaming readout | 14, CV 14.2, Lemma 14.8, D.2 | implemented | test_torch_sconm (LOO exact single/multi-head, LSE recovery, vertex bound, scopes) | no contraction claim — conditional on unmeasured L_g (S1; studies S1/S2); 0.1.2 adds opt-in power-iteration spectral norm (note D1) |
| Full model, integration, LSR, CPD | App. B, CV 15.2/15.3, Rem. 15.4, 7.8.2, 15 | implemented | test_torch_model (the paper's own sixteen-join toy-scale integration test, ablations, LSR bit-exactness, CPD incl. streaming exact-at-period-1) | 0.1.1 streaming CPD runs via periodic PLRB re-hash, exact at rehash_period=1, but does not reach O(L_out) — DDHH still recomputes (finding F3); CPD stops gracefully at the context limit |
| Losses, schedules, training loop, PTCC | 16.3, 18, 19, CV 18.2/18.4/18.10 | implemented | test_training (a real toy run that learns + checkpoint round-trip), test_data_ptcc_pcc | L1, T1-T3 decisions; no mixed precision / Flash Attention / grad checkpointing (throughput devices, no GPU here) |
| PCC | 24, CV 24.2 | implemented | test_data_ptcc_pcc (losslessness incl. adversarial streams, tier accounting) | ratios are corpus claims — only CV 24.2's qualitative pattern asserted (C1) |
| Data layer + CLI | task Sec. 5 | implemented | test_data_ptcc_pcc, test_cli (end-to-end train->generate on real files) | formats defined precisely in data.py's docstring |
| Scale studies | task Sec. 2 | implemented | test_scale_studies | untrained instantiations only; see report header |
| Thinker-mode (deliberation loop) | task Sec. 5 | prototype (opt-in) | test_torch_model (default == single-pass, live + trainable, parallelism preserved) | mechanism only; genuine reasoning gains require training the steps (note K1) |
| JAX parity backend (layers + PhysisBlock) | task Sec. 3a | implemented (layer/block level) | test_jax_parity (12 tests, torch-vs-JAX at copied weights, ~1e-10 in x64) | full end-to-end hourglass/PLRB/SConM forward NOT ported — tested foundation only (note J1) |
| Hugging Face wrapper | task Sec. 3b | implemented | test_hf (config round-trip, save/from_pretrained identical logits, AutoModel, non-AR contract) | optional import; `generate()` raises -> `generate_cpd`; no pretrained weights |
| Appendix D spectral-norm kernel | App. D | power-iteration path implemented; fused CUDA kernels out of scope | test_torch_sconm (converges to SVD sigma_max; gradient flow) | no GPU for fused kernels; gradcheck fails by design (surrogate gradient, note D1) |
| Any-to-Physis weight transplant | task Sec. 4 | **deferred to a later release** | — | intentionally not shipped in this version; design in docs/SECTIONS_3_4_DESIGN.md |

Test suite: **1497 tests, all passing** (`pytest -q`), dominated by cheap
parametrized pure-math sweeps plus hypothesis-generated cases; every
Computational Verification remark in the paper that can run on CPU is
re-executed against this codebase's actual code.

## The findings and their 0.1.1 fixes

- **F3** — Streaming DDHH's premise ("levels above j* do not change") is false:
  spectral mixing is a global circular convolution at every level (quantified in
  study S7). *0.1.1:* `generate_cpd(streaming=True)` now runs — bit-identical to
  naive CPD at `rehash_period=1`, and periodic PLRB re-hashing (Remark 15.1) for
  larger periods — but the DDHH still recomputes each chunk, so it does not reach
  the paper's O(L_out) streaming cost. The finding stands; the fix is honest
  about what it does and doesn't buy.
- **F4** — the literal Section 19.5 claim that padding to a longer batch length
  leaves behavior identical is structurally impossible (padding changes ell, and
  extending the input shifts the output placeholders). The achievable invariant
  — batch-content independence at fixed shape — holds to 1e-10 and is tested
  (study S6). *0.1.1:* `fixed_depth_padding=True` pins ell/n0 to a constant
  graph; a true logit-level length invariance was investigated and shown
  impossible here.
- **F5** (found by this implementation) — at initialization the residual stream
  collapsed geometrically through the attention-pool downsamplers (~x0.02-0.09
  per level), pushing deep levels under RMSNorm's eps floor (study S3). *0.1.1
  (ON by default):* the norm-preserving downsampler holds every level above the
  floor across the whole grid measured; S3 is now a before/after table.
- **R2** — Section 9.7 is self-contradictory: it says the routing argmax "is not
  differentiated through" yet "gradients flow normally through W_route". The
  literal argmax gives W_route exactly zero gradient (tested), so the default
  fixed-hashing path trains everything except W_route. *0.1.1:*
  `plrb_soft_routing=True` supplies a differentiable soft-routing path so
  W_route trains (prototype; off by default).

## Batching and train/inference consistency

Because n0 = M * r**ell depends on the padded input length, this architecture's
output for a fixed sequence is **not** invariant to how much padding you add:
different padded lengths mean a different number of DDHH levels and a different
spectral DFT length (finding F4). Two practical consequences:

1. The guarantee you *can* rely on is **batch-content independence at a fixed
   batch shape**: a sequence's logits do not depend on which other sequences
   share its batch, as long as the batch's padded length is the same. This holds
   to floating-point precision and is tested.
2. To keep training and inference consistent, either (a) use the **same nref and
   the same batch-shaping discipline** in both — so a given input lands at the
   same ell in training and at inference — or (b) set
   `PhysisCoreConfig(fixed_depth_padding=True)`, which pins every input to
   ell_max / n0 = M*r**ell_max. Option (b) makes the computation graph constant
   across input lengths (removing the batch-shape-dependent DFT-length drift) at
   the cost of always running at maximum depth; it does not make two
   different-length inputs produce identical logits (that is impossible here),
   but it removes padding-length as a source of train/inference mismatch.

## Thinker-mode (prototype)

`PhysisCoreConfig(thinker_steps=T, thinker_scratchpad_slots=S)` turns on an
opt-in deliberation loop (task Section 5). Instead of a single deep-bottleneck
pass over the compressed latent Z, the bottleneck is applied T times, adding a
learnable per-step code each iteration, with S optional learnable "scratchpad"
latent slots as working memory (dropped before readout so the output shape is
unchanged). Every latent slot is refined simultaneously at each step, and the
loop iterates over deliberation steps, never over output tokens — so the
architecture's parallelism is preserved and there is **no** autoregressive
fallback. `thinker_steps=1` with no scratchpad (the default) is byte-for-byte
the single-pass model.

What this is and isn't: the mechanism is implemented, live (non-zero step codes
change the output), and trainable (gradients reach the step codes and
scratchpad) — all tested. It is **not** a demonstrated reasoning improvement.
The per-step codes are zero at initialization, so an untrained model's first
step equals the ordinary pass and further steps just re-apply the bottleneck;
making the extra steps do genuinely new work is a *training* problem (train with
`thinker_steps > 1`, most plausibly with a ponder/ACT-style objective). This
release ships the mechanism honestly labeled as a prototype, not a claim that
iterating an untrained bottleneck reasons better. See `docs/IMPLEMENTERS_NOTES.md`
note K1.

## Repository layout

```
physis_lm/
  core.py            geometry, cost/parameter oracles, LSH math, config (no torch)
  data.py            byte-level corpus loading + batching (stage-1 padding only)
  pcc.py             Physis Context Compression (Section 24; no torch)
  cli.py             the `physis-lm` command
  scale_studies.py   Section-2 measurements -> markdown report
  torch_backend/
    layers.py blocks.py plrb.py sconm.py model.py losses.py train.py ptcc.py
tests/               1470 tests; conftest.py holds the paper's toy scale
docs/                PLAN.md (pre-implementation plan), IMPLEMENTERS_NOTES.md,
                     SCALE_FINDINGS.md
```

## Citing

If you use this implementation, cite the Physis-LM preprint (Omur Bera Isik,
2026). This codebase: `physis-lm` 0.1.2, made with AI assistance, Apache-2.0 license.
