Metadata-Version: 2.4
Name: fluidlayer
Version: 2.1.0
Summary: A research memory layer for language models with optional GPT-2 and Qwen2 adapters
Author: Kapil Dhinwa
License: MIT
Project-URL: Homepage, https://github.com/kapildhinwa2010-dev/fluidtech
Project-URL: Repository, https://github.com/kapildhinwa2010-dev/fluidtech
Project-URL: Issues, https://github.com/kapildhinwa2010-dev/fluidtech/issues
Project-URL: Changelog, https://github.com/kapildhinwa2010-dev/fluidtech/blob/main/docs/CHANGELOG.md
Keywords: llm,memory,long-context,kv-cache,retrieval,cellular-sheaves,persistent-homology
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<3,>=1.23
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: build; extra == "dev"
Provides-Extra: gpu
Requires-Dist: torch>=2.1; extra == "gpu"
Requires-Dist: transformers>=4.40; extra == "gpu"
Requires-Dist: accelerate>=0.30; extra == "gpu"
Requires-Dist: safetensors>=0.4; extra == "gpu"
Dynamic: license-file

# FluidLayer

**A research memory layer for LLMs — attachable with one residual-stream insertion.**

`16 KB` research-scale fluid state as a compact long-range channel ·<br>
exact recall of retained token IDs from outside the attention window ·<br>
pure NumPy core, with optional PyTorch support ·<br>
eleven measured negative results, published on purpose

[![CI](https://github.com/kapildhinwa2010-dev/fluidtech/actions/workflows/ci.yml/badge.svg)](https://github.com/kapildhinwa2010-dev/fluidtech/actions/workflows/ci.yml)
![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)
![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)
![Self-verification](https://img.shields.io/badge/self--verification-50%2F50-brightgreen.svg)

---

## The problem, in one number

A 70B model on 2×H100 80GB cannot hold a single 128K-token session.

The weights take **131.5 GB**; the KV cache for 128K tokens needs another **40 GB**, and only **28.5 GB** is left.
This is a memory-budget illustration, not a claim that this prototype
replaces a KV cache or is a drop-in serving optimization.

The usual answer is to quantise it, compress it, or make the model smaller.
This project asks a different question:

> What if a compact, constant-size memory channel sat beside attention and
> could be addressed by content instead of position?

---

## What this is

A research memory layer designed to attach to a causal LM:

```text
h' = h + W_e · E(h)  +  W_l · S(h)
         ^^^^^^         ^^^^^
         exact tier     lossy tier (the constant-size fluid state)
```

- The **exact tier** is a bounded, content-keyed store of token IDs.
  **20 bytes per packed slot.** A recalled value is an `int32` — bit-exact, not approximate.
  The attachable store tracks majority continuations and evicts the
  least-corroborated entries. Bit-exactness describes the stored token ID,
  not its truth or provenance. The Python reference's memory estimate
  excludes Python object overhead.

- The **lossy tier** is a sheaf-diffusion aggregate over `M` anchors — `M × r` floats, constant size, whatever you stream through it.

The two tiers exist because a diffusion smears. If the exact value of a token lives only inside the fluid, it is mixed with its neighbours and is provably no longer exact.

So:

> **The fluid answers WHERE. The register answers WHAT.**

---

## The three numbers

| Claim | Measured result |
|---|---|
| Attaching the layer changes nothing | The logits are **bit-identical** (max diff `0.000e+00`). The output projections start at zero, so the branch is a **provable no-op** until someone trains it. |
| It adds a capability the model did not have | Cross-window verbatim recall: **`0.050 → 0.800`** (chance `0.125`) — **16× the base model** — for a fact planted 400 tokens outside the attention window. |
| The stored token ID is exact | A retained token ID is returned bit-exactly. This does not verify whether the token is true, safe, or authenticated. |
| The model gets better | Gated retrieval improves the base model’s perplexity by **`−0.176 nats`** (`ppl 13.61 → 11.42`) — the model’s own confidence decides when the memory speaks. |
| It beats its closest prior art in the regime it targets | Against a **kNN-LM** (the standard way to use a datastore), with a **16 KB** store the gated layer wins by **7.4×** on loss (`−0.1134 vs −0.0154`). With a 1 MB store the kNN-LM wins by 6.4×, so the claim is precise: **the gate is the contribution under a hard memory budget.** |
| The fluid tier earns its place | Used as the **router** (the geometry as an address space), it rescues recall when the exact store is too small (`0.100 → 0.625`) and fixes hard contexts where the base model has no idea (`+0.053 → −0.071 nats`) — at **0.0005%** of a layer’s compute. |
| The diffusion tier delivers none of it | With the lossy tier alone: **`0.050`**, i.e. the base model’s own number. The exact tier delivers **all** of it. |

Measured on a frozen base model, in [`results/e6_graft.json`](results/e6_graft.json).
The recipe is in [`docs/GRAFT.md`](docs/GRAFT.md).

---

## Try it, in 60 seconds

```bash
git clone https://github.com/kapildhinwa2010-dev/fluidtech.git
cd fluidtech
pip install -r requirements.txt

python selftest.py              # 50 machine-checked claims
python examples/quickstart.py   # the tour
python examples/graft_demo.py   # the capability, end to end
python examples/gate_demo.py    # the gate: recall AND a better perplexity
python experiments/e11_exact.py # the exact memory (the full benchmark)
```

To install the published package instead of cloning the repository:

```bash
python -m pip install fluidlayer
```

Expected output:

```text
base model        : 1/20 correct  (it never saw the fact)
base + the layer  : 20/20 correct  (the layer retrieved it)
```

---

## Using it

```python
import torch
from fluidlayer.torch_graft import SAABranchTorch

# Assume `model` is a loaded causal LM and `windows` contains 1-D token-ID tensors.
embedding = model.get_input_embeddings()
branch = SAABranchTorch(
    M=64,
    r=8,
    d=embedding.weight.shape[1],
    emb=embedding,
    ctx_len=1,
    read_gain=300.0,
    transform="pc1",
    use_exact=True,
    use_lossy=False,
).to(embedding.weight.device)

branch.reset()                       # a new session

with torch.no_grad():
    for input_ids in windows:        # one 1-D token-ID window at a time
        content = embedding(input_ids)
        residual = content
        h, info = branch(residual, content, input_ids)
        logits = model(inputs_embeds=h.unsqueeze(0)).logits[0]
        logits = branch.gated_logits(logits, info)
```

- **4 KB** of fluid state for `M=64, r=8`, plus a logical **20 B per populated context or anchor slot**; the Python reference has additional object overhead.
- The base model is frozen the whole time.
- The output projections start at zero, so attachment initially preserves
  the model output. Enabling retrieval is a separate, calibrated operation.
- This sketch omits model-specific positional embeddings, attention masks,
  session lifecycle, and gain calibration. See the [integration notes](docs/GETTING_STARTED.md#attach-the-optional-pytorch-adapter).
- The NumPy reference implementation is [`fluidlayer/graft.py`](fluidlayer/graft.py), and the Torch one mirrors it exactly.

Start with the [documentation guide](docs/README.md), then read
[Getting started](docs/GETTING_STARTED.md) and the [API reference](docs/API.md).

---

## What it does not do

Being explicit, because this is where grafts usually oversell:

- It does **not** make the model better at language — it makes it better at remembering what it has already seen.
- It does **not** extend the attention window — the window is still 128 tokens; the layer can quote from outside it, not reason over it.
- It does **not** compose, infer, or answer questions about what it holds.
- The diffusion tier has **no measurable upside** on real text and used to damage the model it was attached to (`+0.81 nats`) until its training signal was fixed — keep it off.

The full list, with numbers: [`docs/LIMITATION.md`](docs/LIMITATION.md)

---

## At ten million tokens

Give a model 10M tokens with the layer attached, and:

- The state is still **16 KB**.
  The same tokens as a KV cache would require **1.31 TB** for an 8B model, **3.28 TB** for a 70B one.
- Filling it costs **80 seconds** and **no model forward at all** — the exact tier is model-free.
- With distinctive keys it recalls **6/6** facts planted from 100K to 9M tokens deep.
  With realistic 12-gram keys and a 16 KB budget it recalls **0/6** — its reach is the last ~10,000 tokens.
- The perplexity delta is **`−0.0057 nats`** at the right injection gain, and **`+1.08 nats`** at the gain that maximises recall — the two knobs are three orders of magnitude apart.
  The gate closes that gap: scaling the override by the model’s own uncertainty keeps recall at **`1.000`** and improves perplexity to **`11.42`**.

Details: [`docs/TEN-MILLION.md`](docs/TEN-MILLION.md)

---

## The negative results

This is the part that makes the rest believable.
Eleven negative or unexpected findings are documented. Most have checked-in
result files; the archived transformer and real-GPT-2 outputs are now
checked in. Their original model/script environments are not fully included,
so those historical runs cannot be independently repeated as-is.

| What we expected | What happened |
|---|---|
| The **topological** selection criterion picks the right facts | It does **not** — `0.000` precision vs `0.970` for plain geometric affinity. |
| `‖L·1‖∞` detects the frozen state | It does **not** — the frustration index does; the old claim is now a test case. |
| `λ₂` predicts long-context quality | Barely — Pearson `0.20`. |
| A transformer loses at some scale | The archived run says it **wins**; see `results/train_results.json`. |
| A domain prior beats a frequency table | It helps (`−0.006 nats`) but a **unigram table wins 6× more**. |
| The diffusion tier is neutral on a frozen model | It **degrades** the model by **`+0.81 nats`** (fixed → `+0.0000`, but no upside either). |
| Training the layer helps | It **hurts** — the training-free identity map is better (`0.800 vs 0.650`). |

Read it: [`docs/NEGATIVE-RESULTS.md`](docs/NEGATIVE-RESULTS.md)

---

## The repository

```text
fluidlayer/            the package (NumPy core; optional PyTorch adapters)
  graft.py             the attachable memory layer
  memory.py            the frozen register (20 B per fact, bit-exact)
  torch_graft.py       the same layer in PyTorch, for a real LLM
  gpt2_model.py        a GPT-2 forward pass in PyTorch (no transformers needed)
  gpt2_tokenizer.py    the GPT-2 byte-level BPE, in pure Python
  qwen2_model.py       an optional PyTorch Qwen2 forward pass
  sheaf.py topology.py diffusion.py blocks.py autodiff.py ...

experiments/           every measurement; each writes a JSON into results/
  e14_real_gpt2.py     the graft on a REAL pretrained GPT-2 (124M)
  e15_scale_model.py   the graft on Qwen2.5-0.5B
  e13_knn_baseline.py  the layer vs. the closest prior art (kNN-LM)

tools/                 the artifact checker, bundler, checkpoint downloader,
                       and PyPI fetcher
fluidlayer/bpe.py      the byte-level BPE tokenizer

tests/                 the unit tests (pytest, or run them without it)

examples/              quickstart.py, graft_demo.py, gate_demo.py, appkit/

docs/                  the design docs, the API, the limits, the theory report

results/               every number in the README, as JSON
```

---

## Documentation

Browse the [documentation guide](docs/README.md) for a topic-based index.
The [getting-started tutorial](docs/GETTING_STARTED.md) and
[API reference](docs/API.md) are the recommended entry points. Research
claims and their limits are explained in the
[graft report](docs/GRAFT.md), [negative results](docs/NEGATIVE-RESULTS.md),
and [limitations](docs/LIMITATION.md).

---

## Real-model experiment status

The E14 GPT-2 experiment script, forward implementation, tokenizer, and
archived `results/e14_real_gpt2.json` are included. The checkpoint is not;
rerun E14 to independently reproduce the recorded measurements. See
[the reproduction guide](docs/REPRODUCING.md) for download and run commands.

E15 adds a from-scratch Qwen2.5-0.5B forward implementation and app-kit
integration. Its archived run records bit-identical attachment, exact recall
of a planted fact 1,411 tokens back, and a best gated loss improvement of
`0.041` nats; the kNN-LM baseline improves loss by `0.274` nats, so this run
does not show the gate outperforming that baseline. The checkpoint is not
bundled. Reproduction requires the optional GPU dependencies and local model
weights; see [E15 instructions](docs/REPRODUCING.md#qwen2-scale-experiment-e15).

The Torch twin is now tested against the NumPy reference in [`tests/test_torch_parity.py`](tests/test_torch_parity.py): the two implementations must agree on the router, the store and the gated override.

That test found **four real port bugs**:

1. A different route scale.
2. The wrong routing content — the residual stream instead of the position-free embedding.
3. A different confidence formula.
4. A read/write order that broke the snapshot semantics.

Four bugs that would have shipped silently.

The checks cover the tested behaviors; they do not prove every research
claim or validate production-scale operation.

---

## A research prototype, honestly labelled

The core research experiments use small CPU models and character-level or
512-token subword corpora. E14 and E15 are separate pretrained-model
experiments; their checkpoints are not included.

The subword repeat is the closest thing to a realistic tokenizer in this repository.
The GPU run, [`experiments/gpu_graft.py`](experiments/gpu_graft.py), is written and has not been executed here.

---

## Reproducing

```bash
python selftest.py                    # 50 checks
python experiments/run_all.py --full  # every measurement (~40 min, CPU)
```

Everything is pure NumPy, CPU-only, and needs no network.

The details and expected outputs: [`docs/REPRODUCING.md`](docs/REPRODUCING.md)

---

---

## 🤝 Contributions & corrections

Suggestions, bug reports, and corrections are welcome, especially measurement
disagreements. Please open an issue or submit a pull request.

If a number in the docs does not match what you get, that is a bug in the code or in the documentation, and it is the most valuable report you can file.

See [`CONTRIBUTING.md`](CONTRIBUTING.md).

---

## License

MIT — see [`LICENSE`](LICENSE).

If you use this, please cite it: [`CITATION.cff`](CITATION.cff).

---

## Confession

The project began as:

> “Replace attention with a sheaf diffusion.”

The measurements have said three times that this is not the way.

What is left is smaller and much more defensible:

A research memory layer with an operator-level stability bound, a bounded in-memory register that returns retained token IDs bit-exactly, 50 machine-checked claims, and published measurements of its limitations. The register is not tamper-evident, encrypted, or tenant-isolated by default.

## Origin and acknowledgments

The project began with the author's idea of flowing information through a
topological structure. The author describes it as a school project and
credits AI assistance for implementation and repository organization. The
research decisions, measurements, and limitations are documented alongside
the code so the work can be evaluated and reproduced.
