Metadata-Version: 2.4
Name: cotangent
Version: 0.1.0
Summary: A deep learning framework built from scratch. Reverse-mode autograd, tensors, CUDA, and a transformer trained on top of it.
License: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: numpy>=1.26
Provides-Extra: cuda
Requires-Dist: cupy-cuda12x; extra == 'cuda'
Provides-Extra: dev
Requires-Dist: graphviz; extra == 'dev'
Requires-Dist: hypothesis; extra == 'dev'
Requires-Dist: matplotlib; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: torch; extra == 'dev'
Description-Content-Type: text/markdown

# Cotangent

[![CI](https://github.com/u7k4rs6/Cotangent/actions/workflows/ci.yml/badge.svg)](https://github.com/u7k4rs6/Cotangent/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/github/license/u7k4rs6/Cotangent)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![CUDA: supported](https://img.shields.io/badge/CUDA-supported-76B900.svg)](cotangent/cuda/kernels.py)

Cotangent: a deep learning framework built from scratch. Autograd, tensors, CUDA, and a transformer trained on top of it.

Reverse-mode autodiff propagates *cotangents* backward through the computation graph, one `backward()` call at a time; the framework is named for exactly what it computes, not for a metaphor about the process.

![Cotangent vs PyTorch: char-GPT loss curves, overlaid, and their per-step residual](https://raw.githubusercontent.com/u7k4rs6/Cotangent/main/docs/parity.png)

Cotangent's loss curve and an identical PyTorch model's are indistinguishable (left); the residual (right) sits at the float64 machine floor, with max divergence **1.78e-15**, and does not grow.

The runtime framework is entirely self-contained: torch is not a runtime dependency. PyTorch appears only in the test suite, as a numerical oracle to validate every operation against; the `cotangent` package never imports it, enforced by a fail-closed CI grep gate (`.github/workflows/ci.yml`'s `no-unsafe-deserialization` job).

## What this is

A deep learning framework built from scratch in NumPy and CuPy: reverse-mode automatic differentiation over a dynamically-built computation graph, an N-dimensional `Tensor` with NumPy-style broadcasting, a CPU/GPU device abstraction with one hand-written CUDA kernel, SGD/Adam/AdamW, an `nn.Module` layer, and a char-level GPT (causal self-attention, weight tying, pre-norm residual blocks) trained on top of it. Every implemented op is verified against PyTorch to float64 machine precision (`1e-10`), not "close enough": see `docs/gradcheck.md` and `docs/design.md` for the numbers.

![Cotangent's package layering: a shared autograd core, device dispatch underneath it as a cross-cutting concern, and ops/nn/optim/GPT layered on top, not a linear pipeline](https://raw.githubusercontent.com/u7k4rs6/Cotangent/main/docs/architecture.png)

This is not a linear `Tensor -> Autograd -> CUDA -> nn -> Optimizer -> GPT` pipeline. `Tensor` and `Function`/`Context` are the same layer (`Function`s *are* the autograd nodes); device dispatch (`cotangent/core/device.py`) sits underneath everything as a cross-cutting concern every op resolves through, not a stage any op passes through once; and the ops library, `nn`, `optim`, and the GPT are genuinely layers, each built on the one below it, with `nn` and `optim` as siblings rather than one stacked on the other. `cotangent/scalar/` (the 150-line teaching engine below) is deliberately not part of this stack at all.

```python
import cotangent as ct

x = ct.tensor([[2.0, 3.0]], requires_grad=True)
w = ct.randn(2, 1, requires_grad=True)
loss = (x @ w).relu().sum()
loss.backward()
print(x.grad)
```
```
Tensor([[-0.27666122,  2.0209932 ]], dtype=float32)
```
Run verbatim, this session: `w` is unseeded here, so the exact values differ run to run. The shape, dtype, and that it runs through the public `Tensor` method surface (`@`, `.relu()`, `.sum()`, `.backward()`) do not.

## How it's built

Scalar autograd (`cotangent/scalar/`, a 150-line teaching artifact that stays in the repo) → an `N`-dimensional `Tensor` engine with `Function`/`Context` and full broadcasting → a CPU/GPU device abstraction (`xp` dispatch over NumPy/CuPy), optimizers, and `nn.Module` → a char-level GPT trained on TinyShakespeare. The GPT's *forward/backward math* is proven identical to PyTorch's (the plot above); its *trained loss* is a separate, honestly-short-of-target number; see "What's proven, and what isn't" below before assuming the second follows from the first. `docs/plan/02_TECHNICAL_ARCHITECTURE.md` section 9 has the full 40-step build order this repo's commit history follows one step per commit.

![A Value computation graph from cotangent/scalar/, rendered after backward(): each node shows its forward data and the gradient backward() computed for it](https://raw.githubusercontent.com/u7k4rs6/Cotangent/main/docs/graph.png)

The scalar engine's own graph, rendered after `backward()`: two leaves (`data=2`, `data=3`) feed a multiply and an add, both feeding a final multiply, and every node carries the gradient `backward()` actually computed for it, not a hand-worked example. `docs/graph.dot` is the checked-in, reproducible source; see "Install and run" below to regenerate it.

## What's proven, and what isn't

**Proven, and reproducible from this repo:**
- Every implemented op (forward, backward, gradcheck, torch-parity) verified to `1e-10` in float64. `docs/gradcheck.md` (`pytest --report`) has the per-op table: 36 ops, worst-case max relative error `2.7e-6`, most in the `1e-8`–`1e-9` range.
- The assembled GPT (causal mask, multi-head attention, weight tying, pre-norm blocks) matches an identical PyTorch model to the float64 floor over a real training run: the plot above, `1.78e-15` max residual, not merely "close."
- An MLP trained purely on Cotangent reaches **96.38%** test accuracy on MNIST, CPU only:

  ![MNIST training curve: train loss and test accuracy per epoch](https://raw.githubusercontent.com/u7k4rs6/Cotangent/main/docs/mnist_curve.png)

  `examples/mnist_curve.py` (5 epochs, batch_size=64, lr=0.1, plain SGD, the same run `examples/mnist_mlp.py` trains) writes this plot and `docs/mnist_curve.csv` together; the 96.38% above is this run's own last data point, not a separately quoted number.
- `grep -rn "import torch" cotangent/` is clean, checked in CI.

**Not claimed:**
- **The GPT does not hit the paper-loss target.** Trained 8000 steps (warmup + cosine LR decay, causal char-GPT, `d=128`, 4 heads, 4 layers) on TinyShakespeare: final validation loss **2.965 bits/char**, against a target of `<1.6`. Stated as-is, not hidden, not dressed up. `docs/design.md`'s "Where the gaps are" section has the full diagnosis: three separate pieces of direct evidence point at a model-capacity/data-scale limit, not a schedule or correctness bug (a flat-LR run and the scheduled run land at nearly the same loss at matched step count; the loss plateaus while LR is still actively decaying; weight tying's own measured regularization cost is `~0.4` bits/char). The model does generate real structure at this scale (character-name-style tags, dialogue-shaped line breaks, real short words), which is the qualitative bar a model this size clears; `1.6` bits/char is a further, quantitatively distinct bar this repo does not claim to have cleared.
- **The custom CUDA kernel is not fast.** `cotangent/cuda/kernels.py`'s hand-written tiled matmul loses to cuBLAS by roughly `5x`–`7x` once past the smallest problem size, the gap widening with problem size (see the benchmark chart below). It exists to prove the raw-CUDA capability, not to compete with a production BLAS. `docs/design.md` names the four missing optimizations (tensor cores, register blocking, bank-conflict-free layout, occupancy autotuning) and why each matters.
- **Not a production framework.** No operator fusion, no memory planner, no `torch.compile` equivalent, no in-place ops, no higher-order gradients, no distributed training, no mixed precision. `docs/design.md`'s "Where the gaps are" section quantifies what this costs (e.g. `~8%` measured graph-construction overhead per forward pass) and states what PyTorch does instead, for each gap.

## The benchmark

`benchmarks/matmul.py`, CPU (NumPy) vs. GPU (CuPy/cuBLAS) vs. the hand-written kernel, real GFLOP/s on an RTX 4060 Laptop:

![CUDA matmul benchmark: CPU vs cuBLAS vs the hand-written kernel, GFLOP/s, log scale](https://raw.githubusercontent.com/u7k4rs6/Cotangent/main/docs/benchmark.png)

| n | CPU GFLOP/s | cuBLAS GFLOP/s | custom kernel GFLOP/s | custom / cuBLAS |
|---|---|---|---|---|
| 256 | 267.0 | 1219.9 | 713.5 | 1.7x |
| 512 | 330.1 | 4957.0 | 904.4 | 5.5x |
| 1024 | 391.4 | 6309.9 | 902.2 | 7.0x |
| 2048 | 368.7 | 6590.6 | 886.0 | 7.4x |

GFLOP/s varies somewhat run to run with this hardware's thermal/power state; the gap and its direction (cuBLAS pulling further ahead as the problem grows) is the stable, reproducible claim, and `docs/design.md` explains the mechanism. Every GPU number here is `cp.cuda.Stream.null.synchronize()`-gated before the clock stops (CuPy queues kernels asynchronously; an unsynchronized read times the empty queue, not the compute). `benchmarks/plot_matmul.py` writes the chart and `docs/benchmark.csv` together from a single run; the table above is that same run's numbers, not a separately quoted one.

## Install and run

```
pip install -e ".[dev]"
```

Runtime dependencies are exactly `numpy>=1.26` and, optionally, `cupy-cuda12x` (`pip install -e ".[cuda]"`) for the GPU path; `cotangent/` imports nothing else. `torch` is a `[dev]`-only dependency, used solely as the parity oracle in `tests/` and `examples/parity_gpt.py`; it is never imported under `cotangent/`.

`graphviz` in `[dev]` is the Python wrapper package only. It shells out to the `dot` binary at render time; pip cannot install that binary. Install it separately:

```
apt-get install graphviz  # Debian/Ubuntu
brew install graphviz     # macOS
```

Verify with `dot -V`. `docs/graph.png` is generated from `docs/graph.dot`, the checked-in, reproducible source of truth:

```
dot -Tpng docs/graph.dot -o docs/graph.png
```

Run the test suite (1134 tests, float64 gradcheck + torch parity throughout):

```
pytest
```

Regenerate `docs/gradcheck.md` from a real run:

```
pytest --report
```

Train the toy examples and regenerate the charts above:

```
python examples/xor.py          # M1: a 2-4-4-1 MLP, loss below 1e-3
python examples/mnist_mlp.py    # M2: 96.38% test accuracy, CPU only
python examples/mnist_curve.py  # writes docs/mnist_curve.png + .csv
python examples/train_gpt.py    # M4: the char-GPT, checkpointed to examples/.cache/
python benchmarks/plot_matmul.py  # writes docs/benchmark.png + .csv (requires CUDA)
```

## Further reading

- [`docs/design.md`](docs/design.md): the real writeup, covering why a tape and not a closure-based graph, the gradcheck tolerance taxonomy, the weight-tying invariant enforced at three separate boundaries, why the custom kernel loses to cuBLAS, and the full "Where the gaps are" honesty section.
- [`docs/gradcheck.md`](docs/gradcheck.md): the per-op gradcheck table, generated by `pytest --report`.
- [`docs/plan/`](docs/plan/): the four binding planning docs (PRD, architecture, security, frontend spec) this repo was built against, one build-order step per commit.
