Metadata-Version: 2.5
Name: torch-sql
Version: 0.1.0rc2
Summary: PyTorch programs as SQL on XQL (prototype)
Project-URL: Homepage, https://github.com/xqlsystems/torch-sql
Project-URL: Documentation, https://github.com/xqlsystems/torch-sql/blob/main/docs/design.md
Project-URL: Issues, https://github.com/xqlsystems/torch-sql/issues
Author-email: Alex Merose <al@merose.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: autodiff,datafusion,duckdb,pytorch,sql
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: ddxdb<0.5,>=0.4.10
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=21
Requires-Dist: torch>=2.15
Provides-Extra: datafusion
Requires-Dist: ddxdb[datafusion]<0.5,>=0.4.10; extra == 'datafusion'
Provides-Extra: duckdb
Requires-Dist: ddxdb[duckdb]<0.5,>=0.4.10; extra == 'duckdb'
Description-Content-Type: text/markdown

# torch-sql

torch-sql is a skunkworks prototype that runs PyTorch programs, or parts of
them, as SQL on [XQL](https://xql.systems), a group of database projects for
scientific arrays. It has three frontends:

- a co-processor, which leaves the model in PyTorch and takes over the
  training data pipeline and evaluation;
- a device, `device="sql"`, on which every tensor is a table;
- a `torch.compile` backend.

Its first real model is [Samudra](https://github.com/m2lines/Samudra), an
ocean emulator.

The first milestone, M1 in the design, is under way: a two-layer MLP, then
nanoGPT (Andrej Karpathy's small GPT), trained on `device="sql"`. Start with
the docs:

- [docs/design.md](docs/design.md): the design.
- [docs/samudra-on-beta.md](docs/samudra-on-beta.md): the production plan for
  Samudra on Empire AI's Beta cluster.
- [docs/spikes/](docs/spikes/README.md): small programs that answer the
  design's open questions, with their results.

## Install

torch-sql is on PyPI, as an alpha. Install it with an engine, DuckDB or
DataFusion, to run anything:

```bash
pip install "torch-sql[duckdb]"       # or torch-sql[datafusion], or both
```

It needs Python 3.12 or later and PyTorch 2.15 or later.

## Development

torch-sql uses [uv](https://docs.astral.sh/uv/). From the repository root:

```bash
uv sync --all-extras   # PyTorch (CPU), ddxdb, DataFusion and DuckDB
uv run pytest          # every test runs on both engines
uv run ruff check && uv run ruff format --check
```

On NixOS, first put `libstdc++` and `zlib` on `LD_LIBRARY_PATH`, as the XQL
repositories' `flake.nix` files do.

## Example

nanoGPT's `model.py`, unchanged, trains on `device="sql"`
and matches PyTorch step by step (`tests/test_nanogpt.py`). A smaller
example, a two-layer MLP run on DuckDB:

```python
import torch, torch_sql

with torch_sql.Environment("duckdb"):
    mlp = torch.nn.Sequential(torch.nn.Linear(3, 5), torch.nn.Tanh(), torch.nn.Linear(5, 2))
    mlp = mlp.double().to("sql")
    y = mlp(torch.randn(4, 3, dtype=torch.float64, device="sql"))
    print(y)  # runs one SQL query on DuckDB
```
