Metadata-Version: 2.4
Name: glyd
Version: 0.21.0
Summary: Lossless AI compression: 33% less GPU memory, bit for bit. Models load with their weights packed on the GPU (glyd.from_pretrained); a codec for files and object storage.
License: BSD-3-Clause OR GPL-2.0-only; bundled libglyd_store BUSL-1.1; glyd.gpu BUSL-1.1
Project-URL: Homepage, https://getglyd.com
Project-URL: Documentation, https://getglyd.com/docs/
Project-URL: Source, https://github.com/surya-koritala/Glyd
Project-URL: Changelog, https://getglyd.com/changelog/
Project-URL: Issues, https://github.com/surya-koritala/Glyd/issues
Keywords: compression,lossless,llm,gpu,cuda,bfloat16,transformers,pytorch,object-storage
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Archiving :: Compression
Classifier: Environment :: GPU :: NVIDIA CUDA :: 12
Classifier: Environment :: GPU :: NVIDIA CUDA :: 13
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: COPYING
License-File: LICENSE-glyd-store
License-File: LICENSE-glyd-gpu
Provides-Extra: gpu
Requires-Dist: torch>=2.5; extra == "gpu"
Requires-Dist: transformers<6,>=5.17; extra == "gpu"
Requires-Dist: accelerate>=1.1; extra == "gpu"
Requires-Dist: safetensors>=0.8; extra == "gpu"
Requires-Dist: huggingface_hub<2,>=1.5; extra == "gpu"
Dynamic: license-file

# Glyd for Python

Lossless AI compression: 33% less GPU memory, bit for bit.

    pip install "glyd[gpu]"   # models on the GPU: Linux x86_64 / aarch64, an NVIDIA GPU (Ampere or later)
    pip install glyd          # the codec alone: Linux x86_64 / aarch64, macOS arm64

```python
import glyd
model = glyd.from_pretrained("Qwen/Qwen3-8B")               # packed on the GPU as it loads
model = glyd.from_pretrained("Qwen/Qwen3-8B", exact=True)   # logits bit for bit bf16's
```

Every option: [on the GPU](https://github.com/surya-koritala/Glyd/tree/main/bindings/python#on-the-gpu-a-models-weights-held-compressed-bit-for-bit) below, and
[getglyd.com](https://getglyd.com/docs/) for which models fit which GPU.

## The codec

Wheels for Linux x86_64 and aarch64 and macOS arm64. From a checkout:
`bindings/python/build.sh && pip install bindings/python`.

```python
import glyd
c = glyd.compress(data)                       # --max; level="ultra" / "cold" / "default"
c = glyd.compress(log_bytes, records=True)    # logs, dumps, CSV, JSON lines as typed columns
data = glyd.decompress(c)

p = glyd.pack(events)                         # many small objects as one stream
event = glyd.unpack(p, 7)

with glyd.Store("bucket/") as s:              # objects compressed across each other
    i = s.put("wed.tar", data)                # a delta against the object it most resembles
    data = s.get(i)
with glyd.Store("meta/", s3="s3://bucket/prefix") as s:   # objects in S3
    ...
```

A thin ctypes layer over `include/glyd.h` (BSD-3-Clause OR GPL-2.0); no build step
beyond placing the shared library. `build.sh` places `libglyd_store`,
which carries the codec and the store; with `libglyd` alone (the codec
crate, BSD-3-Clause OR GPL-2.0) everything but `Store` works.

## On the GPU: a model's weights held compressed, bit for bit

Needs a CUDA GPU (Ampere or later) and `pip install "glyd[gpu]"`
(PyTorch 2.5+ built for CUDA 12 or 13, transformers 5.17+, accelerate,
safetensors, huggingface_hub). The Linux wheels (x86_64, aarch64; glibc 2.28 or later)
carry the kernels, `libglyd_gpu_cuda12.so` and `libglyd_gpu_cuda13.so`
(CUDA 12.8 and 13.0, their runtime linked in), and the one for PyTorch's
CUDA is taken; `GLYD_GPU_LIB` names another. From a checkout,
`bash gpu/build_lib.sh bindings/python/glyd/gpu` builds the one for your
nvcc. `fit` needs none of it.

```python
import glyd
from transformers import AutoTokenizer
model = glyd.from_pretrained("Qwen/Qwen3-8B")            # any bf16 checkpoint, packed as it loads, on the GPU
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
out = model.generate(**tok("Hello", return_tensors="pt").to(model.device), max_new_tokens=64)

model = glyd.from_pretrained("Qwen/Qwen3-8B", exact=True) # logits bit-identical to bf16's
glyd.save_pretrained(model, "qwen3-8b-glyd")               # the packed format, loads without repacking
model = glyd.from_pretrained("qwen3-8b-glyd", verify=True) # re-hashes every weight against the source
print(glyd.fit("Qwen/Qwen3-32B", gpu="48GB"))              # will it fit, bf16 against Glyd
```

`glyd.from_pretrained(name_or_path, *, device="cuda:0", layout="auto", exact=False, merge=True, verify=False, **hf_kwargs)`
returns the transformers model (a causal LM, else an image-text-to-text
one), ready for `generate()`. transformers loads it a tensor at a time and
every Linear's weight is packed on the GPU as it arrives: the GPU holds
the packed model and the largest weight not packed yet, the host about a
shard.

- `device`: the GPU. `device_map="auto"` (with accelerate) or a device map
  in `hf_kwargs` spreads the layers over several.
- `layout`: `"auto"` picks for the GPU (the tiered layout on Ada, and
  wherever only it fits; the 12-bit one on A10, A100 and H100); `"mma"`,
  the tiered layout (10.8 bits a weight); `"mma12"`, the 12-bit layout
  (12.0 bits, a lighter decode). Embeddings go in the fast format, their
  rows decoded as they are looked up.
- `exact`: every product decodes its matrix whole and multiplies as
  `nn.Linear` does, so the logits are bf16's bit for bit. By default the
  products run straight from the packed weights (decoded in registers):
  faster, their sums in another order than cuBLAS's, so late tokens may
  differ from bf16's as between any two GEMM kernels.
- `merge`: q, k, v and gate, up as one product each, as serving engines
  run them (not with `exact`).
- `verify`: every pack decoded and compared with its weights bit for bit
  as it is made; from a saved checkpoint, every tensor decoded and its
  sha256 checked against `glyd.json`.
- `hf_kwargs`: transformers' `from_pretrained`'s (`revision`, `token`,
  `device_map`, `attn_implementation` ...); the dtype is bf16.

`glyd.gpu.compress(model, *, layout="auto", exact=False, merge=True)`
packs a model already loaded in bf16 in place, on the GPU its weights are
on (the current one for weights on the CPU), and returns it
(`glyd.compress` is the codec's, for bytes).

A mixture of experts (the Experts modules of transformers 5.17: OLMoE,
granite MoE ...) is packed too, each layer's experts as one matrix, and
run by `glyd`, an experts implementation registered with transformers;
with `exact=True` the experts the tokens are routed to are decoded and
bf16's own implementation runs on them. glyd-v1 holds no packed experts
yet, and a model with them can't be copied or pickled (`copy.deepcopy`,
`torch.save`): load it again.

Compiled: `model.generate(..., cache_implementation="static")` compiles
the forward as transformers does (`torch.compile`,
`mode="reduce-overhead"`: CUDA graphs), and `torch.compile(model.forward,
mode="reduce-overhead", fullgraph=True)` compiles it as it would the bf16
model's: each GLinear and GEmbedding is one op of the graph
(`glyd::linear`, `glyd::embedding`), with no graph break, and the CUDA
graph captures Glyd's kernels, so a step's host time goes and what is
left is its GPU time, less than bf16's. Tokens/s generating 128 tokens at
1 / 8 sequences on an RTX 4080 SUPER, `from_pretrained` as above against
transformers' bf16 model:

| | bf16 eager | Glyd eager | bf16 compiled | Glyd compiled |
| :--- | ---: | ---: | ---: | ---: |
| Qwen3-1.7B | 88.9 / 693 | 98.3 / 780 | 151.9 / 1020 | **187.8 / 1282** |
| Qwen3-4B-Instruct-2507 | 61.5 / 455 | 75.7 / 563 | 73.9 / 481 | **95.2 / 612** |
| Qwen3-8B | does not fit | 48.5 / 364 | does not fit | **55.6 / 386** |

Eager, a step's product is one C call, with less host time than
`nn.Linear`'s. Compile the whole forward, as these do: an op names its
module by a number the graph is specialized on, so compiling each layer
on its own compiles every layer anew (and meets torch._dynamo's
recompile limit). With `exact=True` each product in the graph is
`F.linear`'s, as eager, but the logits are the bf16 model's compiled the
same way bit for bit only with
`torch._inductor.config.emulate_precision_casts = True`: by default
Inductor keeps a bf16 value in fp32 across a fused kernel, and fuses
differently around Glyd's op than around bf16's matmul.

`glyd.save_pretrained(model, path)` writes glyd-v1: the packs in the
tiered layout as safetensors (each packed Linear's buffers under its
module path, `.glyd_data`, `.glyd_blocks`, `.glyd_block_base`), the rest
of the model as it is, `glyd.json` (the format, the source repo and
revision, and for every packed tensor its shape and the sha256 of its
bf16 bytes), and the source's config, generation config and tokenizer
files. `from_pretrained(path)` loads the packs as saved; on a GPU where
the 12-bit layout is the pick, it decodes and packs them again.

`glyd.fit(name_or_path, gpu="48GB", context=8192)`: whether the model
fits one GPU in bf16 and with Glyd, by the site's rule: the weights (with
Glyd, bf16 tensors at the ratio measured for the model, else 0.673 of
their bytes; FP8 and 4-bit as they are), a KV cache for `context` tokens
and 1.5 GiB for the runtime, against the memory nvidia-smi reports
(`16GB` ... `141GB`, or a number of bytes). From the Hub's metadata for a
repo id, from the files for a directory.

    python -m glyd.gpu fit Qwen/Qwen3-32B --gpu 48GB
    python -m glyd.gpu pack Qwen/Qwen3-8B qwen3-8b-glyd     # packed, checked, saved as glyd-v1
    python -m glyd.gpu verify qwen3-8b-glyd

`glyd.gpu` is under the Business Source License 1.1 (`LICENSE-glyd-gpu`),
as the rest of Glyd's GPU code; the codec under BSD-3-Clause OR GPL-2.0.
