Metadata-Version: 2.4
Name: pawzip
Version: 1.0.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Rust
Classifier: Topic :: System :: Archiving :: Compression
Classifier: Typing :: Typed
Requires-Dist: numpy>=1.24
Requires-Dist: maturin>=1.7,<2 ; extra == 'dev'
Requires-Dist: pytest>=8 ; extra == 'dev'
Requires-Dist: pytest-cov>=5 ; extra == 'dev'
Requires-Dist: hypothesis>=6.100 ; extra == 'dev'
Requires-Dist: ruff>=0.6 ; extra == 'dev'
Requires-Dist: mypy>=1.10 ; extra == 'dev'
Requires-Dist: pre-commit>=3 ; extra == 'dev'
Requires-Dist: build>=1 ; extra == 'dev'
Requires-Dist: twine>=5 ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
Summary: Lossless compressor that records where each byte value occurs and describes those positions with series. Writes .paw files.
Keywords: compression,compressor,lossless,file format,burrows-wheeler,range coder
Author: Pawel Stalowczyk
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/Stalowczyk/pawzip/blob/main/CHANGELOG.md
Project-URL: Format specification, https://github.com/Stalowczyk/pawzip/blob/main/spec/FORMAT.md
Project-URL: Issues, https://github.com/Stalowczyk/pawzip/issues
Project-URL: Source, https://github.com/Stalowczyk/pawzip

# pawzip

pawzip is a lossless compressor that writes `.paw` files. Instead of storing data byte by
byte, it records where each byte value appears and describes those positions with arithmetic
series and nested loops where it can, storing the rest as compactly coded gaps.

On the Silesia benchmark it compresses better than bzip2 and nearly as well as xz, but it is
slower than both (see Limitations). It does best on structured and repetitive data, and data
it can't compress, such as noise or files that are already compressed, is stored as it is.

```
pip install pawzip
```

Requires Python 3.10+ and NumPy. The wheels include the compressor's core compiled from Rust;
installing from source needs a Rust toolchain ([rustup.rs](https://rustup.rs)). Setting the
environment variable `PAW_PURE=1` switches to the pure-Python reference implementation
(same output, much slower).

## Python API

```python
import paw

data = b"hello world\n" * 1000
blob = paw.compress(data)  # bytes -> .paw bytes
assert paw.decompress(blob) == data  # .paw bytes -> the original bytes

meta = paw.info(blob)  # read the headers without decompressing (a path works too)
print(meta.original_size, meta.compressed_size, meta.ratio, meta.mode_summary)

paw.compress_file("notes.txt")  # writes notes.paw, remembering ".txt"
paw.decompress_file("notes.paw", "copy.txt")  # without a destination: notes.txt

with paw.open("log.paw", "wt", encoding="utf-8", extension="txt") as f:  # like gzip.open
    f.write("hello\n")
```

- `compress(data, modes=None)` tries every suitable mode and keeps the smallest result. Pass
  `modes=[paw.Mode.MAP]` (or another `paw.Mode`) to try only those modes, which is faster.
- `decompress(blob, max_size=None)` verifies a CRC-32 of every block. **For untrusted
  input, pass `max_size`**: a tiny file can claim an enormous size, and `max_size`
  refuses it before any memory is allocated.
- Decoding a block needs about 7x its size in memory (20x in pure Python), so blocks
  larger than `max_block_size` (default 64 MiB) are refused the same way. Files written
  with the default block size are never affected; pass a larger `max_block_size` (CLI
  `--max-block-size`) for files written with bigger blocks.
- File names: `photo.png` compresses to `photo.paw`, and the extension `png` is stored
  inside it. Decompressing gives back `photo.png`, or `abc.png` if you renamed the file to
  `abc.paw`. Only the extension is stored, never the name. For `compress` and
  `compress_stream`, pass `extension="png"` yourself.
- Already-compressed data (JPEG, ZIP, video) is recognised from a small sample and stored
  as it is, in a fraction of a second. `thorough=True` (CLI `--thorough`) tries every mode
  anyway: many times slower, for a percent or so.
- `compress_file` / `decompress_file` never overwrite an existing file unless
  `overwrite=True`, and never leave a partial output file behind on failure.
- Data is split into independent blocks (default 16 MiB, `block_size=`), so files of
  any size can be compressed and decompressed with bounded memory. `compress_file`,
  `decompress_file`, `paw.open`, `compress_stream` and `decompress_stream` all stream.
- `compress_file`, `decompress_file`, `compress_stream` and `decompress_stream` accept a
  `progress` callback: it is called after each block, and once at the end, with the
  number of input bytes processed so far (handy for a progress bar).
- Parallel compression and decompression are opt-in: `workers=4` (CLI `-T 4`), or
  `workers="auto"` to use the CPU cores that fit into half of the currently free memory.
  Files with several blocks gain: Webster in 4 MiB blocks compresses at 13.6 MB/s and
  decompresses at 78 MB/s with 4 workers. The default is a single worker, and `"auto"`
  also uses one where free memory can't be measured (currently macOS; pass a number
  there). Workers are threads; only the pure-Python fallback uses processes, which need
  the usual `if __name__ == "__main__":` guard on Windows/macOS.
- Errors: `paw.FormatError` (not a valid or a damaged .paw file) and `paw.SizeLimitError`
  (over `max_size` or `max_block_size`), both subclasses of `paw.PawError`, which is a
  `ValueError`.

The package is fully type-annotated.

## Command line

```
paw compress notes.txt                 # -> notes.paw
paw decompress notes.paw               # -> notes.txt (also after renaming it)
paw test archive.paw                   # check it decompresses correctly
paw info archive.paw                   # sizes, blocks and modes
cat data.bin | paw compress - > data.bin.paw
paw decompress --max-size 500M -c big.paw > big
paw compress -T auto -B 8M huge.iso    # parallel, 8 MiB blocks
paw compress --thorough photo.jpg      # squeeze already-compressed files too (slow)
```

When stderr is a terminal, a progress line shows how far each file has got and roughly how
long is left (`-q` hides it). Run `paw COMMAND --help` for all options. Exit status is 0 on
success, 1 if any file failed, 2 on usage errors and 130 if interrupted.

## Limitations

- **Speed** (Rust core, Silesia benchmark): on one core, compression runs at about
  1.7 MB/s, close to xz -9, and decompression at about 18 MB/s, a little slower than bzip2
  and about 3x slower than xz. Compression tries its modes in parallel, so with several
  cores it is faster: about 3.5 MB/s on a 4-core Intel Core i3-12100F. The
  pure-Python reference (`PAW_PURE=1`) is about 50x slower.
- **Memory**: compressing needs up to about 100x the block size per worker (about 1.6 GB
  for a full 16 MiB block, when several modes are tried at once); decompressing needs
  about 7x (about 110 MB; 20x in pure Python). Use a smaller `block_size` on machines
  with little memory.
- The `.paw` format v1 is frozen: files written now will decode with every later version.

