Metadata-Version: 2.4
Name: terselib
Version: 0.1.0
Summary: Open-source implementation of the z/OS TERSE (AMATERSE / PACK / SPACK) compression format
Author: TerseLib contributors
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/mainframed/terselib
Project-URL: Documentation, https://terselib.readthedocs.io
Project-URL: Repository, https://github.com/mainframed/terselib
Project-URL: Issues, https://github.com/mainframed/terselib/issues
Keywords: terse,amaterse,mainframe,z/OS,compression,ebcdic,lzw,lzmw
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Archiving :: Compression
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: full
Requires-Dist: ebcdic>=1.1; extra == "full"
Requires-Dist: python-magic>=0.4; extra == "full"
Provides-Extra: docs
Requires-Dist: sphinx>=7; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=2; extra == "docs"
Dynamic: license-file

# TerseLib

A Python, implementation of the **z/OS TERSE** compression format 
(as produced by **AMATERSE** and the older **TRSMAIN**), written from scratch.

Compress files on Linux/macOS/Windows and unpack them with AMATERSE on the
mainframe — or terse a data set on z/OS and decompress it anywhere.

* Both compression methods: **PACK** (LZW) and **SPACK** (LZMW) are supported
* Host (12-byte header) and native (4-byte header) containers
* Fixed (F/FB) and variable (V/VB) record formats, including RDW framing
* Text (EBCDIC ⇄ ASCII) and binary rendering, with selectable code pages
* Smart decompression: auto-detects text vs binary content and picks an
  output file extension by content sniffing (libmagic)
* Validated against the IBM `tersedecompress` test corpus: byte-identical
  output in both directions for every PACK/SPACK, FB/VB, text/binary
  combination (only exceptions: two corpus files that AMATERSE itself
  produced broken)

> **Why no off-the-shelf LZW library?** TERSE's PACK is a nonstandard LZW:
> fixed 12-bit codes, a dictionary that starts *full* (pre-seeded with
> EBCDIC blanks) and least-recently-used slot recycling instead of a
> dictionary reset. SPACK is LZMW, which has no maintained Python
> implementation. Both codecs are therefore implemented here, straight
> from the spec. The [`ebcdic`](https://pypi.org/project/ebcdic/) package
> (extra code pages) and
> [`python-magic`](https://pypi.org/project/python-magic/) (file-type
> detection) are used when installed.

## Install

```console
$ pip install terselib                # from PyPI; add [full] for ebcdic + magic
$ pip install terselib[full]
```

Or from a checkout (this directory contains `pyproject.toml`):

```console
$ pip install .
```

## Command line

```console
$ terse report.txt                     # -> report.txt.trs (SPACK, text, VB)
$ terse -b image.png                   # binary, lossless (FB LRECL 1)
$ terse -m pack --recfm f -l 80 x.txt  # PACK, fixed 80-byte records

$ unterse DATA.TRS                     # auto: text/binary + extension sniffing
$ unterse -b DATA.TRS out.bin          # force exact binary output
$ unterse -t -c cp037 DATA.TRS out.txt # text via a specific code page
$ unterse -i DATA.TRS                  # show header info only
```

`terselib compress|decompress|info` is an umbrella alias for the same
tools, also reachable as `python -m terselib`.

### Moving files to/from z/OS

Transfer `.trs` files in **binary** (`ftp> binary`). On z/OS, allocate the
receiving data set as `RECFM=FB LRECL=1024` (the file size is always a
multiple of 1024), then unpack with:

```jcl
//UNPACK  EXEC PGM=AMATERSE,PARM='UNPACK'
//SYSUT1   DD DISP=SHR,DSN=YOUR.TERSED.FILE
//SYSUT2   DD DSN=YOUR.OUTPUT,DISP=(NEW,CATLG),...
```

## Library

```python
import terselib

# compress / decompress bytes
blob = terselib.compress(b"HELLO\nWORLD\n", text=True)     # SPACK, VB
result = terselib.decompress(blob)                          # auto-detect
assert result.data == b"HELLO\nWORLD\n" and result.text

# work with files; binary output gets a sniffed extension
terselib.compress_file("payload.bin", text=False, variable=False,
                        record_length=1)
path, result = terselib.decompress_file("payload.bin.trs")

# inspect a header without decompressing
print(terselib.read_header("DATA.TRS").describe())

# verbose logging (off by default)
terselib.enable_verbose_logging()
```

See the [documentation](docs/) (Sphinx / Read the Docs) for the full API
reference.

## Folder layout

```
.
├── pyproject.toml     packaging metadata; run pip install from here
├── src/terselib/      the library + terse/unterse/terselib CLIs
├── tests/              stdlib unittest suite
├── scripts/            corpus_check.py -- validates against a reference
│                        z/OS test corpus (see below)
└── docs/               Sphinx sources (readthedocs.io)
```

## Development

```console
$ python3 -m unittest discover -s tests            # unit tests
$ python3 scripts/corpus_check.py PATH/TO/tersedecompress-testdata \
      --skip-large --roundtrip                     # corpus validation
```

`scripts/corpus_check.py` needs a checkout of the `tersedecompress`
project's `tersedecompress-testdata` (not included in this repo) — real
AMATERSE-compressed files plus their z/OS-decompressed references, used to
validate this implementation byte-for-byte in both directions.

## License

GPL-3.0-or-later, see [LICENSE](LICENSE).
