Metadata-Version: 2.4
Name: ntcompress
Version: 0.3.0
Summary: Pure-Python implementations of every Microsoft ESE record-compression scheme and Windows ntdll RtlCompressBuffer/RtlDecompressBuffer format.
Keywords: ese,esent,jet,ntdll,xpress,xpress9,xpress10,lznt1,lz4,ms-xca,compression,forensics
Author: ntcompress contributors
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Security
Classifier: Topic :: System :: Archiving :: Compression
Classifier: Typing :: Typed
Requires-Python: >=3.10
Project-URL: Repository, https://github.com/StrongWind1/ntcompress
Project-URL: Documentation, https://strongwind1.github.io/ntcompress/
Project-URL: Changelog, https://github.com/StrongWind1/ntcompress/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/StrongWind1/ntcompress/issues
Description-Content-Type: text/markdown

# ntcompress

<p align="center">
  <a href="https://github.com/StrongWind1/ntcompress/actions/workflows/ci.yml"><img src="https://github.com/StrongWind1/ntcompress/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/ntcompress/"><img src="https://img.shields.io/pypi/v/ntcompress.svg" alt="PyPI"></a>
  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.10+-blue.svg" alt="Python 3.10+"></a>
  <a href="https://github.com/StrongWind1/ntcompress/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License: Apache 2.0"></a>
  <a href="https://strongwind1.github.io/ntcompress/"><img src="https://img.shields.io/badge/docs-mkdocs-blue.svg" alt="Docs"></a>
</p>

Pure-Python, standard-library-only implementations of every compression format used by Microsoft's Extensible Storage Engine (ESE/ESENT, the "JET Blue" engine behind `NTDS.dit`, Exchange stores, `Windows.edb`, `SRUDB.dat`, and `WebCacheV01.dat`) and Windows `ntdll.dll` compression (`RtlCompressBuffer` / `RtlDecompressBuffer`). No runtime dependencies beyond the standard library.

## Scope

This library covers two distinct compression layers used by Windows:

### ESE record compression (vs `esent.dll`)

ESE stores a one-byte header on each compressed record cell; the top 5 bits select the scheme (`first_byte >> 3`) and the low 3 bits carry scheme-specific flags.

| ESE ID | Format | Decompress | Compress | Byte-Identical to esent.dll | Authority |
|--------|--------|-----------|----------|---------------------------|-----------|
| `0x0` | NONE (uncompressed sentinel) | — | — | — | `compression.cxx:504` |
| `0x1` | 7-bit ASCII | yes | yes | yes | ESE source |
| `0x2` | 7-bit Unicode | yes | yes | yes | ESE source |
| `0x3` | XPRESS (Plain LZ77) | yes | yes | yes | `[MS-XCA] §2.3/2.4` |
| `0x4` | SCRUB (erase marker) | produce | recognize | n/a | ESE source |
| `0x5` | XPRESS9 | yes | yes | yes* | ESE MIT source (ported) |
| `0x6` | XPRESS10 (LZ4 + CRC) | yes | yes | yes** | ESE source |
| `0x7` | LZ4 (standard block) | yes | yes | yes | LZ4 block format |

\*XPRESS9 encoder matches the MIT ESE C reference byte-for-byte excluding the non-deterministic session signature (`CRC32(__rdtsc())`), which even `esent.dll` varies on every call. \*\*XPRESS10 verified byte-identical by component construction (LZ4 block + CRC-32C + CRC-64/NVME each individually proven); not triggerable on commodity hardware (requires Corsica/QAT flight flag).

### ntdll compression (vs `RtlCompressBuffer`)

Windows `ntdll.dll` exposes a separate set of `CompressionFormatAndEngine` IDs used by SMB, NTFS, RDP, and other protocols. These use raw bitstreams with no ESE framing.

| ntdll ID | Name | Decompress | Compress | Byte-Identical to ntdll | Builds |
|----------|------|-----------|----------|------------------------|--------|
| `0x0002` | LZNT1 | yes | yes | yes (ENGINE_MAX) | XP SP3 – Server 2025 |
| `0x0003` | XPRESS (Plain LZ77) | yes | yes | yes | Win8.1 – Server 2025 |
| `0x0004` | XPRESS_HUFF (LZ77+Huffman) | yes | yes | yes (default engine) | Win8.1 – Server 2025 |
| `0x0005` | Compact XPRESS9 | yes | yes | — | Server 2022+ |
| `0x0006` | XP10 (LZ4 block) | yes | yes | yes | Win11 / Server 2025 |
| `0x0007` | raw DEFLATE | yes | yes | yes (both levels) | Win11 / Server 2025 |
| `0x0008` | ZLIB | yes | yes | yes (both levels) | Win11 / Server 2025 |

Format 0x0005 is an undocumented XPRESS9-family Huffman LZ77 codec (magic `0xC039E510`) with a compact 10-byte header, reverse-engineered from `ntdll.dll` Build 20348. It uses the same core algorithm as ESE XPRESS9 (magic `0x4E86D72A`) but with different outer framing. Format 0x0006 (XP10) is the same LZ4 block codec used by ESE scheme 0x6 and 0x7, without the ESE header.

## Design principles

- **Standard library only at runtime.** No third-party dependencies. `zlib`/`lz4` are deliberately not used; the CRCs and the LZ4 block codec are implemented directly.
- **Two subpackages, one interface pattern.** `ntcompress.ese` for ESE record-compression, `ntcompress.ntdll` for ntdll standalone codecs. Both offer Shape A (enum dispatch) and Shape B (direct module import) APIs.
- **Traceable to a spec or a source line.** Every wire constant cites `[MS-XCA] §x.y` or an ESE `file:line`. Where the shipped C diverges from the spec text, both are documented.
- **Byte-identical verification.** Compress output is verified against gold vectors captured from `esent.dll` and `ntdll.dll` on 16 Windows builds (XP SP3 through Server 2025).
- **Typed and documented.** Python 3.11+, full type hints, Google-style docstrings, `ruff`-clean under `select = ["ALL"]`, `ty`-clean.

## Installation

Install from [PyPI](https://pypi.org/project/ntcompress/):

```sh
uv add ntcompress        # recommended
pip install ntcompress    # or with pip
```

Or install from source:

```sh
uv add git+https://github.com/StrongWind1/ntcompress
```

Requires Python >= 3.11.

## Usage

### ESE record compression

```python
import ntcompress.ese as ese

# Decompress a cell (format is in byte 0)
plaintext = ese.decompress(cell)
size = ese.decompressed_size(cell)

# Compress with a specific format
cell = ese.compress(plaintext, ese.Format.XPRESS)
cell = ese.compress(plaintext, ese.Format.LZ4)
cell = ese.compress(plaintext, ese.Format.XPRESS9)

# Or use Shape B (direct module access)
from ntcompress.ese import xpress
cell = xpress.compress(plaintext)
plaintext = xpress.decompress(cell)
```

### ntdll standalone codecs

```python
import ntcompress.ntdll as ntdll

# Shape A (enum dispatch)
compressed = ntdll.compress(data, ntdll.Format.LZNT1)
plaintext = ntdll.decompress(compressed, ntdll.Format.LZNT1)

# Windows constant aliases
compressed = ntdll.compress(data, ntdll.COMPRESSION_FORMAT_XPRESS_HUFF)

# Shape B (direct module access)
from ntcompress.ntdll import lznt1, xpress_huff, xpress9, xp10, deflate
from ntcompress.ntdll import zlib as ntdll_zlib

plaintext = lznt1.decompress(compressed_stream)
compressed = xpress_huff.compress(data)
plaintext = xpress9.decompress(stream)         # ntdll 0x0005 (compact XPRESS9)
compressed = xp10.compress(data)               # ntdll 0x0006 (LZ4 block)
compressed = deflate.compress(data)            # ntdll 0x0007
compressed = ntdll_zlib.compress(data)         # ntdll 0x0008
```

## License and attribution

Apache-2.0 (`LICENSE`). The XPRESS9 codec is an attributed port of Microsoft's MIT-licensed ESE source; the Plain LZ77, Xpress Huffman, and LZNT1 codecs are derived from the published `[MS-XCA]` specification; the LZ4 block codec follows the public LZ4 format. Full notices are in `THIRD-PARTY-NOTICES.md`.
