Metadata-Version: 2.4
Name: zh-catmut
Version: 0.1.0
Summary: Native LUT remapping for large Pandas categorical code buffers
Author: Mohamed Hossam
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/mohamedhossammohamed/zh_catmut
Project-URL: Repository, https://github.com/mohamedhossammohamed/zh_catmut
Project-URL: Documentation, https://mohamedhossammohamed.github.io/zh_catmut/
Project-URL: Issues, https://github.com/mohamedhossammohamed/zh_catmut/issues
Project-URL: Author, https://github.com/mohamedhossammohamed
Project-URL: X, https://x.com/MohamedHz72007
Keywords: pandas,categorical,numpy,zig,ffi,native-extension,dataframe
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Zig
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: twine; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Provides-Extra: release
Requires-Dist: build; extra == "release"
Requires-Dist: cibuildwheel; extra == "release"
Requires-Dist: pytest; extra == "release"
Requires-Dist: twine; extra == "release"
Dynamic: license-file

# zh_catmut

![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)
![Native Core](https://img.shields.io/badge/native-Zig-orange)
![DataFrame](https://img.shields.io/badge/pandas-categorical-green)
![License](https://img.shields.io/badge/license-Apache--2.0-blue)

**Remap huge Pandas categorical columns without expanding them into object arrays or allocating boolean masks.**

## Before / After

```python
# Before: high-level remapping can allocate temporary arrays or trigger
# categorical re-encoding work on the full column.
series = series.cat.rename_categories({"pending": "active"})
series = series.cat.set_categories(["active", "disabled"])

# After: build a dense metadata LUT, mutate integer categorical codes natively,
# and return a refreshed categorical object.
from zh_catmut import remap_categorical

series = remap_categorical(
    series,
    {"pending": "active", "disabled_old": "disabled"},
    copy_fallback=True,
)
```

## Installation

For released wheels:

```bash
pip install zh-catmut
```

For a source checkout:

```bash
python -m pip install .
```

Verify the installed wheel and bundled native library:

```bash
zh-catmut doctor
```

The import package name uses an underscore:

```python
import zh_catmut
```

## Quickstart

```python
import pandas as pd
from zh_catmut import remap_categorical

s = pd.Series(pd.Categorical(["new", "old", "old", None]))
out = remap_categorical(s, {"old": "new"}, copy_fallback=True)
```

`copy_fallback=True` is the safest default for application code because it allocates a Python-owned destination codes buffer and keeps the original object unchanged.

## Why it's fast

- Pandas categoricals store the full column as dense integer `codes`; category labels are only metadata.
- `zh_catmut` builds a small dense `int64` lookup table from category labels in Python.
- A bundled Zig shared library receives only raw pointers, lengths, primitive flags, and POD reports through a C ABI.
- The native loop remaps contiguous `int8`, `int16`, `int32`, or `int64` code buffers in one batch.
- `ctypes.CDLL` is used so the GIL is released while the native remap executes.
- Python runs shape, dtype, contiguity, writeability, and Copy-on-Write safety gates before exporting pointers.
- Native validation is all-or-nothing: invalid source codes, invalid target codes, null pointers, and oversized lengths are rejected before mutation.
- If in-place mutation is unsafe, `copy_fallback=True` uses a Python-owned destination buffer and keeps the source buffer unchanged.

## Public API

```python
from zh_catmut import remap_categorical, remap_codes_inplace
```

Use `remap_categorical` for Pandas objects and `remap_codes_inplace` only when you already own a writable, contiguous NumPy codes buffer and a dense `int64` LUT.

### `remap_categorical`

```python
remap_categorical(
    obj,
    mapping,
    *,
    assume_unique=False,
    copy_fallback=False,
    threads=0,
)
```

- `obj`: a Pandas `Series` with categorical dtype or a `pd.Categorical`.
- `mapping`: mapping from old labels to new labels. Labels not present in the mapping are preserved.
- `copy_fallback`: when `True`, allocate a destination codes buffer and avoid in-place ownership risk.
- `assume_unique`: expert-only escape hatch for controlled in-place mutation after all other gates pass.
- `threads`: reserved for future native parallelism. The current V1 kernel accepts `0` or `1` and uses the scalar remap path.

### `remap_codes_inplace`

```python
remap_codes_inplace(
    codes,
    lut,
    *,
    target_category_count,
    missing_code=-1,
    allow_missing=True,
    threads=0,
)
```

- `codes`: one-dimensional, C-contiguous NumPy array with dtype `int8`, `int16`, `int32`, or `int64`.
- `lut`: one-dimensional, C-contiguous `np.int64` lookup table.
- `target_category_count`: every mapped non-missing code must be in `[0, target_category_count)`.
- `missing_code`: default `-1`, matching Pandas categorical codes.
- `allow_missing`: when `False`, missing input or missing LUT output is rejected before mutation.

## Command-line help for installed users

`zh-catmut` installs a small diagnostic CLI:

```bash
zh-catmut --help
zh-catmut info
zh-catmut example
zh-catmut doctor
python -m zh_catmut doctor
```

- `info` prints the project purpose, install command, API summary, and support links.
- `example` prints a minimal `copy_fallback=True` Pandas example.
- `doctor` verifies NumPy/Pandas imports, native library loading, ABI version, and low/high-level remap smoke tests.
- If your Python user scripts directory is not on `PATH`, use `python -m zh_catmut ...` instead of `zh-catmut ...`.

## Testing and native transparency

The native implementation is tracked in `native/src/root.zig`, with the C ABI declared in `include/zh_catmut_abi.h`. Tests are included so the compiled shared library is not a black-box artifact:

```bash
python -m pip install -e ".[test]"
python -m pytest -q
cd native && zig build test
```

The Python tests load the packaged native library, verify the ABI, exercise every supported integer code dtype, check all-or-nothing validation, run the Pandas copy-fallback path, and execute `python -m zh_catmut doctor`.

## Troubleshooting

### `NativeLibraryLoadError`

Run:

```bash
zh-catmut doctor
```

If the package was installed from a wheel, reinstall:

```bash
python -m pip install --force-reinstall zh-catmut
```

If pip built from source, Zig must be installed and visible as `zig` on `PATH`, or set:

```bash
ZIG=/path/to/zig python -m pip install zh-catmut
```

### Copy-on-Write safety errors

For ordinary Pandas application code, prefer:

```python
out = remap_categorical(series, mapping, copy_fallback=True)
```

Use `assume_unique=True` only when you control the object lifetime and know the underlying categorical codes buffer is not shared.

### Supported runtime inputs

- Python 3.9+
- CPython
- NumPy + Pandas
- Pandas categorical codes backed by contiguous signed integer arrays
- Code dtypes: `int8`, `int16`, `int32`, `int64`

## Links

- GitHub profile: [mohamedhossammohamed](https://github.com/mohamedhossammohamed)
- X profile: [@MohamedHz72007](https://x.com/MohamedHz72007)
- License: [Apache-2.0](LICENSE)
