Metadata-Version: 2.4
Name: cppjieba-mmap
Version: 0.1.0
Summary: Read-only mmap CppJieba with a merged Taiwan dictionary
License-Expression: Apache-2.0 AND MIT AND BSD-2-Clause AND BSD-3-Clause AND RSA-MD
Project-URL: Repository, https://github.com/coseto6125/cppjieba-mmap
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: C++
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: licenses/CPPJIEBA-MIT.txt
License-File: licenses/DARTS-BSD.txt
License-File: licenses/JIEBA-MIT.txt
License-File: licenses/LIMONP-MIT.txt
License-File: licenses/OPENCC-APACHE-2.0.txt
License-File: licenses/PYBIND11-BSD.txt
License-File: licenses/RSA-MD5.txt
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Dynamic: license-file

# cppjieba-mmap

Linux native Chinese tokenizer for CPython 3.13 and later. Version 0.1.0
bundles a merged Taiwan dictionary and uses CppJieba Mix segmentation with HMM.
The dictionary DAT uses a read-only shared memory mapping. Workers reuse its
file-backed pages instead of allocating a private dictionary trie per process.
HMM tables remain process-private.

Release wheels target manylinux_2_28 x86_64 and aarch64 for CPython 3.13,
3.14 and free-threaded 3.14. Wheels need no compiler at installation time.
Source builds use setuptools, pybind11 3.0.1 and a C++17 compiler, without CMake.
Runtime needs no OpenCC, jieba, network dictionary downloads or model server.

## Install from source

Run these commands from this repository root:

```sh
uv venv --python 3.14
uv pip install '.[test]'
.venv/bin/python -m pytest
```

```python
import cppjieba_mmap

tokenizer = cppjieba_mmap.Tokenizer(
    str(cppjieba_mmap.DICT_PATH),
    str(cppjieba_mmap.HMM_PATH),
    "/tmp/my-private-tokenizer-cache",
)
words = tokenizer.cut("超商取貨")
```

`DICT_PATH` and `HMM_PATH` are readable bundled `pathlib.Path` objects from
import time. `DICT_SHA256` identifies the exact uncompressed dictionary bytes.
The dictionary is packaged uncompressed. No runtime conversion occurs.
Its 582704 entries combine the pinned CppJieba dictionary with an OpenCC
`s2twp` conversion of the supplied jieba-next dictionary. Duplicate surfaces
retain the larger frequency. See `PROVENANCE.md` for exact inputs and hashes.

## Output contract

`cut(text: str) -> list[str]` uses `Jieba::Cut(text, words, true)` (Mix + HMM),
then removes whitespace-only tokens, matching the cpp-tw retrieval evaluation.
Other token text and ordering stay unchanged. English case is preserved.
Empty and whitespace-only inputs return `[]`. In particular, `256GB` remains
`256 / GB`; the wrapper does not silently alter this evaluated behavior.

## Cache and concurrency

Cache names contain dictionary SHA-256, the native cache format revision,
architecture and endianness. Changed dictionary bytes cannot select the same
DAT. HMM data is loaded separately and does not determine the DAT identity.

Every constructor locks a per-key `flock` file. Exactly one simultaneous
constructor builds a missing DAT. A private temporary directory holds the
dictionary snapshot and incomplete DAT. Publication uses an atomic rename
within the cache filesystem after validation, fsync and chmod 0444.
Readers attach using `PROT_READ | MAP_SHARED`. Locks remain on disk for reuse.
Truncated or header-mismatched cache files are rebuilt under the same lock.
Existing instances retain their mapped inode while a replacement is published.

Use a trusted local filesystem and one OS user for all workers. Cache directories
must not be writable by other users. Input models must be trusted and remain
unchanged while constructing a tokenizer. Dictionary bytes are snapshotted
even on cache hits, avoiding hash/read races. Temporary snapshots are deleted
after construction. A killed process can leave a private `.build-*` directory;
it is never treated as a published cache. Remove such directories only while
no constructors run. The package does not evict older DAT files.

## Threads and free threading

The binding releases the GIL during native construction and cutting. One
instance supports concurrent `cut` calls: the engine exposes no mutators,
dictionary/model lookups are const, and cut/DAG/Viterbi buffers are call-local.
The native module declares `py::mod_gil_not_used()`.
The process-global upstream `umask` mutation is removed. Constructors use
separate engine instances and per-cache file locks. Python result filtering
runs after the native call returns.

This is not a claim of subinterpreter support or lock-free Python execution.
Tests exercise two threads with a released GIL and three spawned processes.
HMM tables and per-call buffers remain process-private; only the DAT is shared.

## Limits

The DAT header is validated, but this is not a sandbox for malicious native
cache contents or untrusted model files. Do not reuse caches written by other
users. Inputs consume memory proportional to their length. No Windows support,
user-dictionary mutation, POS tagging API, query/full mode or cache eviction is
exposed. See `PROVENANCE.md` for source pins, patches and third-party licenses.

## Releases

CI builds and tests one CPython 3.14 x86_64 wheel on pushes and pull requests.
The release workflow tests all six Python/architecture wheels on native Linux
runners, builds an sdist, and publishes only after every build succeeds.

Before the first release, configure a PyPI Trusted Publisher for project
`cppjieba-mmap`, owner `coseto6125`, repository `cppjieba-mmap`, workflow
`release.yml`, and environment `pypi`. Configure that GitHub environment with
release approval protection. No PyPI API token is required.

Set the package version in `pyproject.toml`, then push the matching `v<version>`
tag. For this version the tag is `v0.1.0`. The workflow rejects a version mismatch.
Only the isolated publish job receives `id-token: write`.

## License

Package glue is MIT-licensed under `LICENSE`. Bundled source, dictionary and
conversion resources retain their own notices in `licenses/` and
`PROVENANCE.md`. The distribution's combined license expression includes these
third-party licenses; it does not relicense them as MIT.
