Metadata-Version: 2.5
Name: codeccat-train
Version: 0.1.0
Summary: Offline trainer and evaluator for CodecCat native encoding models
Project-URL: Repository, https://github.com/Johnny-Kao/CodecCat
Project-URL: Issues, https://github.com/Johnny-Kao/CodecCat/issues
Project-URL: Documentation, https://github.com/Johnny-Kao/CodecCat/blob/main/native-prototype/training/docs/GETTING_STARTED.md
Author: Johnny Kao
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: General
Requires-Python: >=3.10
Requires-Dist: numpy>=2.0
Requires-Dist: scikit-learn<2,>=1.4
Description-Content-Type: text/markdown

# codeccat-train — independent offline training toolkit

Independent Python distribution for training and exporting CodecCat native
`.ccm` models. **Not a runtime dependency** of `codeccat`. The train
environment requires NumPy and scikit-learn; production inference does not.

**Release candidate: `codeccat-train 0.1.0` (independent package).** The core offline Trainer and end-to-end published-Runtime model-loading workflow are implemented and verified. This repository branch contains a stable release candidate; **PyPI availability is not implied until a separate verified publishing run succeeds**. Historical official L2/L3 reproduction establishes *reproducibility*, not a better model or permission to publish third-party training texts.

The same Trainer is used for both workflows: private community-supplied datasets
remain the user's responsibility; the project-maintained dataset is governed by
a separate rights and default-model acceptance decision. No user dataset or
generated model is automatically submitted to the CodecCat project.

From a checkout (works before and after the first PyPI publication):

```bash
python -m pip install ./native-prototype/training
python -m codeccat_train fit --manifest dataset/manifest.jsonl --output custom.ccm --report training-report.json
```

Dataset manifest format (one JSON object per line, paths relative to the
manifest file):

```json
{"path":"samples/a.bin","encoding":"utf-8","split":"train","group":"document-a"}
{"path":"samples/b.bin","encoding":"shift_jis","split":"validation","group":"document-b"}
```

The trainer checks strict decodability, rejects path traversal, conflicting
duplicate contents and train/validation source-group overlap. Supply varied,
lawfully obtained text samples with independently known encoding labels.
The training report stores both `manifest_sha256` (the manifest file bytes) and
`dataset_content_fingerprint_sha256` (a deterministic digest of accepted
input-file hashes, codec labels, group IDs and split assignments). Unlike a
manifest hash alone, the second value changes when an input file's content
changes at the same path. This does **not** prove dataset licensing or
independent holdout status, and does not expose raw training text.

The generated report describes model provenance and validation; it is **not**
a guarantee of accuracy on arbitrary real-world text.

See the [CodecCat repository](https://github.com/Johnny-Kao/CodecCat)
for the native runtime and [rights and provenance policy](https://github.com/Johnny-Kao/CodecCat/blob/main/RIGHTS_AND_PROVENANCE.md).

## Verification and release boundary

- **Implemented and CI-verified:** grouped source-manifest ingestion and isolation,
  deterministic same-environment CCM1/CCM2 training, native Runtime loading,
  source-content fingerprints, candidate-vs-Stable Unicode comparison,
  optional L3 training, exact pinned official historical L2/L3 replay.
- **Platform evidence:** training and Runtime interoperability checked on Linux
  x64/ARM64 and macOS ARM64; custom-model Runtime portability checked on
  macOS Intel and Windows x64.
- **Stable-candidate release boundary:** the independent `0.1.0` source/Wheel and release CI must pass before tagging. It is not available via PyPI until its own Trusted Publisher is configured and a separate upload succeeds. Historical training provenance is documented with residual risk, not universally cleared.
- **Not an automatic Runtime update:** installing or training with this tool
  does not alter `codeccat` or the bundled Stable model. A candidate default
  requires a separately approved evaluation and release.

CI evidence: [Trainer E2E](https://github.com/Johnny-Kao/CodecCat/actions/runs/42240463787),
[macOS ARM64 native trainer](https://github.com/Johnny-Kao/CodecCat/actions/runs/42237800042),
and [model submission standards](../../docs/MODEL_SUBMISSIONS.md).

## Evaluate a trained model

Install the public `codeccat==0.1.0` Runtime only in the evaluation environment, then compare against the official bundled model:

```bash
python -m codeccat_train evaluate --model custom.ccm \
  --manifest dataset/test-manifest.jsonl --split test --report comparison.json
```

This compares decoded Unicode text (primary) and exact codec labels (secondary), with per-codec results and paired improves/regressions. `--split validation` can be used for development, but **do not call it an independent holdout**. The command does not replace or update the official model.

## Optional L3 and historical reproducibility

For the optional L3 Unicode-trigram artifact, the **same offline Trainer**
supports a train-only manifest. Unlike the L2 `--residual-focus` replay,
L3 uses every eligible unfiltered input record:

```bash
codeccat-train fit-l3 --manifest dataset/train-only.jsonl \
  --output language-l3.json --report l3-report.json
```

L3 stays disabled by default. The pinned historical L2 CCM2 and L3 JSON
have both been reproduced byte-for-byte in GitHub Actions; see
[official source and reproducibility status](../../datasets/official/README.md).
The manual strict replay gate fails when required hashes, source
fingerprints or numeric identity evidence differ. This is **not** rights
clearance or authorization to replace a production model.

See [quick start](docs/GETTING_STARTED.md). The legacy `python -m codeccat_train --manifest ...` training command remains supported.

## Compact CCM2 export

The same trainer can export `CCM2` float32 weights instead of the default `CCM1` float64 weights:

```bash
codeccat-train fit --manifest dataset/manifest.jsonl --output custom-f32.ccm \
  --report train-f32.json --precision f32
```

`--precision f64` remains the default for compatibility. CCM2 stores IDF and bias as float64 while compressing the weight matrix to float32. A smaller model does not imply better accuracy; compare both models on independently held-out bytes before choosing one. This is a format option for custom models, **not** reproduction of the published official CCM2 model.
