Metadata-Version: 2.4
Name: zh-chapter-splitter
Version: 0.1.0
Summary: Zero-dependency, deterministic chapter splitter for Chinese web-novel TXT files (encoding detection, rule-fusion heading detection, quality report)
Project-URL: Homepage, https://github.com/felixchaos/zh-chapter-splitter
Project-URL: Issues, https://github.com/felixchaos/zh-chapter-splitter/issues
Author: Stellatrix Labs
License-Expression: MIT
License-File: LICENSE
Keywords: chapter,chinese,ingest,split,tokenize,txt,web-novel
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: Chinese (Simplified)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# zh-chapter-splitter

Zero-dependency, **deterministic** chapter splitter for Chinese web-novel `.txt`
files — encoding detection, rule-fusion heading recognition, and a rich quality
report. No LLM, no network, pure standard library. Battle-tested on a 4.85M-character
web novel.

- **No dependencies.** Pure standard library (`re`, `dataclasses`, `pathlib`, `statistics`).
- **Deterministic.** The same bytes always split the same way — no model, no randomness.
- **Encoding-aware.** Auto-detects UTF-8, UTF-8-BOM, UTF-16 (LE/BE), GB18030, GBK, Big5.
- **Rule fusion.** Preset heading rules plus one self-derived rule compete by a
  5-dimension structural score; the best split wins and over-long blocks are
  re-scanned for missed headings.
- **Honest reports.** Every split returns a diagnostics dict: confidence, problem
  category, per-chapter anomalies, author-note flags, weird-title flags, and
  ordinal gaps — nothing is silently deleted.

## Install

```bash
pip install zh-chapter-splitter
```

Requires Python 3.10+.

## Quickstart

```python
from zh_chapter_splitter import split_chapters_with_report

text = open("novel.txt", encoding="utf-8").read()
chapters, report = split_chapters_with_report(text)

for ch in chapters[:3]:
    print(ch["chapter_number"], ch["title"], len(ch["content"]))

print(report["mode"], report["confidence"], report["problem_label"])
```

**Split straight from a file (encoding auto-detected):**

```python
from zh_chapter_splitter import split_file

chapters, report, text = split_file("novel.txt")
print(report["encoding"])  # e.g. "gb18030"
```

**Just detect the encoding, or just clean the text:**

```python
from zh_chapter_splitter import decode_bytes, clean_text

text, encoding = decode_bytes(open("novel.txt", "rb").read())
cleaned = clean_text(text)  # strip pirate ads / mojibake, normalize whitespace
```

Each chapter is a plain dict:

```python
{
    "title": str,                    # heading line, e.g. "第12章 归途"
    "content": str,                  # chapter body
    "chapter_number": int,           # 1-based, contiguous
    "volume_title": str,             # owning volume heading, or ""
    "is_author_note": bool,          # author's aside (leave note / afterword)
    "exclude_from_extraction": bool, # == is_author_note
    "title_confidence": float,       # 0..1; low means a meme/quip title
    "content_descriptor": str,       # first-sentence descriptor for low-confidence titles
}
```

## API

| Function | Signature | Description |
| --- | --- | --- |
| `split_chapters_with_report` | `(text, *, split_rule="auto", custom_pattern="", source_name="", title="") -> (list[dict], dict)` | Split text into chapters and return a diagnostics report. |
| `split_chapters` | `(text, ...) -> list[dict]` | Same, chapters only. |
| `split_file` | `(path, *, split_rule="auto", custom_pattern="", title="") -> (list[dict], dict, str)` | Read a file, auto-detect encoding, split; report includes `encoding`. |
| `decode_bytes` | `(content: bytes) -> (str, str)` | Detect encoding and decode to `(text, encoding_label)`. |
| `clean_text` | `(text: str) -> str` | Normalize whitespace and strip pirate ads / mojibake. |
| `build_custom_pattern` | `(template: str) -> re.Pattern \| None` | Compile a `*`-template or raw regex into a heading pattern (ReDoS-guarded). |

Lower-level building blocks are also exported: `sanitize_corpus`,
`sanitize_corpus_text`, `filter_non_content`, `annotate_weird_titles`,
`adaptive_split`, `structural_score`, `split_by_heading_regex`, `extract_seq`,
`build_candidate_rules`, and the `ChapterSplitter` class (with its
`chapter_splitter` singleton).

### Split rules

`split_rule` defaults to `"auto"` (rule fusion). You can also force a preset:
`chapter_cn`, `corpus`, `chapter_en`, `number_dot`, `paren_num`, or `custom`
(with `custom_pattern`, where `*` stands for a chapter number, e.g. `卷一-第*章`).

## Command line

Installs a `zh-chapter-split` console script (pure `argparse`):

```bash
# Split into one .txt per chapter plus a report.json.
zh-chapter-split novel.txt -o out/

# Print the chapter-title list only, write nothing.
zh-chapter-split novel.txt --preview

# Force a preset rule or a custom template.
zh-chapter-split novel.txt -o out/ --split-rule chapter_cn
zh-chapter-split novel.txt -o out/ --split-rule custom --custom-pattern "卷一-第*章"
```

## How it works

1. **Decode & clean.** Detect the encoding (BOM-aware), normalize newlines and
   full-width spaces, then delete whole advertising lines, strip inline promo
   fragments, and drop garbled (mojibake) lines — all deterministically.
2. **Split.** In `auto` mode, rule-fusion adaptive splitting competes against a
   set of built-in detectors (numbered sections, pagination headings, strong /
   weak headings, fixed-window fallback). Candidates are ranked by a
   5-dimension structural score (ordinal continuity, size uniformity, count
   sanity, marker consistency, coverage) and the winner is chosen.
3. **Post-process & annotate.** Drop empty ghost chapters, window over-long
   blocks, dedupe exact repeats, then flag author notes and weird titles and
   reconcile ordinal gaps.
4. **Report.** Return the chapters plus a diagnostics dict for human review.

## License

MIT — see [LICENSE](LICENSE).

## Acknowledgements

Extracted from the RPG Roleplay platform (https://github.com/felixchaos/rpg-roleplay-platform).
中文说明见 [README.zh-CN.md](README.zh-CN.md).
