Metadata-Version: 2.4
Name: khmercut
Version: 0.2.0
Summary: A (fast) Khmer word segmentation toolkit.
Home-page: https://github.com/seanghay/khmercut
Author: Seanghay Yath
Author-email: seanghay.dev@gmail.com
License: Apache License 2.0
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Requires-Python: >3.5
Description-Content-Type: text/markdown
Requires-Dist: python-crfsuite
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: coverage; extra == "test"
Provides-Extra: nn
Requires-Dist: numpy; extra == "nn"
Requires-Dist: onnxruntime; extra == "nn"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

### khmercut

A (fast) Khmer word segmentation toolkit.

```shell
pip install khmercut
```

### Python

```python
from khmercut import tokenize

tokenize("ឃាត់ខ្លួនជនសង្ស័យ០៤នាក់ ករណីលួចខ្សែភ្លើង នៅស្រុកព្រៃនប់")
# => ['ឃាត់ខ្លួន', 'ជនសង្ស័យ', '០៤', 'នាក់', ' ', 'ករណី', 'លួច', 'ខ្សែភ្លើង', ' ', 'នៅ', 'ស្រុក', 'ព្រៃនប់']
```

### Neural model

A distilled neural segmenter. More accurate, and it can split compound words.

```shell
pip install khmercut[nn]
```

```python
from khmercut.nn import tokenize

tokenize("ឃាត់ខ្លួនជនសង្ស័យ០៤នាក់")
# => ['ឃាត់ខ្លួន', 'ជនសង្ស័យ', '០៤', 'នាក់']

tokenize("ឃាត់ខ្លួនជនសង្ស័យ០៤នាក់", deep=True)
# => [['ឃាត់', 'ខ្លួន'], ['ជន', 'សង្ស័យ'], ['០៤'], ['នាក់']]
```

With `deep=True` each word becomes a list of its sub-words. Spaces are returned
as `' '` and newlines as `'\n'`.

Both backends load lazily, so importing one does not pull in the other.

### Reference

- [Khmer language processing toolkit](https://github.com/VietHoang1512/khmer-nltk)
- [Asian Language Treebank](http://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/index.html)
