Metadata-Version: 2.4
Name: pre_bpe_morph_tr
Version: 0.1.2
Project-URL: Homepage, https://github.com/Esat-Karakaya/zemberek-python/tree/llm_tooling
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: antlr4-python3-runtime==4.8
Requires-Dist: numpy>=1.19.0
Dynamic: license-file

This project is derived from the Python port of Zemberek by Loodos, which is itself based on the original Zemberek Java project by Ahmet A. Akın and Mehmet D. Akın.

# Goal of pre_bpe_morph
This package recognizes verbs, nouns, and named entities and their suffixes. Before BPE, this package removes suffixes and replaces them with their respective special tokens for identifiability. It also precedes a word type (Verb/Noun/NamedEntity) before any set of suffixes. The goal is to simplify Turkish grammar rules for small language models.

## Usage
```python
from pre_bpe_morph_tr import MorphTokenizer
tokenizer=MorphTokenizer("<|", "|>")

tokenizer.tokenize("gülüveriniz")
# response: ['g', 'ü', 'l', '<|Verb|>', '<|Hastily|>', '<|Req|>']

tokenizer.detokenize(['k', 'o', 'ş', '<|Verb|>', '<|Fut|>', '<|A1sg|>'])
# response: "koşacağım"
```

## Developed for language model
Since this is developed for LM training, I tried to avoid preprocessing text (like converting "hal" to "hâl", or lowercasing words). The text should remain unchanged when encoded and then decoded.
