Metadata-Version: 2.4
Name: ts-postagger
Version: 0.1.0
Summary: Turkish Part-of-Speech tagger
Author: Taner Sezer
License: MIT
Project-URL: Homepage, https://github.com/tanerim/ts_postagger
Keywords: turkish,nlp,pos-tagging,tokenization,spacy
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: spacy<4.0,>=3.8
Requires-Dist: ts-tokenizer
Dynamic: license-file

# TS POSTagger

TS POSTagger is a Turkish part-of-speech tagging library with a hybrid pipeline:

1. [`ts-tokenizer`](https://pypi.org/project/ts-tokenizer/) produces token boundaries and deterministic token classes.
2. A bundled spaCy POS model predicts contextual grammatical tags.
3. A resolver preserves structural token types such as hashtags, mentions, URLs, dates, and punctuation when they are more informative than a generic grammatical label.

The package exposes:

- a Python API: `from ts_postagger import pos`
- a CLI: `ts-postagger`

## Installation

```bash
pip install ts-postagger
```

Requirements:

- Python `>=3.11`

The trained model is bundled with the package. No separate download step is required.

## Quick Start

```python
from ts_postagger import pos

tokens = pos("Defne'nin heyecanla beklediği #viyana yolculuğu bugün başladı.")

for token in tokens:
    print(token.text, token.pos)
```

Example output:

```text
Defne'nin   PropN
heyecanla   Adv
beklediği   Adj
#viyana     Hashtag
yolculuğu   Noun
bugün       Adv
başladı     Verb
.           Punc
```

## Python API

The main entrypoint is `pos(text: str) -> list[TSToken]`.

Each returned `TSToken` has these fields:

| Field | Description |
| --- | --- |
| `text` | Original surface form |
| `lower` | Turkish-aware lowercase form |
| `token_type` | Deterministic token class from [`ts-tokenizer`](https://pypi.org/project/ts-tokenizer/) |
| `tag` | Contextual grammatical prediction from the model |
| `pos` | Final output POS label |

`pos` is the field you should use as the final annotation.

### Minimal example

```python
from ts_postagger import pos

text = pos("Bugün yeni ve güzel bir gün!")

for token in text:
    print(token.pos)
```

### Convert results to dictionaries

`TSToken` is a dataclass, so standard dataclass helpers work:

```python
from dataclasses import asdict

from ts_postagger import pos

tokens = pos("#YeniBilgi yayımlandı.")
rows = [asdict(token) for token in tokens]

for row in rows:
    print(row)
```

Example dictionary:

```python
{
    "text": "#YeniBilgi",
    "lower": "#yenibilgi",
    "token_type": "Hashtag",
    "tag": "Noun",
    "pos": "Hashtag",
}
```

### Empty input

```python
from ts_postagger import pos

print(pos(""))
```

Output:

```python
[]
```

## Why `token_type`, `tag`, and `pos` are different

The library intentionally keeps multiple annotation layers.

For lexical tokens, the final output usually follows the POS model:

```text
çalışmalar  Valid_Word  Noun  Noun
yayımlandı  Valid_Word  Verb  Verb
```

For structural or social-media tokens, the final output stays deterministic even when the model predicts a regular grammatical tag:

```text
#YeniBilgi  Hashtag  Noun  Hashtag
@yeni  Mention  Noun  Mention
19.10.2026  Date  Num  Date
https://example.org  URL  Noun  URL
```

Meaning of each layer:

- `token_type`: deterministic label from the tokenizer
- `tag`: raw contextual prediction from the POS model
- `pos`: final POS output of TS POSTagger

## Turkish-aware lowercasing

The `lower` field uses Turkish-aware lowercasing from [`ts-tokenizer`](https://pypi.org/project/ts-tokenizer/).

```python
from ts_postagger import pos

tokens = pos("IĞDIR İSTANBUL")

for token in tokens:
    print(token.text, token.lower)
```

Output:

```text
IĞDIR  ığdır
İSTANBUL  istanbul
```

`lower` is a lowercase surface form. It is not a lemma.

## CLI

Installing the package also installs the `ts-postagger` command.

The CLI accepts either:

- a single positional text argument, or
- standard input

Default output format:

```text
TOKEN<TAB>POS
```

### Tag inline text

```bash
ts-postagger "Bugün yeni ve güzel bir gün!"
```

Example output:

```text
Bugün	Adv
yeni	Adj
ve	Conj
güzel	Adj
bir	Det
gün	Noun
!	Punc
```

### Lowercase only

```bash
ts-postagger -low "Bugün yeni ve güzel bir gün!"
```

Example output:

```text
bugün
yeni
ve
güzel
bir
gün
!
```

### Raw model tag

```bash
ts-postagger -tag "Bugün yeni ve güzel bir gün!"
```

Example output:

```text
Bugün	Adv
yeni	Adj
ve	Conj
güzel	Adj
bir	Det
gün	Noun
!	Punc
```

### Full output

```bash
ts-postagger --full "Bugün yeni ve güzel bir gün!"
```

Example output:

```text
Bugün	bugün	Adv
yeni	yeni	Adj
ve	ve	Conj
güzel	güzel	Adj
bir	bir	Det
gün	gün	Noun
!	!	Punc
```

Columns:

```text
TOKEN<TAB>LOWER<TAB>POS
```

### Read from stdin

```bash
echo "Bugün yeni ve güzel bir gün!" | ts-postagger
```

### Version

```bash
ts-postagger -V
```

### Help

```bash
ts-postagger --help
```

## Notes

- The package name for installation is `ts-postagger`.
- The Python import package is `ts_postagger`.
- The CLI command is `ts-postagger`.

## Citation

If you use TS POSTagger in academic work, please cite the associated doctoral dissertation:

> Sezer, T. (2025). [*Dizilerden birimlere: Bilişimsel dilbilim çerçevesinde bir birimlendirici tasarımı*](https://tez.yok.gov.tr/UlusalTezMerkezi/TezGoster?key=Xau5rw3KuCgEuy-FuJQtsNVGSOOMCSQba2T5bZaDSDUTfOiTTVCpuBZPjDrUgB0i) [Doctoral dissertation, Hacettepe University].

## License

This project is licensed under the [MIT License](./LICENSE).
