Metadata-Version: 2.4
Name: taglid
Version: 0.1.1
Summary: A word-level Language Identification (LID) tool for Tagalog-English (Taglish) text.
Author-email: Andrian Lloyd Maagma <maagmaandrian@gmail.com>
License-Expression: MIT
Project-URL: homepage, https://github.com/andrianllmm/taglid
Project-URL: repository, https://github.com/andrianllmm/taglid
Project-URL: issues, https://github.com/andrianllmm/taglid/issues
Keywords: Language Identification,LID,Taglish,Code-switching,Code-mixed,Tagalog,English
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: lemminflect>=0.2.3
Requires-Dist: nltk>=3.8.1
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pandas>=2.2.2
Requires-Dist: symspellpy>=6.7.7
Requires-Dist: tabulate>=0.9.0
Requires-Dist: tglstemmer>=0.1.0
Dynamic: license-file

<div align="center">

# TagLID

**A word-level Language Identification (LID) tool for Tagalog-English (Taglish) text**

[![PyPI version](https://img.shields.io/pypi/v/taglid.svg?style=flat)](https://pypi.org/project/taglid/)
[![Downloads](https://pepy.tech/badge/taglid)](https://pepy.tech/project/taglid)
[![License](https://img.shields.io/github/license/andrianllmm/taglid?style=flat)](https://github.com/andrianllmm/taglid/blob/main/LICENSE)
[![GitHub stars](https://img.shields.io/github/stars/andrianllmm/taglid?style=flat)](https://github.com/andrianllmm/taglid/stargazers)
[![CI](https://github.com/andrianllmm/taglid/actions/workflows/ci.yml/badge.svg)](https://github.com/andrianllmm/taglid/actions/workflows/ci.yml)

</div>

## About

TagLID labels each word in a Taglish (Tagalog-English mix) text by language. It
gives either a simple tag (`tgl` or `eng`) or detailed frequency info with
flags indicating how the word was identified. It is a rule-based, opinionated
system that relies mostly on dictionary lookups, with additional logic for
skipping numbers, names, and interjections, and for handling slang,
abbreviations, contractions, stemming and lemmatizing inflected words,
intrawords, and misspelling correction.

## Installation

```sh
pip install taglid
```

## Usage

TagLID can be used as a library via `import taglid`, or as a CLI application
via `python -m taglid`.

### Library

#### Textual data

Use the `lid` module for textual data.

Use `lang_identify` to identify each word in a text. It takes any string and
returns a list of words with their corresponding English and Tagalog scores,
flag, and correction.

```python
from taglid.lid import lang_identify

labeled_text = lang_identify("hello, mundo")
print(labeled_text)
```

Output:

```
[{'Word': 'hello', 'eng': 1.0, 'tgl': 0.0, 'Flag': 'DICT', 'Correction': None}, {'Word': 'mundo', 'eng': 0.0, 'tgl': 1.0, 'Flag': 'DICT', 'Correction': None}]
```

Use [`tabulate`](https://pypi.org/project/tabulate/) to view the output in
tabular format.

```python
from tabulate import tabulate

print(tabulate(labeled_text, headers="keys"))
```

Output:

```
word      eng    tgl  flag    correction
------  -----  -----  ------  ------------
hello       1      0  DICT
mundo       0      1  DICT
```

Use `simplify` to reduce the output to only the words and their language. It
takes the return value of `lang_identify` and returns a list of tuples
containing the word and its language.

```python
from taglid.lid import simplify

simplified_text = simplify(labeled_text)
print(simplified_text)
```

Output:

```
[('hello', 'eng'), ('mundo', 'tgl')]
```

#### Datasets

Use the `lid_dataset` module for datasets.

Use `lang_identify_df` to label each word in each cell of a
[`pandas`](https://pypi.org/project/pandas/) DataFrame. It takes a DataFrame
of multiple rows and columns, where each cell contains textual data, and
returns a labeled DataFrame where each token is a row labeled by its original
row, original column, and token index.

```python
import pandas as pd
from taglid.lid_dataset import lang_identify_df

data = [["hello po", "ano?"], ["mag-aask lang po", "what?"]]

df = pd.DataFrame(data)

labeled_df = lang_identify_df(df)
print(labeled_df)
```

Output:

```
     col  token_index      word  eng  tgl  flag correction
row
0      0            1     hello  1.0  0.0  DICT       None
0      0            2        po  0.0  1.0  DICT       None
0      1            1       ano  0.0  1.0  FREQ       None
1      0            1  mag-aask  0.5  0.5  INTW       None
1      0            2      lang  0.0  1.0  FREQ       None
1      0            3        po  0.0  1.0  DICT       None
1      1            1      what  1.0  0.0  DICT       None
```

### CLI

Run TagLID from the terminal.

```sh
python -m taglid.lid
```

Then type a sentence when prompted.

```
text: hello, mundo
```

Output:

```
word      eng    tgl  flag    correction
------  -----  -----  ------  ------------
hello       1      0  DICT
mundo       0      1  DICT
```

Add `--simplify` to only show the words and their language.

```sh
python -m taglid.lid --simplify --text hello, mundo
```

Output:

```
-----  ---
hello  eng
mundo  tgl
-----  ---
```

Use `lid_dataset` with Excel files to directly label spreadsheets.

```sh
python -m taglid.lid_dataset in_path out_path
```

## Accuracy

The accuracy hasn't been tested yet.

## Development

This project uses [uv](https://docs.astral.sh/uv/) for dependency management.

Clone the repo and sync dependencies (including dev and test groups):

```sh
git clone https://github.com/andrianllmm/taglid.git
cd taglid
uv sync --all-groups
```

Run the tests:

```sh
uv run pytest
```

## Contributing

Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for details.

## License

Distributed under the [MIT License](LICENSE).
</content>
