Metadata-Version: 2.5
Name: langfamilysim
Version: 0.1.0
Summary: Language similarity computation based on linguistic family and geographic factors.
Project-URL: Homepage, https://github.com/TangoGroup/language-similarity
Project-URL: Issues, https://github.com/TangoGroup/language-similarity/issues
Author: Sean Boisen
License-Expression: Apache-2.0
License-File: DATA_SOURCES.md
License-File: LICENSE
License-File: NOTICE
Keywords: bible,glottolog,language-similarity,languages,linguistics,low-resource
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.12
Requires-Dist: babel>=2.18.0
Requires-Dist: numpy>=2.5.2
Requires-Dist: pycountry>=26.2.16
Requires-Dist: rapidfuzz>=3.9.0
Description-Content-Type: text/markdown

# langfamilysim

Estimate how similar languages are, three ways:

- **`langfamilysim`**: genealogical similarity from [Glottolog](https://glottolog.org/)'s
  classification tree, with a geographic-distance damped variant. Low-resource
  language in, closest related languages out.
- **`lwc`**: for a language, which *languages of wider communication* its
  speakers probably also speak (a directional bilingualism probability, e.g.
  Isanzu → Swahili), from Glottolog and Unicode CLDR data.
- **`biblesim`**: empirical similarity from verse-aligned Bibles in the
  [eBible corpus](https://github.com/BibleNLP/ebible), comparing words
  (Levenshtein), cognate rate, verse length and word order.

Requires Python 3.12+.

```bash
pip install langfamilysim
```

## Quick start

### Languages of wider communication (works out of the box)

```python
from lwc.lookup import LWCLookup

lwc = LWCLookup()
for x in lwc.likely_lwcs("isn", k=3):      # ISO 639-3 or glottocode
    print(x.name, x.probability, x.country)   # Swahili 0.9 TZ, English 0.69 TZ
```

### Genealogical similarity (build the graph first)

The full graph (`similarity.db`, about 290 MB) is too large for PyPI, so you
build it from Glottolog. From a source checkout:

```bash
git clone https://github.com/TangoGroup/language-similarity.git
cd language-similarity/src/langfamilysim
./fetch_glottolog.sh      # sparse clone of the Glottolog tree, under a minute
python3 build.py          # writes nodes.csv, edges.csv, similarity.db (~20 s)
```

```python
from similarity import LanguageSimilarity, name_lookup

names, iso_to_gc = name_lookup()
sim = LanguageSimilarity()
for other, score, kind in sim.top_related(iso_to_gc["spa"], k=10, by="combined"):
    print(names[other], score, kind)
```

Each edge has a tree-based `score` and a `combined` score damped by
geographic distance (`by="score"` ignores geography). See
`src/langfamilysim/README.md` for the method.

### Bible-based similarity

Download the eBible corpus separately (it is not included), then:

```bash
export BIBLESIM_CORPUS=/path/to/ebible/corpus
python -m biblesim compare spa por
python -m biblesim --stride 10 family indo1319 --langs spa,por,fra,ita,ron
python -m biblesim validate        # Spearman correlation vs the Glottolog scores
```

Language families come from the `similarity.db` built above (`--langdb`).
See `src/biblesim/README.md`.

## Data and licensing

The code is Apache-2.0. The data it ships or derives comes from third parties
under their own licenses: Glottolog and WALS (CC BY 4.0) and Unicode CLDR
(Unicode License). eBible translations are licensed individually, so check
each before redistributing any text. Details, credit lines and citations are in
[`DATA_SOURCES.md`](DATA_SOURCES.md).

Geographic data is approximate: Glottolog gives one point per language, so
distances are only a rough proxy for regional multilingualism.

## Development

```bash
uv sync
uv run pytest
```

## License

Apache License 2.0. See [`LICENSE`](LICENSE) and [`NOTICE`](NOTICE).
