Metadata-Version: 2.4
Name: wordninja-pp
Version: 0.3.1
Summary: Probabilistically split concatenated words across 90 languages — incl. English, Spanish, French, German, Italian, Portuguese, Russian, Hebrew, Chinese, Japanese, Korean, Vietnamese, Arabic, Hindi, Bengali, Tamil, Telugu, Thai, Greek, Turkish, Polish, Dutch, Swedish, Norwegian, Finnish, Marathi, Punjabi, Gujarati, Kannada, Burmese, Swahili, Amharic, Welsh, Breton, Mongolian and 50+ more
Home-page: https://github.com/primepake/wordninja-pp
Author: primepake
License: MIT
Keywords: nlp,word-splitting,segmentation,multilingual,vietnamese,chinese,japanese,korean,arabic,hindi,tamil,telugu,thai,polish,dutch,swedish,turkish,language-detection,tokenization,text-processing
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Natural Language :: English
Classifier: Natural Language :: Spanish
Classifier: Natural Language :: French
Classifier: Natural Language :: German
Classifier: Natural Language :: Italian
Classifier: Natural Language :: Portuguese
Classifier: Natural Language :: Russian
Classifier: Natural Language :: Hebrew
Classifier: Natural Language :: Chinese (Simplified)
Classifier: Natural Language :: Japanese
Classifier: Natural Language :: Korean
Classifier: Natural Language :: Vietnamese
Classifier: Natural Language :: Arabic
Classifier: Natural Language :: Bengali
Classifier: Natural Language :: Bulgarian
Classifier: Natural Language :: Catalan
Classifier: Natural Language :: Croatian
Classifier: Natural Language :: Czech
Classifier: Natural Language :: Danish
Classifier: Natural Language :: Dutch
Classifier: Natural Language :: Esperanto
Classifier: Natural Language :: Finnish
Classifier: Natural Language :: Greek
Classifier: Natural Language :: Hindi
Classifier: Natural Language :: Hungarian
Classifier: Natural Language :: Indonesian
Classifier: Natural Language :: Persian
Classifier: Natural Language :: Polish
Classifier: Natural Language :: Romanian
Classifier: Natural Language :: Slovak
Classifier: Natural Language :: Slovenian
Classifier: Natural Language :: Swedish
Classifier: Natural Language :: Tamil
Classifier: Natural Language :: Telugu
Classifier: Natural Language :: Thai
Classifier: Natural Language :: Turkish
Classifier: Natural Language :: Ukrainian
Classifier: Natural Language :: Urdu
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

![image](https://user-images.githubusercontent.com/2049665/29219793-b4dcb942-7e7e-11e7-8785-761b0e784e04.png)

Word Ninja
==========

Slice your munged together words!  Seriously, Take anything, `'imateapot'` for example, would become `['im', 'a', 'teapot']`.  Useful for humanizing stuff (like database tables when people don't like underscores).

This project is repackaging the excellent work from here: http://stackoverflow.com/a/11642687/2449774

Usage
-----
```
$ python
>>> import wordninja
>>> wordninja.split('derekanderson')
['derek', 'anderson']
>>> wordninja.split('imateapot')
['im', 'a', 'teapot']
>>> wordninja.split('heshotwhointhewhatnow')
['he', 'shot', 'who', 'in', 'the', 'what', 'now']
>>> wordninja.split('thequickbrownfoxjumpsoverthelazydog')
['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']
```

Performance
-----------
It's super fast!

```
>>> def f():
...   wordninja.split('imateapot')
... 
>>> timeit.timeit(f, number=10000)
0.40885152100236155
```

It can handle long strings:
```
>>> wordninja.split('wethepeopleoftheunitedstatesinordertoformamoreperfectunionestablishjusticeinsuredomestictranquilityprovideforthecommondefencepromotethegeneralwelfareandsecuretheblessingsoflibertytoourselvesandourposteritydoordainandestablishthisconstitutionfortheunitedstatesofamerica')
['we', 'the', 'people', 'of', 'the', 'united', 'states', 'in', 'order', 'to', 'form', 'a', 'more', 'perfect', 'union', 'establish', 'justice', 'in', 'sure', 'domestic', 'tranquility', 'provide', 'for', 'the', 'common', 'defence', 'promote', 'the', 'general', 'welfare', 'and', 'secure', 'the', 'blessings', 'of', 'liberty', 'to', 'ourselves', 'and', 'our', 'posterity', 'do', 'ordain', 'and', 'establish', 'this', 'constitution', 'for', 'the', 'united', 'states', 'of', 'america']
```
And scales well.  (This string takes ~7ms to compute.) 

How to Install
--------------

```
pip3 install wordninja
```

Custom Language Models
----------------------
#1 most requested feature!  If you want to do something other than english (or want to specify your own model of english), this is how you do it.

```
>>> lm = wordninja.LanguageModel('my_lang.txt.gz')
>>> lm.split('derek')
['der','ek']
```

Language files must be gziped text files with one word per line in decreasing order of probability.

If you want to make your model the default, set:

```
wordninja.DEFAULT_LANGUAGE_MODEL = wordninja.LanguageModel('my_lang.txt.gz')
```
