Metadata-Version: 2.4
Name: wordstats
Version: 1.3.0
Summary: Multilingual word frequency statistics for Python, from subtitle corpora and (optionally) wordfreq
Home-page: https://github.com/zeeguu-ecosystem/Python-Wordstats
Download-URL: https://github.com/zeeguu-ecosystem/Python-Wordstats/archive/v_1.0.7.tar.gz
Author: Mircea Lungu
Author-email: mircea.lungu@gmail.com
License: MIT
Keywords: natural language processing,multilingual
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing :: Linguistic
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: configobj
Requires-Dist: sqlalchemy
Provides-Extra: wordfreq
Requires-Dist: wordfreq<3.2,>=3.1.1; extra == "wordfreq"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: download-url
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: summary


Statistics about word frequencies in different languages based on a corpus of 
movie subtitles as extracted by the Frequency Words (https://github.com/hermitdave/FrequencyWords) project.

Currently supported languages (or language codes to be more precise :): 

    "da", "de", "el", "en", "es", "fr", "it", "nl", "no", "pl", "pt", "ro", "zh-CN" 


### Usage Examples


##### Getting the info about a given word 

    >> from wordstats import Word
    >> print (Word.stats('bleu', 'fr'))
    bleu: (lang: fr, rank: 1521, freq: 9.42, imp: 9.42, diff: 0.03, klevel: 2)
    

##### Comparing the difficulty of two German words

    >> from wordstats import Word
    >> Word.stats('blauzungekrankenheit','de').difficulty > Word.stats('blau','de').difficulty
    True
    
    
##### Top 10 most used words in Dutch

    >> from wordstats import LanguageInfo
    >> Dutch = LanguageInfo.load('nl')
    >> print(Dutch.all_words()[:10])
    ['ik', 'je', 'het', 'de', 'dat', 'is', 'een', 'niet', 'en', 'van']

##### Words common across all the languages

Given that the corpus is based on subtitles, some common names have sliped in.
The `common_words()` function returns a list.

    >> from wordstats.common_words import common_words
    >> for each in common_words():
    >>     if len(each) > 9:
    >>         print(each)
    washington
    christopher
    enterprise


##### Words that are the same in Polish and Romanian

    >> from wordstats import LanguageInfo
    >> Polish = LanguageInfo.load("pl")
    >> Romanian = LanguageInfo.load("ro")
    >> for each in Polish.all_words():
    >>     if each in Romanian.all_words():
    >>         if len(each) > 5 and each not in common_words():
    >>             print(each)
    telefon
    moment
    prezent
    interes
    ...


##### How common a word is, according to wordfreq

`Word.stats()` is based on subtitles. For text like news, where subtitles call
everyday words rare, `Word.zipf_frequency()` uses the lists of
[wordfreq](https://github.com/rspeer/wordfreq), which combine many sources. It
answers exactly as `wordfreq.zipf_frequency` does: log10 of the occurrences per
billion words, so 3 is once per million and 7 is "the".

    >> from wordstats import Word
    >> Word.zipf_frequency('hus', 'da')
    5.26

Needs the `wordfreq` extra (see Installation).


### Memory

Every list, wordstats' own and wordfreq's, is built once into a read-only SQLite
file and read through mmap, so all processes on a machine share one copy in the
OS page cache instead of each holding its own (for all of wordstats' languages:
~6 MB per process instead of ~1.1 GB). `WORDSTATS_CACHE_DIR` sets where the
files go; by default inside the package, or the temp dir if that isn't writable.
Point it at a persistent volume to build each file only once across containers.


### Installation

    pip install wordstats

or, with `Word.zipf_frequency`:

    pip install 'wordstats[wordfreq]'
