Metadata-Version: 2.4
Name: glove-word-embeddings
Version: 0.1.30
Summary: Preprocess, validate, categorize, rate, and embed words for academic research (Wang et al., 2026, Olson et al., 2021 and Beaty & Johnson, 2021).
Author: Dawei Wang
License: MIT
Project-URL: Homepage, https://github.com/dtzx00/GloVe-Word-Embeddings
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: numpy
Requires-Dist: nltk
Requires-Dist: transformers
Requires-Dist: torch
Dynamic: license-file

# GloVe-Word-Embeddings

This is a library to quickly clean, validate, categorise and embed words for academic research.  
Design is based on Wang et al., 2026 (*Nature Human Behaviour*) and Olson et al., 2021 (*PNAS*).

This package has five parts:

1. **pre** — clean, normalise or lemmatize words  
2. **val** — validate against Olson’s list or WordNet nouns  
3. **cat** — classify to proper-noun buckets or WordNet categories
4. **mod** — load embeddings and turn words or phrases into vectors  
5. **rat** — look up concreteness, complexity, and age-of-acquisition ratings

Embedding models and rating norms are hosted on an AWS S3 bucket and downloaded automatically on first use.  
NLTK WordNet and names data are also downloaded automatically on first use of any category helper.

#### Installation

```bash
pip install glove-word-embeddings
```

### Preprocessing (`pre`)

Clean, normalize or lemmatize raw text before validation, classfication or embedding.

```python
from glove_word_embeddings import pre

pre.return_lowercase("  Cat! ")          # "  cat! "
pre.strip_marks("  Cat! ")               # "  Cat  "
pre.strip_stopword("The Cat")            # "Cat"
pre.strip_space("jar of jam")            # "jarofjam"
pre.remove_stopwords("jar of jam")       # ['jar', 'jam']
pre.remove_stopwords("not just a box", how="heavy") # ['box']
pre.space_check("cat")                   # True   (single token)
pre.space_check("jar of jam")            # False

# Full clean (defaults: lower + drop stopwords + strip marks + strip space)
pre.clean_word("  The Cat! ")            # "cat"
pre.clean_word("jar of jam")             # "jarjam"
pre.clean_word("jar of jam", strip_space_bool=False)  # "jar jam"
pre.clean_word("the", strip_stopword_bool=False)      # "the"

# WordNet lemma (cached)
pre.lemmatize("dogs")                    # "dog"
pre.lemmatize("running", pos="v")        # "run"
```

### Validation (`val`)

Check words against Olson’s validated list and WordNet.

```python
from glove_word_embeddings import mod, val

val.word("telescope")    # True  (in Olson list, single token)
val.word("jar of jam")   # False (fails space check after cleaning)
val.noun("telescope")    # True  (has a WordNet noun synset)
val.noun("quickly")      # False

m = mod.load("glove-olson-validated")
v = m.vocab_set()
val.vocab("telescope", v)   # True
val.vocab("ice cream", v)   # True if any of "ice cream" / "ice_cream" / "ice-cream" is in v
```


### Categorization (`cat`)

##### Categorize against a bucket of proper nouns

Check if words are in certain semantic group, room objects, or pure proper nouns.

```python
from glove_word_embeddings import cat

# Ordinary common word? (ignores WordNet proper-noun instances)
cat.check_common("apple")   # True
cat.check_common("nike")    # False
cat.check_common("aaron")   # False

# Which curated buckets does a word belong to?
cat.check_bucket("dog")                              # {"animals"}
cat.check_bucket("paris", check_common=False)        # {"places"}
cat.check_bucket("apple", check_common=True)         # {"fruits"}  (not brands)

# Count across a list of words
cat.count_bucket(words)                              # any bucket ≥ 5
cat.count_bucket(words, name="environment")          # environment objects
cat.count_bucket(words, name="places", number_of_words=1, check_common=False)
cat.count_bucket(words, name="names",  number_of_words=1, check_common=True)
cat.count_bucket(words, name="brands", number_of_words=1, check_common=True)
```

`check_common=False` trusts the seed list (places, environment, semantic groups).  
`check_common=True` drops a seed hit when the word is an ordinary common word (names / brands).

##### Categorize against WordNet category chains

Return the hypernym ladder of a word (general → specific). The word itself is **never** included in the list.

```python
from glove_word_embeddings import cat

# Default: longest path of the primary (most frequent) sense
cat.category_chain("dog")
# ['entity', 'physical_entity', 'object', 'whole', 'living_thing',
#  'organism', 'animal', 'chordate', 'vertebrate', 'mammal',
#  'placental', 'carnivore', 'canine']

# Shortest path
cat.category_chain("dog", path="short")
# ['entity', 'physical_entity', 'object', 'whole', 'living_thing',
#  'organism', 'animal', 'domestic_animal']

# Full set of ancestors (union of every path of the primary sense)
cat.category_chain("dog", path="full")
# ['entity', 'physical_entity', 'object', 'whole', 'living_thing',
#  'organism', 'animal', 'chordate', 'domestic_animal', 'vertebrate',
#  'mammal', 'placental', 'carnivore', 'canine']

# Ambiguous words – primary sense only (default)
cat.category_chain("badger", path="full")
# → hierarchy of the “Wisconsin inhabitant” sense

# Ambiguous words – union of *all* senses
cat.category_chain("badger", path="full", sense="union")
# → merges the person hierarchy and the animal hierarchy
```

### Embedding (`mod`)

Load a vector model and embed single words or short phrases. Embedding models available:

| Key | Dims | Descriptions |
| --- | --- | --- |
| `glove-6b-300d` | 300 | Stanford GloVe trained on Wikipedia + Gigaword (6B tokens). |
| `glove-840b-300d` | 300 | Stanford GloVe trained on Common Crawl (840B tokens). |
| `tasa-lsa-300d` | 300 | TASA latent semantic analysis space (Günther), used in SemDis. |
| `cbow-baroni-400d` | 400 | Baroni et al. (2014) best-predict CBOW, used in SemDis. |
| `cbow-ukwac-subs-300d` | 300 | Mandera et al. CBOW trained on ukWaC + subtitles, used in SemDis. |
| `cbow-subs-300d` | 300 | Mandera et al. CBOW trained on subtitles only, used in SemDis. |
| `wiki-news-300d-1m` | 300 | fastText English vectors trained on Wikipedia + news (1M words). |
| `dolma-300-2024-1m` | 300 | GloVe vectors trained on the 2024 Dolma corpus (1M words). |
| `glove-olson-validated` | 300 | GloVe vectors kept only for words on Olson et al. (2021)'s list. |
| `flair-olson-glove` | — | Flair GloVe embeddings kept only for Olson common words. |
| `flair-olson-extvec` | — | Flair Komninos (extvec) embeddings kept only for Olson common words. |
| `flair-olson-crawl` | — | Flair FastText crawl embeddings kept only for Olson common words. |
| `flair-olson-news` | — | Flair FastText news embeddings kept only for Olson common words. |
| `flair-olson-twitter` | — | Flair Twitter embeddings kept only for Olson common words. |
| `flair-olson-turian` | — | Flair Turian embeddings kept only for Olson common words. |
| `flair-olson-random` | — | Flair random embeddings kept only for Olson common words. |
| `bert-large` | 1024 | BERT-large uncased (Johnson et al., 2023). Context-dependent. |
| `bert-base` | 768 | BERT-base uncased. Context-dependent. Smaller and faster. |

```python
from glove_word_embeddings import mod

m = mod.load("glove-6b-300d")          # downloads on first use, caches locally
m = mod.load("tasa-lsa-300d")
m = mod.load("cbow-baroni-400d")
m = mod.load("cbow-ukwac-subs-300d")
m = mod.load("cbow-subs-300d")
m = mod.load("glove-olson-validated")
m = mod.load("bert-large")

m.embed_exact("cat")                   # exact match only → np.ndarray or None
m.vocab_set()                          # → set of all words in the model

# Multi-word phrases
m.embed_phrase("jar of jam")               # tries exact / underscore / hyphen variants,
                                           # otherwise averages the non-stopword parts
m.embed_phrase("jar of jam", how="heavy")  # remove stopwords as in Beaty & Johnson (2021)
```

Files are cached in `~/.cache/glove-word-embeddings`.

### Ratings (`rat`)

Scalar norms, not embeddings. Load with `rat.load`, not `mod.load`. Missing words return `None`. Stored scales: concreteness is Brysbaert’s 1–5 mean rescaled as `(Conc.M − 1) / 4 × 100`, complexity is 0–100, age of acquisition is years.

`score` and `score_phrase` clean by default (`clean=True`): lowercase and strip punctuation, using `pre.clean_word` without dropping stopwords or spaces. `Arsenal.` hits. `ice cream` stays a bigram. `clean=False` is the exact key.

| Key | What |
| --- | --- |
| `brysbaert-concreteness` | Brysbaert, Warriner & Kuperman (2014). Includes multiword items such as `ice cream`. |
| `maddela-complexity` | Maddela & Xu (2018) word-complexity lexicon. |
| `kuperman-aoa` | Kuperman, Stadthagen-Gonzalez & Brysbaert (2012). |

```python
from glove_word_embeddings import rat

r = rat.load("brysbaert-concreteness")   # downloads on first use, caches locally
r.score("dog")                           # 96.25
r.score("ice cream")                     # bigram hit, not an average
r.score("Arsenal.")                      # punctuation stripped, then case-folded
r.score("Arsenal.", clean=False)         # None
r.score_phrase("jar of jam")             # whole phrase, else mean of known tokens
r.score_phrase("jar of jam", how="heavy")
r.vocab_set()
```

### Citations

If you like this package, please cite the relevant works.

Wang et al., (2026) -- Package owner [CITE THIS!]
```markdown
Wang, D., Huang, D., Shen, H., & Uzzi, B. (2026). A large-scale comparison of divergent creativity in humans and large language models. Nature Human Behaviour, 10(3), 531–540. https://doi.org/10.1038/s41562-025-02331-1
```

Olson et al., (2021)
```markdown
Olson, J. A., Nahas, J., Chmoulevitch, D., Cropper, S. J., & Webb, M. E. (2021). Naming unrelated words predicts creativity. Proceedings of the National Academy of Sciences, 118(25), e2022340118. https://doi.org/10.1073/pnas.2022340118
```

Beaty & Johnson, (2021)
```markdown
Beaty, R. E., & Johnson, D. R. (2021). Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior Research Methods, 53(2), 757–780. https://doi.org/10.3758/s13428-020-01453-w
```

Johnson et al., (2023)
```markdown
Johnson, D. R., Kaufman, J. C., Baker, B. S., Patterson, J. D., Barbot, B., Green, A. E., van Hell, J., Kennedy, E., Sullivan, G. F., Taylor, C. L., Ward, T., & Beaty, R. E. (2023). Divergent semantic integration (DSI): Extracting creativity from narratives with distributional semantic modeling. Behavior Research Methods, 55(7), 3726–3759. https://doi.org/10.3758/s13428-022-01986-2
```

Brysbaert et al., (2014) - *Concreteness ratings*
```markdown
Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014). Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46(3), 904–911. https://doi.org/10.3758/s13428-013-0403-5
```

Kuperman et al., (2012) - *Age of acquisition ratings*
```markdown
Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M. (2012). Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 44(4), 978–990. https://doi.org/10.3758/s13428-012-0210-4
```

Maddela et al., (2018) - *Word complexity*
```markdown
Maddela, M., & Xu, W. (2018). A word-complexity lexicon and a neural readability ranking model for lexical simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 3749–3760). Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1410
```
