Metadata-Version: 2.4
Name: lmsy_w2v_rfs
Version: 0.1.5
Summary: Word-embedding seed expansion and document scoring. Bring your own seeds. Originally Li, Mai, Shen, Yan (2021, RFS).
Author-email: Feng Mai <feng-mai@uiowa.edu>
License: MIT License
        
        Copyright (c) 2026 Kai Li, Feng Mai, Rui Shen, Xinyan Yan
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Paper, https://doi.org/10.1093/rfs/hhaa079
Project-URL: Original implementation, https://github.com/MS20190155/Measuring-Corporate-Culture-Using-Machine-Learning
Project-URL: Repository, https://github.com/maifeng/lmsy_w2v_rfs
Project-URL: Documentation, https://maifeng.github.io/lmsy_w2v_rfs/
Project-URL: Changelog, https://github.com/maifeng/lmsy_w2v_rfs/blob/main/CHANGELOG.md
Keywords: word2vec,gensim,culture,nlp,corenlp,finance,text-as-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: gensim<5,>=4.3.3
Requires-Dist: numpy<3,>=1.24
Requires-Dist: pandas<3,>=2
Requires-Dist: scikit-learn<2,>=1.3
Requires-Dist: tqdm<5,>=4.66
Requires-Dist: smart_open<8,>=6
Requires-Dist: nltk<4,>=3.8
Provides-Extra: spacy
Requires-Dist: spacy<4,>=3.7; extra == "spacy"
Provides-Extra: stanza
Requires-Dist: stanza<2,>=1.8; extra == "stanza"
Requires-Dist: protobuf<6,>=4; extra == "stanza"
Provides-Extra: corenlp
Requires-Dist: stanza<2,>=1.8; extra == "corenlp"
Requires-Dist: protobuf<6,>=4; extra == "corenlp"
Provides-Extra: all
Requires-Dist: spacy<4,>=3.7; extra == "all"
Requires-Dist: stanza<2,>=1.8; extra == "all"
Requires-Dist: protobuf<6,>=4; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: jupytext>=1.16; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
Requires-Dist: mkdocs-material>=9.5; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.25; extra == "docs"
Requires-Dist: mkdocs-include-markdown-plugin>=6; extra == "docs"
Dynamic: license-file

# `lmsy_w2v_rfs`: Word2Vec dictionary expansion and document scoring

[![Open in Colab](https://img.shields.io/badge/Colab-quickstart-orange?logo=googlecolab&logoColor=white)](https://colab.research.google.com/github/maifeng/lmsy_w2v_rfs/blob/main/notebooks/01_quickstart_colab.ipynb)
[![PyPI version](https://img.shields.io/pypi/v/lmsy_w2v_rfs.svg)](https://pypi.org/project/lmsy_w2v_rfs/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/maifeng/lmsy_w2v_rfs/blob/main/LICENSE)

`lmsy_w2v_rfs` implements the Word2Vec seed-expansion method for document scoring
introduced in Li, Mai, Shen, and Yan (2021). A researcher specifies a small set of
seed words for each concept to be measured; the package trains Word2Vec on the
target corpus, expands each concept's seeds into a corpus-specific dictionary of
related words and multi-word phrases, and produces document-level scores by
TF-IDF–weighted dictionary matching. 

## Citation

If you find the package useful, please cite the paper the method is based on:

> Li, Kai, Feng Mai, Rui Shen, and Xinyan Yan (2021), "Measuring Corporate Culture
> Using Machine Learning," *Review of Financial Studies* 34(7):3265–3315,
> [doi.org/10.1093/rfs/hhaa079](https://doi.org/10.1093/rfs/hhaa079).

<details>
<summary>BibTeX</summary>

```bibtex
@article{li2021measuring,
  title={Measuring Corporate Culture Using Machine Learning},
  author={Li, Kai and Mai, Feng and Shen, Rui and Yan, Xinyan},
  journal={The Review of Financial Studies},
  volume={34}, number={7}, pages={3265--3315}, year={2021},
  doi={10.1093/rfs/hhaa079}
}
```
</details>

This package is a general update of the paper's method. The original
code for the paper is at
[MS20190155/Measuring-Corporate-Culture-Using-Machine-Learning](https://github.com/MS20190155/Measuring-Corporate-Culture-Using-Machine-Learning).

---

## Install

```bash
pip install -U lmsy_w2v_rfs
```

The base install runs out of the box with `preprocessor="none"` (whitespace
tokenization). For richer Phase 1 parsing (lemmatization, named-entity masking,
and dependency-based multi-word expressions) install an optional backend:

```bash
pip install -U "lmsy_w2v_rfs[spacy]" && python -m spacy download en_core_web_sm
```

For reproduction of the 2021 paper, use the CoreNLP backend (slower; needs Java and a
one-time ~1 GB download):

```bash
pip install -U "lmsy_w2v_rfs[corenlp]"
lmsy-w2v-rfs download-corenlp
```

---

## Quickstart

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/maifeng/lmsy_w2v_rfs/blob/main/notebooks/01_quickstart_colab.ipynb) &nbsp; **Try it now**, no install, runs on a bundled 2,000-review demo corpus.

Researchers usually start from a table of documents. Point the pipeline at a CSV,
declare a few seed words per concept, and run:

```python
from lmsy_w2v_rfs import Pipeline, Config

seeds = {
    "innovation":   ["innovation", "innovative", "creativity", "creative"],
    "teamwork":     ["teamwork", "collaboration", "collaborate", "supportive"],
    "compensation": ["pay", "salary", "compensation", "benefits", "bonus"],
}

p = Pipeline.from_csv(
    "reviews.csv", text_col="text", id_col="review_id",
    work_dir="runs/quickstart",
    config=Config(seeds=seeds),     # preprocessor="none" by default
)
p.run()                     # phrase + train + expand + score
p.show_dictionary(top_k=10) # inspect the expanded dictionary
print(p.score_df("TFIDF"))  # per-document scores
```

The first two concepts come from the paper's culture construct; `compensation` is a
concept *outside* the five culture dimensions, included to show the method
generalizes. On a corpus of employee reviews, the expansion fills each concept with
the corpus's own vocabulary — note that the seeds never mentioned `401k_match`,
`dental`, or `mentorship`:

```text
=== innovation ===
  seeds:    innovation, innovative, creativity, creative
  expanded: creativity, entrepreneurial, passion, open_communication, fostering
=== teamwork ===
  seeds:    teamwork, collaboration, collaborate, supportive
  expanded: collaboration, inclusion, mutual_respect, caring, fosters
=== compensation ===
  seeds:    pay, salary, compensation, benefits, bonus
  expanded: salary, competitive, 401k_match, dental, medical, bonuses
```

### Other ways to load documents and seeds

The same `Pipeline` also accepts in-memory lists, DataFrames, JSONL, and directories:

```python
Pipeline(texts=[...], doc_ids=[...], work_dir=..., config=cfg)              # in-memory list
Pipeline.from_csv("docs.csv", text_col="text", id_col="id", ...)            # CSV
Pipeline.from_dataframe(df, text_col="text", id_col="id", ...)              # DataFrame
Pipeline.from_directory("./docs/", pattern="*.txt", ...)                    # one file per doc
Pipeline.from_text_file("docs.txt", id_path="ids.txt", ...)                 # one doc per line
Pipeline.from_jsonl("docs.jsonl", text_key="text", id_key="id", ...)        # JSONL
```

Seeds accept a Python dict, a JSON file, or a plain text file:

```python
from lmsy_w2v_rfs import load_seeds
Config(seeds=load_seeds("my_seeds.json"))     # or .txt, or pass a dict directly
```

CLI: `lmsy-w2v-rfs run --seeds my_seeds.txt --input docs.csv --input-format csv --out runs/x`.

### Reproducing Li et al. (2021)

The package ships the paper's 47 seed words across five culture dimensions, and the
CoreNLP backend reproduces the paper's Phase 1 parsing:

```python
from lmsy_w2v_rfs import Pipeline, Config, load_example_seeds

seeds = load_example_seeds("culture_2021")    # 47 seeds, 5 dimensions
config = Config(seeds=seeds, preprocessor="corenlp")  # needs Java; see Install
```

---

## The construction procedure

The package implements the four-step construction procedure of Li et al. (2021). Each step is a method on `Pipeline`; calling `.run()` executes them in order and saves intermediate artifacts under `work_dir/` so any step can be redone without redoing the others.

### Step 1: Two-step phrase construction

Phrases carry meaning that single words cannot. The package extracts them in two complementary steps targeting different kinds of phrases.

**Step 1a, parser-based (general-English phrases).** A dependency parser identifies fixed multiword expressions (`with_respect_to`, `rather_than`) and compound words (`intellectual_property`, `healthcare_provider`). The parser also lemmatizes (`stocks` → `stock`) and masks named entities as `[NER:ORG]` placeholders so proper nouns do not bias the vector space. The 121-token SRAF generic stopword list is removed in the cleaning pass that follows.

| `Config(preprocessor=...)` | Backend | Needs |
|---|---|---|
| `"none"` *(default)* | whitespace tokenize, lowercase only | base install |
| `"static"` | NLTK `MWETokenizer` over a curated list | base install |
| `"spacy"` | spaCy (lemmas, NER, dependency MWEs) | `[spacy]` extra + a model |
| `"corenlp"` *(paper-faithful)* | Stanford CoreNLP via `stanza.server` | `[corenlp]` extra + Java |
| `"stanza"` | stanza `Pipeline` | `[stanza]` extra |

**Step 1b, statistical (corpus-specific phrases).** After Step 1a, gensim's `Phrases` scans the parsed corpus for statistically significant adjacent-token co-occurrences and joins them with `_`. A second pass over the bigram-joined corpus learns trigrams. This step identifies recurring collocations specific to the corpus: an earnings-call corpus surfaces `forward_looking_statement` and `cost_of_capital`; a product-review corpus surfaces `customer_service` and `delivery_time`; a Glassdoor corpus surfaces `work_life_balance` and `growth_opportunity`.

```python
from lmsy_w2v_rfs import Config, load_example_seeds

seeds = load_example_seeds("culture_2021")  # or any dict[str, list[str]]
Config(
    seeds=seeds,
    use_gensim_phrases=True,
    phrase_passes=2,            # 1 = bigrams; 2 = bigrams + trigrams
    phrase_min_count=10,        # works on a ~270k-doc corpus
    phrase_threshold=10.0,      # for smaller corpora try 3 / 5.0
)
```

The phrase-tagged corpus is written to `work_dir/corpora/pass2.txt` and can be opened directly to inspect the joined phrases.

### Step 2: Word2Vec

`Pipeline.train()` fits a `gensim.models.Word2Vec` on the phrase-tagged corpus. Every word and phrase receives a 300-dimensional vector. Defaults match the 2021 paper:

```python
from lmsy_w2v_rfs import Config, load_example_seeds

seeds = load_example_seeds("culture_2021")  # or any dict[str, list[str]]
Config(seeds=seeds, w2v_dim=300, w2v_window=5, w2v_min_count=5, w2v_epochs=20)
```

The model is saved at `work_dir/models/w2v.mod` and is available as `p.w2v` for ad-hoc queries.

### Step 3: Seed expansion

`Pipeline.expand_dictionary()` builds the per-concept dictionary by:

1. Averaging the in-vocabulary seed vectors for the concept.
2. Taking the top `n_words_dim` (default 500) tokens by cosine similarity to that mean.
3. Resolving cross-loadings: a token close to multiple concepts is assigned to the one whose seed mean it is closest to.
4. Dropping `[NER:*]` placeholders so named entities never enter the dictionary.

The result is written to `work_dir/outputs/expanded_dict.csv`, one column per concept, sorted by descending similarity to the seed mean.

```python
p.show_dictionary(top_k=10)         # prints per-concept seeds + top expansions
p.dictionary_preview(top_k=10)      # DataFrame for notebook display
```

### Step 4: Manual dictionary inspection

Nearest-neighbor expansion surfaces noise: off-topic terms, industry-specific outliers, words too general to be informative. Two ways to remove them, both atomic across the in-memory dictionary and the on-disk CSV:

```python
# Programmatic, replicable in a notebook:
p.edit_dictionary(
    remove={"innovation": ["fantastic", "incredible"]},
    add={"innovation": ["patent"]},
)

# Spreadsheet-driven, faster on a big dictionary:
#   1. open p.dict_path in Excel or any text editor
#   2. edit, save
#   3. p.reload_dictionary()
```

Cached scores are dropped after curation. Call `p.score()` to rescore against the curated dictionary.

---

## Scoring

A document's score on a concept is the sum of TF-IDF weights for every dictionary token present in the document, divided by total document length.

| Method | Weight per dictionary hit | Source |
|---|---|---|
| `TFIDF` | `tf · log(N/df)` | 2021 paper (the published measure) |
| `TF` | `tf` | alternative |
| `WFIDF` | `(1 + log tf) · log(N/df)` | alternative (sublinear `tf`) |
| `TFIDF+SIMWEIGHT`, `WFIDF+SIMWEIGHT` | × `1/ln(2 + rank)` | rank-weighted variant |

The `+SIMWEIGHT` variants additionally weight each word by its **rank** in the
similarity-ordered dictionary (`1/ln(2 + rank)`), so words nearer the seed
centroid count more and peripheral expansion words count less. This rank-based
similarity weighting is the scheme several studies building on the method have
adopted. It is rank-based, not a function of the raw cosine-similarity value.

```python
p.score(methods=("TFIDF",))
p.score_df("TFIDF")
```

Outputs land at `work_dir/outputs/scores_<METHOD>.csv`.

### Which words drive a dimension?

To validate dictionary quality, decompose each dimension's score into the
contribution of each dictionary word across the corpus:

```python
contrib = p.word_contributions("TFIDF")   # dimension, word, contribution, relative, cumulative
```

This writes `work_dir/outputs/word_contributions_<METHOD>.csv` and shows, per
dimension, each word's share and the running cumulative share — the standard
way to check that (say) `innovation` is driven by genuine innovation terms
rather than a few high-IDF artifacts.

---

## Large corpora

Once parsing finishes, downstream stages stream through disk: `clean` reads parsed sentences line by line; `phrase` and `train` use gensim's `PathLineSentences` so the training corpus is never fully materialized. The bottleneck is the **input stage**: the document loader holds the corpus in a Python list before parsing begins.

For corpora beyond a few hundred thousand documents, or when running on a cluster, see the [Run on HPC how-to](docs/how-to/run-on-hpc.md) for the multi-shard workflow, SLURM and SGE templates, and BLAS thread-cap instructions.

---

## Configuration parameters

```python
Config(
    seeds=...,                         # required: dict[str, list[str]]

    # Step 1a
    preprocessor="none",               # "none" | "static" | "spacy" | "corenlp" | "stanza"
    mwe_list=None,                     # None | "finance" | path to a curated list
    spacy_model="en_core_web_sm",
    parse_chunk_size=0,                # >0 processes docs in batches (caps memory on big corpora)
    n_cores=4,
    corenlp_memory="6G",
    corenlp_port=9002,
    corenlp_timeout_ms=120_000,        # per-request CoreNLP timeout (ms)
    corenlp_max_char_length=1_000_000, # raise for very long transcripts
    corenlp_properties={},             # extra CoreNLP server properties (override/add)

    # Step 1b
    use_gensim_phrases=True,
    phrase_passes=2,
    phrase_threshold=10.0,
    phrase_min_count=10,
    phrase_extra={},                   # extra kwargs -> gensim Phrases (e.g. {"scoring": "npmi"})

    # Step 2
    w2v_dim=300,
    w2v_window=5,
    w2v_min_count=5,
    w2v_epochs=20,
    w2v_sg=0,                          # 0 = CBOW (matches the original); 1 = skip-gram
    w2v_extra={},                      # extra kwargs -> gensim Word2Vec (e.g. {"negative": 10, "hs": 0})

    # Step 3
    n_words_dim=500,                   # paper's threshold for the dictionary cutoff
    dict_restrict_vocab=None,
    min_similarity=0.0,

    # Scoring (extensions beyond the 2021 paper)
    tfidf_normalize=False,
    zca_whiten=False,                  # ZCA-decorrelate the concept columns; see docs/how-to/whiten-scores.md
    zca_epsilon=1e-6,

    random_state=42,
)
```

---

## Resources

- **Documentation** (concepts, how-to guides, API reference): [maifeng.github.io/lmsy_w2v_rfs](https://maifeng.github.io/lmsy_w2v_rfs/)
- **Original implementation**: [MS20190155/Measuring-Corporate-Culture-Using-Machine-Learning](https://github.com/MS20190155/Measuring-Corporate-Culture-Using-Machine-Learning)
- **License**: MIT. The citation and BibTeX are [at the top](#citation).
