Metadata-Version: 2.4
Name: lemo-vocabulate
Version: 1.0.0
Summary: Dictionary-based text analysis tool for emotion and sentiment analysis. Python port of Vocabulate (Vine et al., 2020).
Author-email: Steven Mesquiti <sm9518@princeton.edu>
License: MIT License
        
        Copyright (c) 2025 Steven Mesquiti
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
        ---
        
        This software is a Python port of Vocabulate, originally developed by:
        Vine, V., Boyd, R. L., & Pennebaker, J. W. (2020).
        Natural emotion vocabularies as windows on distress and well-being.
        Nature Communications, 11(1), 4525.
        https://doi.org/10.1038/s41467-020-18349-0
Project-URL: Homepage, https://github.com/Bushel-of-Lemons/LEMO_Vocabulate
Project-URL: Issues, https://github.com/Bushel-of-Lemons/LEMO_Vocabulate/issues
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=1.3.0
Requires-Dist: tqdm>=4.62.0
Dynamic: license-file

# Vocabulate Python Edition

**Vocabulate** is a dictionary-based text analysis tool originally developed in C# for Windows.  
This Python package allows you to tokenize, clean, and analyze texts based on custom dictionaries across **any operating system** (Windows, macOS, Linux).

**DISCLAIMER:** I do not take credit for this software. I simply reconfigured it to run using a higher-level programming language (i.e., python instead of C#). All credit goes to the original authors: [Vine et al. (2020)](https://www.nature.com/articles/s41467-020-18349-0)

```bib
@article{vine2020natural,
  title={Natural emotion vocabularies as windows on distress and well-being},
  author={Vine, Vera, Boyd, Ryan L. and Pennebaker, James W.},
  journal={Nature Communications},
  volume={11},
  number={1},
  pages={4525},
  year={2020},
  doi={10.1038/s41467-020-18349-0}
}
```

## Why This Package?

While the original Vocabulate software is powerful, this Python implementation offers several advantages:

- **Cross-platform compatibility**: Works on Windows, macOS, and Linux (original is Windows-only)
- **Flexible input formats**: Analyze text from pandas DataFrames, CSV files, single text files, or folders of text files
- **Modern Python ecosystem**: Integrates seamlessly with pandas, Jupyter notebooks, and other data science tools

---

## Installation

### Option 1: Install from PyPI (Recommended)

The simplest way to install LEMO Vocabulate is via pip:

```bash
pip install lemo-vocabulate
```
That's it! The package includes the AEV dictionary and stopwords file, so you can start analyzing text immediately.

### Option 2: Install in a Conda Environment

For better dependency management, we'd recommend using a conda environment:

```bash
# Create and activate a new environment
conda create -n lemo python=3.8 -y # 

# Install the package
pip install lemo-vocabulate
```

### Option 3: Install from Source

If you want to modify the code or contribute to development:

```bash
# Clone the repository
git clone https://github.com/Bushel-of-Lemons/LEMO_Vocabulate.git
cd LEMO_Vocabulate

# Install in editable mode
pip install -e .
```

**Note:** This package requires Python >= 3.8 and assumes you have python and pip already installed. Please refer to the [official Python installation guide](https://www.python.org/downloads/) if needed.

---

## Quick Start

```python
import pandas as pd
from lemo_vocabulate import run_vocabulate_analysis, get_data_path

# Example using a DataFrame with included data files
df = pd.DataFrame({
    "user_id": ["user_1", "user_2"],
    "text": ["This is a sample text.", "Another example text."]
})

# Use the included dictionary and stopwords
results = run_vocabulate_analysis(
    dict_file=get_data_path("AEV_Dict.csv"),
    input_data=df,
    text_column="text",
    stopwords_file=get_data_path("stopwords.txt"),
    raw_counts=True
)

print(results.head())
```

### Using Custom Files

You can still use your own dictionary and stopwords files, just provide the file paths:

```python
# Use custom files
results = run_vocabulate_analysis(
    dict_file="path/to/your/custom_dict.csv",
    input_data=df,
    text_column="text",
    stopwords_file="path/to/your/custom_stopwords.txt",
    raw_counts=True
)
```
---

## Features

- **Tokenization designed for social media text**
    - Twitter-aware tokenizer that handles:
        - Usernames (@user)
        - Hashtags (#happy)
        - Emojis and emoticons
        - URLs
        - Repeated characters (soooo good)
        - Punctuation-heavy social media text

- **Stopword removal**
    - Flexible stopword handling via file or string input

- **Dictionary matching with multi-word wildcards**
    - Compatible with custom dictionaries in CSV format
    - Example dictionary provided: `Dictionary/AEV_Dict.csv`
    
    **Dictionary breakdown:**
    ```
    Neg          94
    Pos          53
    AnxFear      20
    Anger        16
    Sadness      36
    NegUndiff    15
    Total words in dictionary: 162
    ```

- **Comprehensive text metrics**
    - Word count, type-token ratio, dictionary coverage
    - Category-level statistics (CWR, CCR, counts, unique counts)

- **Flexible output**
    - Returns Pandas DataFrame
    - Optional CSV export

---

## Usage Examples

### Analyzing a Pandas DataFrame

```python
import pandas as pd
from lemo_vocabulate import run_vocabulate_analysis

# Create sample data
df = pd.DataFrame({
    "text_id": [1, 2, 3],
    "text": [
        "I am so agitated and aggravated!",
        "He was afraid of the dark.",
        "I am so happy happy happy! And sad."
    ]
})

# Run analysis
results = run_vocabulate_analysis(
    dict_file="Dictionary/AEV_Dict.csv",
    input_data=df,
    text_column="text",
    stopwords_file="stopwords.txt",
    raw_counts=True,
    output_csv="results.csv"
)

print(results.head())
```

### Analyzing Text Files in a Folder or Single File

```python
# Analyze a single text file
results = run_vocabulate_analysis(
    dict_file="Dictionary/AEV_Dict.csv",
    input_data="path/to/file.txt",
    stopwords_file="stopwords.txt",
    raw_counts=True
)

# Analyze all .txt files in a folder
results = run_vocabulate_analysis(
    dict_file="Dictionary/AEV_Dict.csv",
    input_data="path/to/folder",
    stopwords_file="stopwords.txt",
    raw_counts=False
)
```

### Merging Results with Original Data

```python
# Run analysis
df_results = run_vocabulate_analysis(
    dict_file="Dictionary/AEV_Dict.csv",
    input_data=df,
    text_column="text",
    stopwords_file="stopwords.txt",
    raw_counts=True
)

# Create text_id for merging
df_results['text_id'] = df_results.index

# Merge with original data
df_complete = df_results.drop(['text', 'Filename'], axis=1).merge(
    df,
    on='text_id',
    how='left'
)

# Reorder columns
cols = ['text_id', 'text'] + [col for col in df_complete.columns if col not in ['text_id', 'text']]
df_complete = df_complete[cols]
```

---

## Stopwords

Stopword removal allows you to exclude very common function words (e.g., `the`, `and`, `is`, `I`, `you`) that typically do not carry psychological or semantic content. In Vocabulate, stopwords are removed **after tokenization** and **before dictionary matching**, which improves the interpretability of dictionary categories.

**This package includes a pre-configured stopwords file** that you can use immediately, or you can create your own custom stopwords file

### Using a Stopwords File (Recommended)

Create a `.txt` file with one word per line:

```txt
the
and
is
i
you
to
```

Use it in your analysis:

```python
from lemo_vocabulate import run_vocabulate_analysis, get_data_path
results = run_vocabulate_analysis(
    dict_file=get_data_path("AEV_Dict.csv"),
    input_data=df,
    text_column="text",
    stopwords_file=get_data_path("stopwords.txt")  # Use bundled stopwords
)
```

### Using a Stopwords String

```python
stopwords = "the\nand\nis\nbe\nnot\n"
results = run_vocabulate_analysis(
    dict_file="Dictionary/AEV_Dict.csv",
    input_data=df,
    text_column="text",
    stopwords_text=stopwords
)
```

### How Stopwords Affect Output Metrics

**Stopword removal does NOT affect:**
- `WC` (whitespace word count)
- `TC_Raw` (raw token count)
- `TTR_Raw` (raw type-token ratio)

**Stopword removal DOES affect:**

| Column | Effect |
|--------|--------|
| `TC_Clean` | Tokens after stopword removal |
| `TTR_Clean` | Based on clean tokens |
| `TC_NonDict` | Non-dictionary tokens after cleaning |
| `DictPercent` | Higher if stopwords filtered out |
| Category metrics | Only meaningful content words remain |

---

## Understanding the Output

The `run_vocabulate_analysis` function returns a Pandas DataFrame where each row corresponds to a single input text. Below is a detailed explanation of all output columns.

### General Text Metrics

| Column Name    | Description                                                                            |
| -------------- | -------------------------------------------------------------------------------------- |
| `Filename`     | Name of the file or index of the row from the input DataFrame                          |
| `text`         | The full original text that was analyzed                                               |
| `WC`           | Word count: total number of whitespace-separated tokens in the original text           |
| `TC_Raw`       | Token count after tokenizer processing (including punctuation, emoticons, etc.)        |
| `TTR_Raw`      | Type-Token Ratio for raw tokens: `#unique tokens / TC_Raw * 100`                       |
| `TC_Clean`     | Token count after removing stopwords                                                   |
| `TTR_Clean`    | Type-Token Ratio for cleaned tokens: `#unique tokens / TC_Clean * 100`                 |
| `TC_NonDict`   | Number of tokens not matched to any dictionary concept                                 |
| `TTR_NonDict`  | Type-Token Ratio of non-dictionary tokens                                              |
| `DictPercent`  | Percent of tokens matched to dictionary concepts: `num_matched_tokens / TC_Raw * 100`  |
| `CapturedText` | Concatenated string of all dictionary-matched words from the text                      |

### Category-Specific Metrics

For each category in the loaded dictionary (e.g., `Neg`, `Pos`, `AnxFear`, `Anger`, `Sadness`, `NegUndiff`), four metrics are provided:

| Column Suffix | Description                                                                                                                                                  |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `_CWR`        | **Category Word Ratio**: percentage of unique words in the category relative to total words in text: `unique_count / WC * 100`                               |
| `_CCR`        | **Category Concept Ratio**: percentage of unique words in the category relative to all matched tokens in that category: `unique_count / total_count * 100`   |
| `_Count`      | **Raw Count**: total number of occurrences of words from this category in the text (only if `raw_counts=True`)                                              |
| `_Unique`     | **Unique Count**: number of unique words in the text that matched this category (only if `raw_counts=True`)                                                 |

#### Example: Category "Neg"

- `Neg_CWR` → % of total words in the text that were unique Neg words
- `Neg_CCR` → % of category words that were unique
- `Neg_Count` → Total Neg words matched
- `Neg_Unique` → Number of unique Neg words matched

---

## Example Output

| Filename | text | WC | TC_Raw | TTR_Raw | TC_Clean | TTR_Clean | TC_NonDict | TTR_NonDict | DictPercent | CapturedText | Neg_CWR | Neg_CCR | Neg_Count | Neg_Unique | Pos_CWR | Pos_CCR | Pos_Count | Pos_Unique | AnxFear_CWR | AnxFear_CCR | AnxFear_Count | AnxFear_Unique |
|----------|------|-----|--------|---------|----------|-----------|------------|-------------|-------------|--------------|---------|---------|-----------|------------|---------|---------|-----------|------------|-------------|-------------|---------------|----------------|
| 0 | I am so agitated and aggravated! | 6 | 7 | 100.0 | 2 | 100.0 | 0 | 0.0 | 28.57 | agitated aggravated | 33.33 | 100.0 | 2 | 2 | 0.0 | 0.0 | 0 | 0 | 0.0 | 0.0 | 0 | 0 |
| 1 | He was afraid of the dark.... | 6 | 8 | 100.0 | 3 | 100.0 | 2 | 100.0 | 12.5 | afraid | 16.67 | 100.0 | 1 | 1 | 0.0 | 0.0 | 0 | 0 | 16.67 | 100.0 | 1 | 1 |
| 2 | Nothing to be afraid or agitated about. Yet I'm afraid, and it makes me want to agitate!! | 17 | 21 | 85.71 | 6 | 83.33 | 2 | 100.0 | 19.05 | afraid agitated afraid agitate | 11.76 | 50.0 | 4 | 2 | 0.0 | 0.0 | 0 | 0 | 5.88 | 50.0 | 2 | 1 |
| 3 | I am so happy happy happy! And sad. | 8 | 10 | 80.0 | 4 | 50.0 | 0 | 0.0 | 40.0 | happy happy happy sad | 12.5 | 100.0 | 1 | 1 | 12.5 | 33.33 | 3 | 1 | 0.0 | 0.0 | 0 | 0 |
| 4 | dislike disliked dislikes disliking/doo | 5 | 6 | 100.0 | 5 | 100.0 | 1 | 100.0 | 66.67 | dislike disliked dislikes disliking | 20.0 | 25.0 | 4 | 1 | 0.0 | 0.0 | 0 | 0 | 0.0 | 0.0 | 0 | 0 |

---

## Function Parameters

```python
run_vocabulate_analysis(
    dict_file: str = None,           # Path to dictionary CSV file (required)
    input_data = None,               # DataFrame, file path, or folder path (required)
    text_column: str = None,         # Column name for text (required for DataFrame)
    stopwords_text: str = None,      # Stopwords as newline-separated string
    stopwords_file: str = None,      # Path to stopwords file
    raw_counts: bool = True,         # Include raw counts in output
    encoding: str = "utf-8",         # File encoding
    csv_delimiter: str = ",",        # CSV delimiter
    csv_quote: str = '"',            # CSV quote character
    output_csv: str = None,          # Optional output CSV path
    whitespace_method: str = 'new'   # 'new' (default, recommended 'legacy' (exact C# match)
)
```
**Note about `whitespace_method`**

This parameter controls how the `WC` (word count) metric is calculated and **only affects this one column**.

- **`'new'` (default, recommended)**: Uses standard Python whitespace tokenization that handles edge cases (multiple spaces, leading/trailing whitespace) consistently. Best for new analyses.

- **`'legacy'`**: Replicates the exact word counting behavior of the original C# Vocabulate software. Use this only if you need to exactly match results from the Windows version.

All other metrics (tokenization, dictionary matching, category ratios) are identical between both methods.

---

## Citation

If you use this software in your research, please cite the original paper:

```bib
@article{vine2020natural,
  title={Natural emotion vocabularies as windows on distress and well-being},
  author={Vine, Vera, Boyd, Ryan L. and Pennebaker, James W.},
  journal={Nature Communications},
  volume={11},
  number={1},
  pages={4525},
  year={2020},
  doi={10.1038/s41467-020-18349-0}
}
```
