Metadata-Version: 2.4
Name: hct_bootstrap
Version: 0.2.2
Summary: A package using the Higher Criticism statistic to find discriminating words in two text corpora.
Author-email: Ophira Blumner <ophirablumner@gmail.com>, Alon Kipnis <alon.kipnis@runi.ac.il>
Maintainer-email: Ophira Blumner <ophirablumner@gmail.com>
License-Expression: MIT
Project-URL: Repository, https://github.com/oblumner/hct_bootstrap
Keywords: higher criticism,stylometry,text analysis,nlp,llm,discriminating words
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Development Status :: 3 - Alpha
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas
Requires-Dist: numpy
Requires-Dist: spacy>=3.8
Requires-Dist: click
Requires-Dist: scikit-learn
Requires-Dist: scipy
Provides-Extra: display
Requires-Dist: ipython; extra == "display"
Requires-Dist: matplotlib; extra == "display"
Dynamic: license-file

# HCT-Bootstrap: Discriminating Words in Text Corpora

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A Python package for identifying stable, statistically discriminating words between two text corpora, using Higher Criticism thresholding with bootstrap stabilization and an empirical within-corpus null.

## Overview

This package provides tools to perform a robust comparison of two texts or two collections of texts. The goal is to move beyond simple word frequency counts and identify words that are "surprisingly" frequent in one corpus relative to the other — a hallmark of stylistic difference.

Two problems make this harder than it looks. First, a vocabulary contains thousands of candidate words, so some will appear discriminating by chance alone. Second, a single statistical run is unstable: small perturbations of the data can change which words are selected. This package addresses both. It repeats the selection across bootstrap resamples and keeps only the words that survive every one, and it calibrates how many resamples are needed by running the same procedure on a null comparison built from corpora that are known to share a generating mechanism.

The result is a word list produced without any analyst-chosen frequency cutoff or threshold, together with a false-discovery bound.

The package is particularly useful for analyzing and comparing text generated by Large Language Models, comparing authors, or tracking stylistic change in a corpus over time.

## Higher Criticism for Word-Frequency Tables

The core statistical engine is the **Higher Criticism (HC)** test, adapted for textual analysis as described in Kipnis (2022).

Comparing two corpora involves thousands of individual hypothesis tests — one for each unique word. For each word we can compute a p-value representing the probability of observing its frequency difference by chance. But with thousands of tests, many small p-values will occur randomly: the **multiple testing problem**.

Traditional corrections like Bonferroni are too conservative here, often missing subtle but collectively significant signals. The Higher Criticism statistic is instead designed to detect a **sparse, heterogeneous signal** — which matches the assumption that an author's or model's style is defined not by a shift in every word, but by a small, unknown subset of words used idiosyncratically.

HC examines the ordered sequence of all p-values and finds the threshold where the number of surprisingly small p-values is most significant relative to what the null hypothesis would produce. This makes it well suited to identifying a faint but consistent stylistic fingerprint. The core statistical code is adapted from Alon Kipnis's `TwoSampleHC` repository.

### References

* Kipnis, A. (2022). Higher Criticism for Discriminating Word-Frequency Tables and Authorship Attribution. *The Annals of Applied Statistics*, 16(2), 1236–1252. https://arxiv.org/pdf/1911.01208

* Donoho, D., & Jin, J. (2004). Higher Criticism for detecting sparse heterogeneous mixtures. *The Annals of Statistics*, 32(3), 962–994. https://arxiv.org/pdf/math/0410072

## Installation

1. **Activate your virtual environment.** Python 3.8 or later.

2. **Install from PyPI:**

    ```bash
    pip install hct-bootstrap
    ```

3. **Download the spaCy model.** The package uses spaCy for tokenization, and the English model is not installed automatically:

    ```bash
    python -m spacy download en_core_web_sm
    ```

4. **Optional display extras.** For the in-notebook results table and the diagnostic plot:

    ```bash
    pip install "hct-bootstrap[display]"
    ```

## How the Method Works

The procedure has three moving parts, and two of them are the arguments you will most often set.

**Bootstrap stabilization.** At step *n*, the package has drawn *n* bootstrap resamples of the two corpora and run HC thresholding on each. A word counts as surviving only if it was selected in **all** *n* runs. The surviving count N̂(n) therefore only shrinks as *n* grows.

**The null comparison.** In parallel, the same procedure runs on a comparison where the two sides are known to come from the same source, so every surviving word is by construction a false discovery. N̂₀(n) is that count, taken as the maximum across the available null comparisons (conservative). Set it with `null_method`:

| `null_method` | Null construction |
| --- | --- |
| `'split'` (default) | Each input corpus is shuffled and split into two non-overlapping halves, which are compared to each other. |
| `'resample'` | Two independent with-replacement resamples are drawn from each input corpus and compared. |
| `'user'` | You supply a same-source corpus for each input via `null_corpus1_docs` / `null_corpus2_docs`; each original corpus is compared against its counterpart. |

**The stopping rule.** How many bootstrap iterations to run is decided by the data, not by you. Set it with `q`:

| `q` | Rule |
| --- | --- |
| `None` (default) | **Strict**: stop at the first *n* where no null word survives (N̂₀(n) = 0). |
| a float in (0, 1] | **FDR**: stop at the first *n* where FDP⁺(n) = (1 + N̂₀(n)) / max(N̂(n), 1) falls to `q`. The selected set then has false-discovery rate at most `q`. |

The final selection is the set of words surviving all *n* iterations, intersected with the words selected by a single HC run on the full corpora.

## Quick Start

This example builds two synthetic corpora that differ only in a small set of injected "AI style" words, then recovers them.

```python
from hct_bootstrap import analyze_and_display
import random

# Set random seed for reproducibility
random.seed(42)

background_vocab = [
    "analysis", "system", "data", "model", "research", "method", "result", "time", "user", "network",
    "process", "study", "group", "problem", "level", "information", "effect", "change", "design", "feature",
    "performance", "approach", "case", "point", "development", "variable", "pattern", "value", "algorithm", "test",
    "environment", "structure", "condition", "behavior", "factor", "application", "control", "theory", "practice", "solution",
    "science", "technology", "paper", "review", "framework", "function", "state", "rule", "context", "form",
    "concept", "equation", "experiment", "measure", "source", "type", "area", "base", "number", "size",
    "quality", "property", "task", "technique", "principle", "resource", "material", "energy", "rate", "scale",
    "phase", "stage", "component", "class", "architecture", "mechanism", "relation", "role", "impact", "access"
]

ai_style_words = ["delve", "intricate", "underscore", "meticulous", "crucial",
                  "tapestry", "robust", "realm", "nuance", "leverage"]

human_corpus = []
ai_corpus = []

# 1. Create the 'Human' corpus: 1000 documents of 250 random background words
for _ in range(1000):
    doc = " ".join(random.choices(background_vocab, k=250))
    human_corpus.append(doc)

# 2. Create the 'AI' corpus: same length and vocabulary, plus the injected style words
for _ in range(1000):
    doc = " ".join(random.choices(background_vocab, k=250))
    doc += " " + " ".join(ai_style_words)
    ai_corpus.append(doc)

# 3. Run the analysis and display the report.
#    coupled=False because the two lists are not paired one-to-one. If you had
#    generated both corpora from a shared list of prompts, so that
#    corpus1_docs[i] and corpus2_docs[i] come from the same prompt, you would
#    set coupled=True (the lists must then be of equal length).
#    q=None uses the strict stopping rule; null_method='split' builds the null
#    by halving each input corpus.
results = analyze_and_display(
    corpus1_docs=human_corpus,
    corpus1_name="Human",
    corpus2_docs=ai_corpus,
    corpus2_name="AI",
    coupled=False,
    q=None,
    null_method="split",
)
```

### Example Report Output

```text
=== BS-HCT Analysis Report ===

Comparing: 'Human' vs. 'AI'
Coupled test comparison: False
Null construction: split (always uncoupled)
Stopping rule: strict (stop when no null word survives)
Max t: 1000
--------------------
  t=    1 | N_test=   17 | N0_max=  12 | FDP+=0.7647
  t=    2 | N_test=   12 | N0_max=   6 | FDP+=0.5833
  t=    3 | N_test=   11 | N0_max=   3 | FDP+=0.3636
  t=    4 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    5 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    6 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    7 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    8 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    9 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=   10 | N_test=   11 | N0_max=   1 | FDP+=0.1818
  t=   11 | N_test=   11 | N0_max=   1 | FDP+=0.1818
  t=   12 | N_test=   11 | N0_max=   1 | FDP+=0.1818
  t=   13 | N_test=   11 | N0_max=   0 | FDP+=0.0909

--- Corpus Information (Post-Cleaning) ---
# words in Human: 249849
# words in AI: 259881

--- Full-Corpus HCT ---
HCT candidate words: 17
Full-corpus HC score: 4.6882

--- Stopping ---
Rule fired at iteration n = 13 (ran 13 iterations).

--- Stable Discriminating Words (11) ---

More frequent in 'Human':
principle

More frequent in 'AI':
crucial, delve, intricate, leverage, meticulous, nuance, realm, robust, tapestry, underscore

Selected words with HCT p-values:
```

| word | p_value | more_frequent_in |
| --- | ---: | --- |
| nuance | 4.003404e-271 | AI |
| crucial | 4.003404e-271 | AI |
| delve | 4.003404e-271 | AI |
| intricate | 4.003404e-271 | AI |
| leverage | 4.003404e-271 | AI |
| tapestry | 4.003404e-271 | AI |
| robust | 4.003404e-271 | AI |
| meticulous | 4.003404e-271 | AI |
| underscore | 4.003404e-271 | AI |
| realm | 4.003404e-271 | AI |
| principle | 1.631736e-07 | Human |

```text
=== End of Report ===
```

A plot of N̂(n), N̂₀(n), and FDP⁺(n) against *n* is displayed after the report when matplotlib is installed.

### Reading the report

The per-iteration trace shows the mechanism directly. A single HC run on the full corpora flags 17 candidate words, but stability winnows that to 11 by the third resample. Meanwhile the null count falls from 12 to 0, and the run stops at *n* = 13, the first iteration where no null word survives.

All ten injected style words are recovered. The eleventh selection, `principle`, is a chance artifact: both corpora draw from the same background vocabulary, so any difference in a background word is noise. Notice that its p-value is 264 orders of magnitude larger than the others — it sits exactly at the boundary of what the stability requirement can defend. This is what the reported FDP⁺ of 0.0909 means in practice: one false discovery among eleven selections, matching the bound the procedure derived for itself before anyone looked at the words.

To silence the per-iteration trace, pass `verbose=False`.

## API Reference

The package exposes two functions. They take identical arguments; `analyze_and_display` prints the formatted report and plot shown above, while `discriminate` returns the results silently for programmatic use.

#### `analyze_and_display(...)` and `discriminate(...)`

**Required**

* **`corpus1_docs`**, **`corpus2_docs`** (`List[str]`): the documents of each corpus.
* **`corpus1_name`**, **`corpus2_name`** (`str`): labels used throughout the report and results.
* **`coupled`** (`bool`): `True` if `corpus1_docs[i]` is paired with `corpus2_docs[i]` — for example, both generated from the same prompt. Requires lists of equal length, and resampling then draws the same document indices for both corpora, preserving the pairing. `False` resamples each corpus independently.

**Selection behavior**

* **`q`** (`float`, optional, default `None`): the stopping rule. `None` for the strict rule, or a value in (0, 1] for the FDR rule. See *How the Method Works* above.
* **`null_method`** (`str`, default `'split'`): `'split'`, `'resample'`, or `'user'`.
* **`null_corpus1_docs`**, **`null_corpus2_docs`** (`List[str]`, optional): required when `null_method='user'`. One same-source null corpus per input corpus.
* **`null_coupled`** (`bool` or `(bool, bool)`, optional): whether each user-supplied null corpus is prompt-paired with its test corpus, in which case the pair is cleaned jointly to preserve alignment. Pass a single boolean to apply to both null comparisons, or a two-element tuple to set them separately. Only meaningful with `null_method='user'`; defaults to matching `coupled`.
* **`max_t`** (`int`, default `1000`): safety cap on bootstrap iterations if the stopping rule never fires.
* **`random_seed`** (`int`, default `42`): seed for reproducible resampling.

**Pre-processing**

* **`pos_tags`** (`List[str]`, optional, default `None`): restrict the analysis to these spaCy part-of-speech tags, for example `['NOUN', 'VERB']`. `None` keeps all parts of speech. Named entities, punctuation, and tokens mixing letters with digits are always removed, and hyphenated compounds such as *real-world* are kept intact.
* **`nlp`** (`spacy.Language`, optional): a pre-loaded spaCy pipeline. If `None`, `default_spacy_model` is loaded internally and cached.
* **`default_spacy_model`** (`str`, default `'en_core_web_sm'`): the model to load when `nlp` is not supplied.

**Output**

* **`verbose`** (`bool`, default `True`): print the per-iteration trace of N̂(n), N̂₀(n), and FDP⁺(n).

#### Return value

Both functions return a dictionary:

| Key | Contents |
| --- | --- |
| `selected_words` | Sorted list of the stable discriminating words. |
| `selected_words_df` | Those words with `p_value` and `more_frequent_in`, ascending by p-value. |
| `n_rule`, `t_stop` | The iteration at which the rule fired, or `None`. Both keys hold the same value. |
| `fdp_at_stop` | FDP⁺(n) at the stopping iteration. |
| `rule` | `'strict'` or `'fdr'`. |
| `q`, `null_method` | The settings used, echoed back. |
| `iterations_run` | Number of iterations actually run. |
| `empty_reason` | Explanation if the selection came back empty, otherwise `None`. |
| `fdp_curve` | List of `(n, N̂(n), N̂₀(n), FDP⁺(n))` tuples — the data behind the plot. |
| `full_data_hc_score` | HC score from the single run on the full corpora. |
| `full_data_results_df` | Words selected by that single full-corpus run. |
| `test_survivors_df` | Surviving test words at the stopping iteration. |
| `null_survivors_dfs` | Surviving null words, one DataFrame per null comparison. |
| `cleaned_corpus1`, `cleaned_corpus2` | The cleaned corpora as joined strings. |

## Data

The corpora generated for the accompanying study — LLM-written scientific abstracts across sampling temperatures and model families, human versus LLM legislative summaries, and base versus instruction-tuned model summaries — are available in the `corpora/` directory of this repository.
