Metadata-Version: 2.4
Name: automated-intelligence-tests
Version: 0.1.7
Summary: Automated CAT, DAT, AUT and CWT intelligence tests for humans and AI
Author: dtzx00
License: MIT
Project-URL: Homepage, https://github.com/dtzx00/Automated-Intelligence-Tests
Project-URL: Repository, https://github.com/dtzx00/Automated-Intelligence-Tests
Project-URL: Issues, https://github.com/dtzx00/Automated-Intelligence-Tests/issues
Keywords: psychometrics,creativity,convergent-thinking,divergent-thinking,word-embeddings,llm-evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: glove-word-embeddings>=0.1.10
Requires-Dist: numpy>=1.21
Dynamic: license-file

# Automated Intelligence Tests

Minimal Python package providing automated tests of associative and creative ability, for both
human and artificial intelligence. Each test exposes the same two calls: `instruct()` builds the
stimuli and prompt, `evaluate()` scores the responses.

| Test  | Name                        | `instruct` | `evaluate`                                   |
|-------|-----------------------------|------------|----------------------------------------------|
| `cat` | Convergent Association Task | yes        | Semantic proximity score (Wang et al., WIP)  |
| `dat` | Divergent Association Task  | yes        | Semantic distance score (Olson et al., 2021) |
| `aut` | Alternative Uses Task       | yes        | SemDis originality (Beaty & Johnson, 2021)   |
| `cwt` | Creative Writing Task       | yes        | BERT DSI (Johnson et al., 2023)              |

## Install

```bash
pip install git+https://github.com/dtzx00/Automated-Intelligence-Tests.git
```

Once published to PyPI: `pip install automated-intelligence-tests`.

Dependencies (`glove-word-embeddings`, `numpy`) are declared in `pyproject.toml` and installed
automatically — no separate requirements step. Python 3.9+.

For local development:

```bash
git clone https://github.com/dtzx00/Automated-Intelligence-Tests.git
cd Automated-Intelligence-Tests
pip install -e .
```

## Quickstart

```python
import automated_intelligence_tests as ait

ait.list_available_tests()        # {"CAT": "Convergent Association Task", ...}

# 1. Build the stimuli and prompt
task = ait.instruct("cat", n_words=10, seed=42)
print(task["instructions"])       # give this to a person or a model
print(task["response_format"])    # the shape the answer should come back in

# 2. Fill in "word_user" for each item, then score
responses = {
    "wordset_1": {"word_1": "television", "word_2": "lake", "word_user": "reflection"},
    # ...
}
result = ait.evaluate("cat", responses)
print(result["score"], result["n_valid"])
```

`instruct()` returns a dict with `test`, `instructions`, `response_format`, and the test's own
stimuli. `evaluate()` returns a dict with `score` (`None` if nothing scorable), `n_valid`, and
per-test detail.

## API

```python
ait.list_available_tests()               # -> {short_name: long_name} for every test
ait.instruct(test, **kwargs)             # test in {"cat", "dat", "aut", "cwt"}
ait.call_test_instruction("DAT", **kwargs)   # same as instruct(), case-insensitive name
ait.evaluate(test, responses, **kwargs)
```

Sub-packages can also be used directly:
`from automated_intelligence_tests.cat import instruct, evaluate`.

| Test  | `instruct` arguments                                 | `evaluate` arguments                     |
|-------|------------------------------------------------------|------------------------------------------|
| `cat` | `cue=None, single_item=False, n_words=10, seed=None` | `responses, model_key="glove-840b-300d"` |
| `dat` | `cue=None, n_words=10, seed=None`                    | `responses, model_key=..., minimum=7`    |
| `aut` | `cue=None, n_words=None, seed=None`                  | `responses, model_keys=None`             |
| `cwt` | `cue=None, n_words=3, seed=None`                     | `responses, model_key="bert-large"`      |

**CAT** samples word pairs from a fixed list of 8,069 English word pairs at cosine distance 0.85–0.95. `n_words` controls how many pairs are returned at once. Providing `cue=[(w1, w2), ...]` uses those explicit pairs (and a single pair automatically embeds it in the instruction text).

**DAT** asks for `n_words` words that are as different from each other as possible. Optional `cue` seeds the first word; the participant then supplies the remaining words.

**AUT** samples (or accepts) a common object via `cue`. `n_words` is accepted for API consistency but currently unused.

**CWT** samples (or accepts via `cue`) 1–3 cue words controlled by `n_words` and asks for a short creative story that incorporates every cue word.

## Scoring

- **CAT** — mean cosine *similarity* between the participant's word and each of the two cue words, averaged over items. Higher = stronger convergent association. Invalid or unembeddable words are marked `valid: False` and excluded rather than penalised.
- **DAT** — Olson's procedure exactly: mean pairwise cosine *distance* of the first 7 unique valid words, ×100. Returns `score: None` if fewer than 7 valid words are given.
- **AUT** — Beaty & Johnson SemDis: clean each use (`how="heavy"` stopwords, drop cue + plural), multiply leftover word vectors, take `1 − cosine` to the cue in five spaces (`cbow-subs-300d`, `cbow-ukwac-subs-300d`, `cbow-baroni-400d`, `tasa-lsa-300d`, `glove-6b-300d`). `score` is the mean of those spaces, then the mean across uses. Higher = more original. First call downloads the five spaces into `~/.cache/glove-word-embeddings`.
- **CWT** — Johnson et al. BERT DSI: split the story into sentences, keep BERT-large layers 6 and 7 for every real token, put those vectors in one list, take the mean pairwise `1 − cosine`. Higher = more divergent integration. First call downloads BERT into `~/.cache/glove-word-embeddings/huggingface`.

All will use [`glove-word-embeddings`](https://pypi.org/project/glove-word-embeddings/), default model `glove-840b-300d` for CAT and DAT. The first `evaluate()` call downloads the embedding file and caches it under `~/.cache/glove-word-embeddings`; later calls are offline. Other keys (`glove-6b-300d`, `wiki-news-300d-1m`, the `flair-olson-*` set, …) can be passed via `model_key`.

## Layout

```
automated_intelligence_tests/
    __init__.py          dispatcher: list_available_tests() / instruct() / evaluate()
    cat/  dat/           per test: instruct.py, evaluate.py, metadata.json,
    aut/  cwt/                     readme.md, example.ipynb
    cat/data/cat_word_pairs_en.txt CAT word-pair list
```

Each test folder carries a `metadata.json` holding its short name, long name and description.
That file is what `list_available_tests()` reads, so dropping in a new test folder with one makes
it discoverable automatically. Each also has its own `readme.md` and a runnable `example.ipynb`.

## License

MIT
