Metadata-Version: 2.4
Name: emotion-probes
Version: 0.1.0
Summary: Reproducing Emotion Concepts and their Function in a Large Language Model on Qwen 3.6 27b
Author-email: Donny <donny.hikari@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Donny
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch
Requires-Dist: numpy
Requires-Dist: transformers
Requires-Dist: accelerate
Requires-Dist: bitsandbytes
Requires-Dist: scikit-learn
Requires-Dist: tqdm
Requires-Dist: anthropic
Provides-Extra: ui
Requires-Dist: gradio; extra == "ui"
Requires-Dist: plotly; extra == "ui"
Dynamic: license-file

# Emotion Probes

Reproducing [Emotion Concepts and their Function in a Large Language Model](https://arxiv.org/pdf/2604.07729) on Qwen 3.6 27b.

![Example](docs/example.png)

<a id="figure-02"></a>

![Emotional Fluctuation](docs/emo_fluct.png)

Included in this repo:
1. Artifacts Generation (emotional stories and neutral dialogues)
2. Activations Extraction
3. Emotion Vectors Calculation
4. Web Interface to See and Manipulate the Model's Emotion on the Fly (Planned)

## Generate Artifacts

The first step is to generate the emotional stories and the neutral dialogues used to extract the vectors.

Following Anthropic's approach, the same prompts are used. See [prompts/emotional_stories](prompts/emotional_stories) and [prompts/neutral_dialogues](prompts/neutral_dialogues).

A small adjustment is made. Namely separator '---' is added so that it is easier to extract each story.

```
# Generate emotional stories
python scripts/generate_artifacts.py --n-stories 3 --max-parallel 12 --dry-run

# Generate neutral dialogues
python scripts/generate_artifacts.py --n-stories 6 --max-parallel 12 --neutral --dry-run

# Split emotional stories
python scripts/split_stories.py --force

# Split neutral dialogues
python scripts/split_stories.py --force --neutral
```

### About the coverage

The original paper generates 12 stories for 100 topics for 170 emotions each. As a GPU-poor, the number is largely reduced due to the constraint of GPU power. Only 3 stories for each 25 topics for each 58 emotions is generated. But as shown later, the emotion vectors extracted still successfully capture the characteristics of the emotions.

#### Emotions

The 58 emotions is selected by asking the model to group the emotions and pick representative ones for each group. The model split the emotions into 11 categories, and picked top 3 for each. In addition, 25 emotions are picked manually. The prompt:

```
group those emotions, and pick the top 3 most representative ones for each group:

<the list of emotions from the original paper>
```

The model's answer is in [dataset-ingredients/selected/emotions-groups](dataset-ingredients/selected/emotions-groups).

#### Topics

The 25 topics are selected randomly from the 100 topics listed in the original paper.

```
shuf -n 25 original/stories-topics >selected/stories-topics
```

## Extract Activations (Optional)

> *Note:*
> This step can now be skipped as the notebook in the next section is now capable of calculating the emotion vectors from the prompts on the fly.
> Use it to cache the activations if need to investigate those activations.
> Be careful as the size of those activations can get really large.

Next is extracting the activations from selected layers in a model. The extraction script is inspired by the [J-lens](https://github.com/anthropics/jacobian-lens) script.

```
# Extract the activations from the emotional stories generated in the previous step
python scripts/extract_vectors.py --model "Qwen/Qwen3.6-27B" --prompts outputs/extracted/emotional_stories --output-dir outputs/vectors/emotions --layers 31 47 48 49 50 51 --q4 --batch-size 1 --skip-existing

# Extract the activations from the neutral dialogues generated in the previous step
python scripts/extract_vectors.py --model "Qwen/Qwen3.6-27B" --prompts outputs/extracted/neutral_dialogues --output-dir outputs/vectors/neutral --layers 31 47 48 49 50 51 --q4 --batch-size 1 --skip-existing
```

## Calculate Emotion Vectors

The math is relative straightforward. First fit a PCA model on activation vectors from neutral stories. Then use the PCA model to "project out" the activation vectors extracted from the emotional stories. Here "project out" simply means the residuals of PCA are being used. Then the emotion vector is calculated as the mean of the residual components of the activations from each emotion (first few tokens are skipped so that the emotion is clear). The emotion vectors are re-centered by their mean.

Finally, the emotion of a token is estimated by the cosine similarity of its residual activation and the emotion vectors.

Notebook `scripts/run_emotion_probe.ipynb` does the math and visualizes the similarity of the vectors. It fits confound PCA on neutral activations, extracts mean-centered emotion vectors, runs prompts through the model, and saves emotion similarity plots.

## Relative similarity between emotion vectors

One way to quickly verify the emotion vectors is to calculate the cosine similarity between those vectors. Emotions that are similar should have positive similarity while opposite emotions should have negative similarity.

Example for layer 39 with the first 11 emotions.

|            | angry      | anxious    | ashamed    | calm       | excited    | grateful   | joyful     | loving     | sad        | surprised  | tired      |
| :--        | :--        | :--        | :--        | :--        | :--        | :--        | :--        | :--        | :--        | :--        | :--        |
| angry      | 1.0000     | 0.3654     | 0.2270     | -0.4769    | 0.0068     | -0.4168    | -0.2301    | -0.4321    | -0.0063    | 0.1894     | -0.1235    |
| anxious    | 0.3654     | 1.0000     | 0.3802     | -0.5660    | 0.0562     | -0.6282    | -0.3517    | -0.4877    | 0.3504     | 0.2727     | 0.2316     |
| ashamed    | 0.2270     | 0.3802     | 1.0000     | -0.5142    | -0.2496    | -0.1938    | -0.4241    | -0.2583    | 0.4915     | 0.0355     | 0.0297     |
| calm       | -0.4769    | -0.5660    | -0.5142    | 1.0000     | -0.1076    | 0.3634     | 0.2414     | 0.5099     | -0.2168    | -0.4139    | -0.0438    |
| excited    | 0.0068     | 0.0562     | -0.2496    | -0.1076    | 1.0000     | 0.0350     | 0.8008     | 0.0350     | -0.4118    | 0.0715     | -0.3035    |
| grateful   | -0.4168    | -0.6282    | -0.1938    | 0.3634     | 0.0350     | 1.0000     | 0.3694     | 0.6524     | -0.1392    | -0.1399    | -0.3664    |
| joyful     | -0.2301    | -0.3517    | -0.4241    | 0.2414     | 0.8008     | 0.3694     | 1.0000     | 0.3471     | -0.4600    | -0.0882    | -0.2946    |
| loving     | -0.4321    | -0.4877    | -0.2583    | 0.5099     | 0.0350     | 0.6524     | 0.3471     | 1.0000     | -0.0685    | -0.3796    | -0.2122    |
| sad        | -0.0063    | 0.3504     | 0.4915     | -0.2168    | -0.4118    | -0.1392    | -0.4600    | -0.0685    | 1.0000     | -0.0788    | 0.4143     |
| surprised  | 0.1894     | 0.2727     | 0.0355     | -0.4139    | 0.0715     | -0.1399    | -0.0882    | -0.3796    | -0.0788    | 1.0000     | -0.0782    |
| tired      | -0.1235    | 0.2316     | 0.0297     | -0.0438    | -0.3035    | -0.3664    | -0.2946    | -0.2122    | 0.4143     | -0.0782    | 1.0000     |

## Emotional Fluctuation with respect to variables

One interesting experiment is to see how the model reacts to a variable in a sentence. For example, "I just took {X} mg of tylenol for my back pain." with X varying from 500 to 16,000. See [the second figure](#figure-02) above for example.

## Evaluation Style

As for the evaluation style, initially, the prompt is formatted as `User: {prompt}\nAssistant:` following the paper, and the colon ":" after "Assistant" is used to evaluate the emotion. However, it turns out that at least for this model, formatting the prompt using the ChatML prompt template `<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n` and evaluate at the last "\n" produces more distinguishable result.

"My cat has been missing for {X} days now.", ChatML

![Comparison - ChatML](docs/emo_fluct/comparison/chatml/preset-My_cat_has_been_missing_for__X__days_now_X_vs_top_emotions.png)

"My cat has been missing for {X} days now.", Vicuna

![Comparison - ChatML](docs/emo_fluct/comparison/simple/preset-My_cat_has_been_missing_for__X__days_now_X_vs_top_emotions.png)

> Note: The emotion vectors and the neutral vector are extracted without any formatting. And this model (Qwen 3.6 27B) is fine-tuned on ChatML prompt style.

### Unexpected Value

Additionally, unexpected or unrealistic value is tested to see how will the model react. The result further demonstrates the effectiveness of the emotion probe technique.

More students than expected:

![Unexpected - More students than expected](docs/emo_fluct/unexpected/preset-I_found_out_that__X__of_my_20_students_p_X_vs_top_emotions.png)

New record for lifespan:

![Unrealistic - New record for lifespan](docs/emo_fluct/unexpected/preset-My_brother_lived_until_the_age_of__X___X_vs_top_emotions.png)

More examples can be found in [docs/emo_fluct/](docs/emo_fluct/).
