Metadata-Version: 2.4
Name: pydialign
Version: 0.1.0
Summary: Measures of verbal alignment in dyadic dialogue (Python port of dialign)
Author: Guillaume Dubuisson Duplessis
License-Expression: MIT
Project-URL: Homepage, https://github.com/GuillaumeDD/pydialign
Project-URL: Source, https://github.com/GuillaumeDD/pydialign
Project-URL: Changelog, https://github.com/GuillaumeDD/pydialign/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/GuillaumeDD/pydialign/issues
Keywords: verbal alignment,dialogue,lexical alignment,self-repetition,pattern mining,computational linguistics
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: pygstlib>=0.1
Provides-Extra: analysis
Requires-Dist: pandas>=2.0; extra == "analysis"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Dynamic: license-file

# pydialign: Measures of Verbal Alignment in Dyadic Dialogue

[![PyPI version](https://img.shields.io/pypi/v/pydialign.svg)](https://pypi.org/project/pydialign/)

pydialign is a Python port of [dialign](https://github.com/GuillaumeDD/dialign),
a software that provides automatic and generic measures of verbal alignment
and self-repetitions in dyadic dialogue based on sequential pattern mining at
the level of surface of text utterances. It is built on
[pygstlib](https://github.com/GuillaumeDD/pygstlib) for generalized
suffix-tree pattern mining.

The measures implemented by pydialign are described in the
[reference journal article](#citing-pydialign) (Lang Resources & Evaluation,
2021).

Table of contents:
- [Framework](#framework)
- [Installation](#installation)
- [Input format](#input-format)
- [Usage](#usage)
- [Analysis in notebooks (pandas)](#analysis-in-notebooks-pandas)
- [Caveats](#caveats)
- [Development](#development)
- [Contributors](#contributors)
- [Citing pydialign](#citing-pydialign)
- [License](#license)

## Framework

pydialign is based on the observation that the behaviours of dialogue
participants tend to converge and automatically align at several levels (such
as the lexical, syntactic and semantic ones). One consequence of successful
alignment at several levels between dialogue participants is a certain
repetitiveness in dialogue leading to the development of a lexicon of fixed
expressions. As a matter of fact, dialogue participants tend to automatically
establish and use fixed expressions that become dialogue routines.

More concretely, here follows an excerpt of a dialogue between a human and an
agent operated by a Woz where instances of shared lexical patterns are
coloured (from the [journal article](#citing-pydialign)):

![Excerpt of a dialogue between a human and an agent where instances of shared expressions are coloured.](https://raw.githubusercontent.com/GuillaumeDD/pydialign/main/docs/img/example-dialogue.png)

pydialign provides a framework to quantify the interactive lexical alignment
process and the self-repetition behaviour of dialogue participants (DPs) in
dyadic textual dialogues. This framework focuses on lexical patterns
occurring in dialogue utterances. It distinguishes two main types of such
patterns. The first type is **shared lexical patterns** between DPs, i.e.,
patterns that are initiated (or primed) by a DP, subsequently adopted by the
other DP and possibly reused during the dialogue by any DP. These patterns
are directly related to the interactive verbal alignment process, a
particular type of on-the-fly linguistic adaptation. They can be seen as
shared dialogue routines at the lexical level. They are a way to verbally
align and ultimately share a common language to improve understanding,
collaboration and social connection to a conversational partner. The second
type is **lexical self-repetition**. Contrary to the previous type which
considers patterns that are shared between DPs, self-repetition considers
each DP in isolation. Self-repetitions are lexical patterns appearing at
least twice in the dialogue utterances of a given DP, independently of the
other DP's utterances. Self-repetitions are directly related to the
self-consistency of the linguistic production of a given DP.

![Idea of the framework: automatic building of the shared expression lexicon to derive verbal alignment measures](https://raw.githubusercontent.com/GuillaumeDD/pydialign/main/docs/img/framework.png)

The main concept behind this model is the automatically built lexicon. For
each dialogue transcript, three lexicons are automatically computed:
- [the shared expression lexicon](examples/offline/README.md#the-shared-expression-lexicon):
  keeps track of shared expressions and valuable features about these
  expressions (e.g., who first produced this expression, its frequency);
- [one self-repetition lexicon per DP](examples/offline/README.md#the-self-repetition-lexicons):
  keeps track of self-repetitions and valuable features about these patterns
  (e.g., its frequency).

Lexicons and the dialogue transcript are leveraged by deriving offline and
online measures to quantify aspects of the verbal alignment process and the
self-repetition behaviour of DPs. Offline measures are intended to be used
for past dialogue interactions (e.g., corpus studies) while online measures
are intended for use in a dialogue system.

### Measures Provided by pydialign

pydialign provides a set of measures to characterise both:
1. the interactive verbal alignment process between dialogue participants, and
2. the self-repetition behaviour of each participant.

These measures allow the characterisation of the nature of these processes
by addressing various informative aspects such as their variety, strength,
complexity, stability, and orientation. In a nutshell:
- **variety**: the variety of shared expressions or self-repetitions emerging
  during a dialogue relative to its length. It is directly related to the
  number of unique expressions in a lexicon.
- **strength**: the strength of repetition of the (shared) lexical patterns,
  i.e., how much the patterns are reused.
- **complexity**: the complexity indicates the variety of the types of
  lexical patterns. It is here featured by Shannon entropy measures. High
  entropy indicates the presence of a wide range of lexical patterns relative
  to their lengths in number of tokens (e.g., ranging from a single word to a
  full sentence). On the contrary, low entropy indicates the predominance of
  one type of lexical pattern.
- **extension** and **stability**: the extension and stability of the
  (shared) lexical patterns are related to the size of the lexical patterns.
  The extension indicates the size of the lexical patterns. The longer it is,
  the more extended the lexical pattern is. Extension is directly linked to
  the stability of the processes since the more extended the patterns are,
  the more stable the processes are.
- **orientation**: the orientation of the interactive alignment process,
  i.e., it indicates either a symmetry (both dialogue participants initiate
  and reuse the same number of shared lexical patterns), or an asymmetry (a
  dialogue participant initiates and/or reuses more shared lexical patterns).

### Measures Characterising the Interactive Verbal Alignment Process

#### Speaker-independent

| Measure | Description |  Aspects |
| :---:   | :---       |  :---:   |
|  EV     |  Expression Variety (EV). The shared expression lexicon size normalized by the length of the dialogue (which is its total number of tokens in the dialogue).  | Variety |
|  ER     |  Expression Repetition (ER). The proportion of tokens which DPs dedicate to the repetition of a shared expression.  | Strength |
| ENTR    | Shannon entropy of the lengths in token of the shared expression instances. | Complexity |
| L       | Average length in token of the shared expression instances. | Stability |
| LMAX    | Maximum length in token of the shared expression instances. | Stability |

#### Speaker-dependent

| Measure | Description |  Aspects |
| :---:   | :---       |  :---:   |
|  IE_S     |  Initiated Expression (IE) for locutor S. Ratio of shared expressions initiated by locutor S.   | Orientation |
|  ER_S     |  Expression Repetition (ER) for locutor S. Ratio of tokens produced by S belonging to an instance of a shared expression.  | Strength |

### Measures Characterising Self-Repetition Behaviour of each Dialogue Participant

| Measure | Description |  Aspects |
| :---:   | :---       |  :---:   |
|  SEV_S  |  Self-Expression Variety (SEV) for locutor S. For locutor S, the self-repetition lexicon size normalized by the total number of tokens produced by S in the dialogue.  | Variety |
|  SER_S     |  Self-Expression Repetition (SER) for locutor S. The proportion of tokens which locutor S dedicates to self-repetition.| Strength |
| SENTR_S    |  Shannon entropy of the length in token of the self-repetitions from S. | Complexity |
| SL_S       |  Average length in tokens of the self-repetitions from S. | Stability |
| SLMAX_S    |  Maximum length in token of the self-repetitions from S. | Stability |

### Synthetic Presentation of the Provided Measures

| Aspect      | Speaker-independent Measures (*) | Speaker-dependent Measures (**) |
| :---:       | :---:                            | :---:                           |
| Variety     | EV                               |  SEV_S                          |
| Strength    | ER                               |  ER_S, SER_S                    |
| Complexity  | ENTR                             |  SENTR_S                        |
| Stability   | L, LMAX                          |  SL_S, SLMAX_S                  |
| Orientation |  --                              |  IE_S                           |

(*) all related to interactive verbal alignment

(**) measures starting with 'S' relate to self-repetition behaviour, the
others to interactive verbal alignment

## Installation

```bash
uv add pydialign            # or: pip install pydialign
uv add 'pydialign[analysis]'  # with the pandas analysis layer
```

Requires Python ≥ 3.10. The only required runtime dependency is
[pygstlib](https://pypi.org/project/pygstlib/).

## Input format

One dialogue per file, one utterance per line, tab-separated, UTF-8:

```
LOCUTOR<TAB>TOKENIZED UTTERANCE
```

At most 2 distinct locutors per dialogue. Utterances should be pre-tokenized
(tokens separated by spaces); `-n/--normalisation` lowercases for you.

## Usage

### `dialign` for corpus studies

```bash
dialign -i corpus/ -o output-dialign/
```

produces, per dialogue `<name>` (file name with `.` → `_`):
- `<name>-lexicon.tsv` — the shared expression lexicon;
- `<name>-lexicon-self-rep-A.tsv`, `<name>-lexicon-self-rep-B.tsv` — the
  self-repetition lexicons (speaker A is the alphabetically-first locutor);
- `<name>-dialogue.txt` — the transcript tagged with expressions
  (`[established]`, `_not yet established_`) plus per-turn details;

and, per corpus: `metrics-speaker-independent.tsv` and
`metrics-speaker-dependent.tsv` (one row per dialogue).

To restrict which files of the input directory are processed:

```bash
dialign -i corpus/ -o output-dialign/ \
    -p "dialogue-" \  # required filename prefix
    -s "-cleaned" \   # required filename suffix
    -e "tsv"          # required extension (without the '.')
```

**➡ Follow the [corpus-study tutorial](examples/offline/README.md).**

### `dialign-online` for interactive purposes

```bash
dialign-online -f history.tsv              # interactive REPL
dialign-online -f history.tsv -o out.tsv   # batch export
```

The REPL scores candidate utterances against the running dialogue history
(commands: `h` help, `s` score, `a` add, `p` print, `e` export, `q` quit),
reporting **DER** (dynamic shared-expression repetition) and **DSER**
(dynamic self-repetition). Batch mode replays a transcript, scoring each
utterance incrementally.

**➡ Follow the [dialign-online tutorial](examples/online/README.md).**

### `dialign-online-metrics` — per-utterance SER/ER/Activation

```bash
dialign-online-metrics -i dialogues.tsv -a 2.0 -b 2.0 -l 0.5
```

The input file may hold several dialogues separated by blank lines. Prints a
CSV block per dialogue to stdout with per-utterance SER, ER and a cognitive
activation score (`-a` alpha: temporary activation, `-b` beta: permanent
activation, `-l` lambda ∈ [0,1]: their relative weight).

### `dialign-rstp` / `dialign-sr` — corpus lexicon exporters

```bash
dialign-rstp -i corpus/ -e tsv -r inter -o lexicon.csv
dialign-sr   -i corpus/ -e tsv -o corpus-lexicon-self-repetition
```

`dialign-rstp` aggregates per-dialogue lexicons into one corpus inventory
(`-r all|intra|inter` pattern filter, `-d` delexicalisation of
negotiation-domain nouns, `-m` begin/end markers). `dialign-sr` writes two
corpus self-repetition inventories, `<prefix>-A.csv` and `<prefix>-B.csv`.

### Library usage

```python
from pydialign import (
    DialogueHistory, Utterance, build_lexicon, read_dialogue,
    tokenize_without_markers,
)
from pydialign.metrics.offline import DialogueLexiconMeasures

# Offline: measures for one dialogue file
dialogue = read_dialogue("examples/offline/corpus/excerpt-alice.tsv",
                         tokenize_without_markers)
lexicon = build_lexicon(
    dialogue.utterances, dialogue.get_speaker, dialogue.get_raw_speaker
)
measures = DialogueLexiconMeasures(lexicon)
print(measures.expression_repetition, measures.expression_variety)
# Output: 0.2 0.10476190476190476

# Online: score the next utterance of a running dialogue
history = DialogueHistory([Utterance("Alice", "what's up ?")])
scoring = history.score(Utterance("Bob", "so what's up my dear ?"))
print(round(scoring.der, 4), [e.surface for e in scoring.shared_expressions])
# Output: 0.3333 ["what's up"]
```

## Analysis in notebooks (pandas)

The optional [`pydialign.analysis`](src/pydialign/analysis.py) module turns
corpora, lexicons and online scores into pandas DataFrames
(`pip install 'pydialign[analysis]'`):

```python
from pydialign.analysis import corpus_measures_frame

df = corpus_measures_frame("examples/offline/corpus/")
print(df[["num_tokens", "els", "ev", "er", "entr", "l", "lmax"]])
# Output:
#                    num_tokens  els        ev   er      entr     l  lmax
# dialogue
# excerpt-alice_tsv         105   11  0.104762  0.2  1.756624  1.88     4
```

Also available: `lexicon_frame` (one row per expression), `turns_frame`
(per-turn token coverage and the cumulative ER trajectory),
`online_scoring_frame` (per-utterance DER/DSER), and `analyze_corpus` (one
call loading everything). **➡ See the
[analysis notebook](examples/notebook/analysis.ipynb).**

## Caveats

- **Dyadic dialogues only**: at most 2 distinct locutors per transcript.
- **Surface-level**: pydialign works on the surface form of what you give
  it — utterances should be pre-tokenized and consistently
  cased/normalized.
- **Parity with the Scala dialign**: lexicon inventories and tagged
  transcripts are byte-identical to the original's; metric values are
  numerically identical (number *formatting* follows Java's
  `Double.toString`, and integer-valued columns print as integers). The one
  intentional deviation: where the Scala code prints `Set` contents in
  arbitrary order (turn details, exported expression lists), pydialign
  renders a deterministic sorted order.
- **pandas is optional**: the core install never imports pandas;
  `pydialign.analysis` raises a helpful `ImportError` if the `[analysis]`
  extra is missing.

## Development

```bash
uv venv && uv pip install -e '.[dev]'
uv run pytest          # unit, Scala-oracle, golden, property and naive-reference tests
uv run ruff check .
```

The project is managed BMAD-style: see [`docs/prd.md`](docs/prd.md),
[`docs/architecture.md`](docs/architecture.md), the story backlog in
[`docs/stories/`](docs/stories/) and the QA reviews in
[`docs/qa/`](docs/qa/).

## Contributors

- Guillaume Dubuisson Duplessis (2017, 2020, 2021, 2022; Python port 2026)

## Citing pydialign

If you use pydialign or dialign in research, please cite the reference
journal article:

> Dubuisson Duplessis, G., Langlet, C., Clavel, C., Landragin, F.:
> *Towards alignment strategies in human-agent interactions based on
> measures of lexical repetitions*, Lang Resources & Evaluation 55, 353–388
> (2021).
> [HAL](https://hal.archives-ouvertes.fr/hal-03147824) |
> [DOI](https://dx.doi.org/10.1007/s10579-021-09532-w)

See also:

> Dubuisson Duplessis, G., Clavel, C., Landragin, F.: *Automatic Measures to
> Characterise Verbal Alignment in Human-Agent Interaction*, 18th Annual
> Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL),
> 2017, pp. 71–81.
> [Paper](https://www.sigdial.org/files/workshops/conference18/proceedings/pdf/SIGDIAL10.pdf)

### Contact

- [Guillaume Dubuisson Duplessis](http://www.dubuissonduplessis.fr/)
- [Chloé Clavel](https://clavel.wp.imt.fr/)
- [Frédéric Landragin](http://fred.landragin.free.fr/)

## License

MIT — see the [LICENSE.txt](LICENSE.txt) file.

pydialign is a port of [dialign](https://github.com/GuillaumeDD/dialign)
(licensed under CECILL-B), published under MIT by the original author.
