Metadata-Version: 2.4
Name: vision_enabled_dialogue
Version: 0.3.3
Summary: A dialogue system that combines speech and images in a conversation history with summarisation strategies to improve contextual understanding.
Author-email: Giulio Antonio Abbo <giubots@protonmail.com>
Maintainer: Giulio Antonio Abbo
Maintainer-email: giubots@protonmail.com
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/giubots/vision-enabled-dialogue
Project-URL: Issues, https://github.com/giubots/vision-enabled-dialogue/issues
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: annotated-types==0.7.0
Requires-Dist: anyio==4.14.1
Requires-Dist: certifi==2026.6.17
Requires-Dist: distro==1.9.0
Requires-Dist: exceptiongroup==1.3.1
Requires-Dist: h11==0.16.0
Requires-Dist: httpcore==1.0.9
Requires-Dist: httpx==0.28.1
Requires-Dist: idna==3.18
Requires-Dist: jiter==0.16.0
Requires-Dist: openai==2.17.0
Requires-Dist: pydantic==2.12.5
Requires-Dist: pydantic_core==2.41.5
Requires-Dist: sniffio==1.3.1
Requires-Dist: tqdm==4.68.4
Requires-Dist: typing-inspection==0.4.2
Requires-Dist: typing_extensions==4.16.0
Dynamic: license-file
Dynamic: maintainer
Dynamic: maintainer-email

# VISION-ENABLED DIALOGUE

A dialogue system that combines speech and images in a conversation history with summarisation strategies to improve contextual understanding. It uses a configurable (for example, text-only or multimodal) language model to generate responses. It can be used as a Python library or as a ROS 2 node.

This repository contains the implementation described in the paper _"I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots"_ by Giulio Antonio Abbo and Tony Belpaeme (March 2025) [available here](https://giubots.net/publications/was-blind-but/). Link to the [HRI'25 LBR paper](https://doi.org/10.1109/HRI61500.2025.10973830). For the version of the code used in the paper, see the [v0.2.0](https://doi.org/10.5281/zenodo.14627887).

## Contents

- [Features](#features)
- [Installation](#installation)
- [Usage - Get started](#usage---get-started)
- [Usage - Generating responses](#usage---generating-responses)
- [Usage - Rendering outputs and handling interruptions](#usage---rendering-outputs-and-handling-interruptions)
- [Usage - Manage prompt length](#usage---manage-prompt-length)
- [ROS 2 compatibility](#ros-2-compatibility)
- [Furhat robot compatibility](#furhat-robot-compatibility)
- [Contributing](#contributing)
- [Reference](#reference)

## Features

- Separates conversation management from response generation, so the dialogue flow, model choice, etc., can be changed independently.
- Supports both text-only generation and vision-language generation through a shared response-generator interface.
- Can choose between the text model and the vision model with a lightweight chooser model, or force the next turn to use the VLM.
- In addition to the usual prompt instructions, the system allows for steering instructions to be sent per turn or persistently; these are put at the end of the prompt to further guide the model's output.
- Applies background history-compaction strategies so older frames (images) and mixed content can be summarised instead of keeping every raw item in the prompt.
- Streams generated text in sentence-sized chunks for faster interactions.
- Stores camera frames, spoken turns, thoughts, and generated turns in a typed conversation history.
- Preserves generation metadata in the history for later inspection.

## Installation

For the standalone Python entrypoint, install the main dependencies:

```shell
pip install vision-enabled-dialogue

# Or, if you want to install from source, clone the repository and run:
pip install -r requirements.txt

# To run the examples, you will also need OpenCV:
pip install opencv-python
```

The default GPT client in this repository is configured in code and points to `http://localhost:11434/v1` with the API key `ollama`. If you want to use a different backend, set the environment variables `OPENAI_API_BASE_URL` and `OPENAI_API_KEY` before running the code, or pass a custom `ResponseGenerator` to the `DialogueManager` constructor. To use OpenAI models, set `OPENAI_API_BASE_URL` to an empty string.

## Usage - Get started

The script in `main.py` shows a simple example of using the `DialogueManager` with a webcam and a vision-enabled LLM. Through the default `SummariseHeavy` strategy with a `HidingSummariser, it always keeps four images in memory and hides the rest. It also contains a more in-depth example of how to customise the dialogue manager.

Ensure you have Ollama running with the `gemma3:27b` model on port `11434`, then run the script:

```shell
python main.py
```

## Usage - Generating responses

Responses are generated by the `DialogueManager` class, which takes a `ResponseGenerator`, which represent a set of instructions for the LLM. It uses a `LLM` implementation to produce the actual responses. Currently only the `GPT` implementation is available, which uses the OpenAI conventions, also compatible with Ollama.

The response is streamed in sentence-sized chunks, so the first part of the response is available before the entire generation is complete. The metadata about the generation is stored in the conversation history, so you can inspect it later.

## Usage - Rendering outputs and handling interruptions

Provide a function to handle the generated text chunks to the `DialogueManager` constructor, for example to print them to the console or send them to a TTS engine. If the speech is interrupted, this function should return the text that was actually spoken, so the dialogue manager can store it in the conversation history. The full generated text is also stored in the conversation history.

The `ConversationHistory` class uses two callback functions to notify when a new part is added or when a part is summarised. You can use these callbacks to update the UI or trigger other actions.

## Usage - Manage prompt length

Conversation length is controlled by the history layer in `vision_enabled_dialogue/conversation_history`.
The main building blocks are:

- `ConversationHistory`, which stores the ordered `ConversationPart` objects and applies growth strategies in the background.
- `GrowthStrategy`, which defines how to summarise parts of the conversation using a `Summariser`.
  - `SummariseHeavy`, summarises older _heavy_ parts (e.g., frames) once their count exceeds a limit.
  - `KeepLastNStrategy`, which keeps the last N _important_ parts (e.g., text) and summarises the rest.
- `Summariser`, which defines how to summarise conversation parts, when a growth strategy requires it.
  - `FrameSummariser`, which produces a summary of images.
  - `MixedSummariser`, which produces a text summary for mixed conversation parts.
  - `HidingSummariser`, which hides parts from the transcript (but does not remove them from memory).

For instance, **to summarise the initial part of the conversation** when it becomes too long, you can use a `KeepLastNStrategy` with a `MixedSummariser`.

If you want **to reduce the prompt size**, substituting consecutive frames with a text summary, you can use `SummariseHeavy` strategy with a `FrameSummariser`, defining the minimum and maximum number of frames to keep in the prompt. If instead you want to hide older parts from the prompt, while keeping them in memory, you can use `HidingSummariser` instead.

## ROS 2 compatibility

This package is compatible with ROS 2 (tested on Humble). An example node is provided in `vision_enabled_dialogue/main_ros.py`. It declares the `persistent_steering` parameter and uses these interfaces:

- Subscribed topics: `/humans/voices/anonymous_speaker/speech`, `dialogue/steering`, `/camera/image_raw`, `/camera/image_raw/compressed`
- Published topics: `/conversation`, `/transcript`
- Action client: `/tts_engine/tts`

To test the node, you can install this repo as a ROS 2 Python package (in `your-workspace/src/vision_enabled_dialogue`), make sure the dependencies are installed, then build the workspace, source and run it with:

```shell
colcon build --symlink-install
source ./install/setup.bash
ros2 run vision_enabled_dialogue main
```

## Furhat robot compatibility

Following the deprecation of Furhat's RemoteAPIs, this repository no longer provides a Furhat-specific entrypoint. You can use this library in your existing Furhat project.

## Contributing

Contributions are welcome! Please open an issue or submit a pull request. For major changes, please open an issue first to discuss what you would like to change.

## Reference

If you use this code in your work, please cite the paper:

```bibtex
@inproceedings{10.5555/3721488.3721641,
author = {Abbo, Giulio Antonio and Belpaeme, Tony},
title = {I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots},
year = {2025},
publisher = {IEEE Press},
booktitle = {Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction},
pages = {1176–1180},
numpages = {5},
keywords = {conversation, dialogue, hri, large language model, prompt engineering, ros, vision language model},
location = {Melbourne, Australia},
series = {HRI '25}
}
```
