Metadata-Version: 2.3
Name: wattelse
Version: 1.2.3
Summary: Flexible RAG
License: MPL 2.0
Author: Jérôme Picault
Author-email: jerome.picault@rte-france.com
Requires-Python: >=3.11,<3.13
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: PyJWT (>=2.10.1,<3.0.0)
Requires-Dist: accelerate (>=1.6.0,<2.0.0)
Requires-Dist: bert-score (>=0.3.13,<0.4.0)
Requires-Dist: black (>=24.10.0,<25.0.0)
Requires-Dist: bs4 (>=0.0.2,<0.0.3)
Requires-Dist: chromadb (==0.5.11)
Requires-Dist: django (>=5.2,<6.0)
Requires-Dist: docxtpl (>=0.18.0,<0.19.0)
Requires-Dist: evaluate (>=0.4.3,<0.5.0)
Requires-Dist: fastapi (>=0.115.12,<0.116.0)
Requires-Dist: fschat (>=0.2.36,<0.3.0)
Requires-Dist: langchain (>=0.3.23,<0.4.0)
Requires-Dist: langchain-chroma (>=0.1.4,<0.2.0)
Requires-Dist: langchain-community (>=0.3.21,<0.4.0)
Requires-Dist: langchain-openai (>=0.3.12,<0.4.0)
Requires-Dist: llama-index-core (>=0.11.16,<0.12.0)
Requires-Dist: locust (>=2.34.1,<3.0.0)
Requires-Dist: loguru (>=0.7.3,<0.8.0)
Requires-Dist: mammoth (>=1.9.0,<2.0.0)
Requires-Dist: markdown (>=3.7,<4.0)
Requires-Dist: numpy (<2)
Requires-Dist: openai (>=1.72.0,<2.0.0)
Requires-Dist: openpyxl (>=3.1.5,<4.0.0)
Requires-Dist: pandas (>=2.2.3,<3.0.0)
Requires-Dist: plotly (>=5.24.1,<6.0.0)
Requires-Dist: pymupdf (>=1.25.5,<2.0.0)
Requires-Dist: pysqlite3-binary (>=0.5.4,<0.6.0)
Requires-Dist: python-docx (>=1.1.2,<2.0.0)
Requires-Dist: python-multipart (>=0.0.12,<0.0.13)
Requires-Dist: python-pptx (>=1.0.2,<2.0.0)
Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
Requires-Dist: rank-bm25 (>=0.2.2,<0.3.0)
Requires-Dist: reportlab (>=4.3.1,<5.0.0)
Requires-Dist: scikit-learn (>=1.6.1,<2.0.0)
Requires-Dist: scipy (>=1.15.2,<2.0.0)
Requires-Dist: seaborn (>=0.13.2,<0.14.0)
Requires-Dist: sentence-transformers (>=3.4.1,<4.0.0)
Requires-Dist: spacy (<3.8)
Requires-Dist: sparqlwrapper (>=2.0.0,<3.0.0)
Requires-Dist: streamlit (>=1.44.1,<2.0.0)
Requires-Dist: tiktoken (>=0.7.0,<0.8.0)
Requires-Dist: torch (==2.4.0)
Requires-Dist: tqdm-joblib (>=0.0.4,<0.0.5)
Requires-Dist: tree_sitter (>=0.24.0,<0.25.0)
Requires-Dist: tree_sitter_languages (>=1.10.2,<2.0.0)
Requires-Dist: typer (>=0.12.5,<0.13.0)
Requires-Dist: unstructured (>=0.15.14,<0.16.0)
Requires-Dist: uvicorn (>=0.31.1,<0.32.0)
Requires-Dist: vllm (>=0.6.3,<0.7.0)
Requires-Dist: whitenoise (>=6.9.0,<7.0.0)
Requires-Dist: xlsx2html (>=0.6.2,<0.7.0)
Description-Content-Type: text/markdown

# WattElse project

WattElse is a NLP suite developed for the needs of RTE (Réseau de Transport d'Electricité).

It is composed of two main modules:

- a simple chatbot interface to interact with any LLM -> **WattElse GPT**
- a Retrieval Augmented Generation (RAG) application -> **WattElse DOC**

Some services are used by several applications/users at the same time. To optimize resource use, these services are implemented in the form of APIs. A description of these services is available in [wattelse/api](wattelse/api).

WattElse also includes helper modules that provide additional functionalities such as summaries, web scrapping, and document parsing.

## Installation

Before trying to install WattElse, you first need to ensure you have:

- `python >= 3.10`
- `sqlite3 >= 3.35`

Clone this project:

```bash
git clone https://github.com/rte-france/wattelse.git
```

Create a virtual environment:

```bash
cd wattelse
python -m venv .venv
source .venv/bin/activate
```

Install WattElse and its dependencies:

```bash
./install.sh
```

## Environnement variables

To run WattElse, you need to set the following environment variables:

- `WATTELSE_BASE_DIR`: path where WattElse data will be stored
- `WATTELSE_CLIENT_SECRET`: secret used by the Django WattElse app to communicate with internal APIs. This value should match the one defined on the `$HOME/client_registry.json` file. Refer to the [API documentation](wattelse/api/README.md) for details.
- `DJANGO_SECRET_KEY`: Django secret key (see [Django documentation](https://docs.djangoproject.com/en/4.2/ref/settings/#std-setting-SECRET_KEY))

To create a Django `SECRET_KEY`, run the following code in a python shell:

```python
from django.core.management.utils import get_random_secret_key
print(get_random_secret_key())
```

## Django initialization

To initialize the Django database, follow these steps:

- Go to the django `web_app` folder:

```bash
cd wattelse/web_app
```

- Create Django tables:

```bash
python manage.py makemigrations
python manage.py migrate
```

- Create Django superuser:

```bash
python manage.py createsuperuser
```

- Collect static files (needed when `DEBUG=False` in Django)

```bash
python manage.py collectstatic
```

## Launch WattElse

To launch WattElse with all services, go to WattElse root folder and run:

```bash
./start_all_services.sh
```

This script starts all services in separated `screens`:

- Embedding API
- RAGOrchestrator API
- Django server

Django web app should be running at: http://localhost:8000

If everything has been set up correctly, you should see the following screen on http://localhost:8000

![img.png](wattelse/docs/images/home_page_wattelse.png)

For your first connection, you can use the superuser credentials you created earlier or create a new user.

## Stop WattElse

To stop all services, run:

```bash
./stop_all_services.sh
```

## Hardware requirements

By default, WattElse only loads an embedding model on start. It requires around 2GB of VRAM if loaded on GPU. To change the embedding model, see [wattelse/api](wattelse/api) and [wattelse/api/embedding](wattelse/api/embedding).

The LLM used depends on the RAG config. By default, no local LLM is loaded so you need to link RAG config to a remote LLM (OpenAI, Azure...). For RAG config management, see [wattelse/rag_backend](wattelse/rag_backend).

If you want to load a local LLM using `vLLM`, you need to have enough VRAM to load the model. For example, the `llama-3.1-8B-instruct` model requires around 16GB of VRAM.

# RAG service

## Overview of main steps

```mermaid
flowchart TB
    subgraph Preprocessing["Data Preprocessing"]
        direction TB
        RawDocs["Raw Documents"]
        TextChunks["Text Chunks"]
        VectorDB[(Vector Database)]

        RawDocs --> |"Text Splitting"| TextChunks
        TextChunks --> |"Embedding Generation"| VectorDB
    end

    subgraph QueryProcessing["Query Processing"]
        direction TB
        UserQuery["User Query"]
        QueryEmbed["Query Embedding"]
        SimilaritySearch["Similarity Search"]
        ContextRetrieval["Context Retrieval"]

        UserQuery --> QueryEmbed
        QueryEmbed --> SimilaritySearch
        SimilaritySearch --> ContextRetrieval
    end

    subgraph ResponseGeneration["Response Generation"]
        direction TB
        LLM["Large Language Model"]
        Response["Generated Response"]

        ContextRetrieval --> LLM
        UserQuery --> LLM
        LLM --> Response
    end

    VectorDB --> SimilaritySearch

```

## Description of components

```mermaid
flowchart TB
    subgraph Input
        User[User Interface]
    end

    subgraph "RAG Core Components"
        RAG[RAG Backend]
        LLM[LLM Service]
        Embedding[Embedding Service]
    end

    subgraph "Storage Layer"
        VectorDB[(Vector Database)]
        DocStore[(Document Store)]
    end

    subgraph "Document Processing"
        Parser[Document Parser]
        Chunker[Text Chunker]
    end

    User <--> RAG
    RAG <--> LLM
    RAG <--> Embedding
    RAG <--> VectorDB
    RAG <--> DocStore
    DocStore <--> Parser
    Parser <--> Chunker

    classDef core fill:#e1f5fe,stroke:#01579b
    classDef storage fill:#e8f5e9,stroke:#1b5e20
    classDef processing fill:#fff3e0,stroke:#e65100
    classDef input fill:#f3e5f5,stroke:#4a148c

    class RAG,LLM,Embedding core
    class VectorDB,DocStore storage
    class Parser,Chunker processing
    class User input

```

## Simplified sequence diagram for RAG

```mermaid


sequenceDiagram
    participant User
    participant UI as User Frontend
    participant LLM as LLM Service
    participant RAG as RAG Backend
    participant Embedding as Embedding Service
    participant VectorDB as Vector Database
    participant DocStore as Document Store
    participant Parser as Document Parser
    participant Chunker as Text Chunker

    User->>UI: Authenticate
    UI-->>User: Authentication OK

    UI->>UI: Create Session

    User->>UI: Upload files
    UI->>RAG: New files
    RAG->>VectorDB: Check cache
    VectorDB->>VectorDB: check file names
    VectorDB-->>RAG: Files not in cache
    RAG->>Parser: Parse files(files)
    Parser->>Chunker: Chunk(documents)
    Chunker-->>RAG: documents chunks
    RAG->>DocStore: store(files)
    RAG->>VectorDB: store(documents chunks)
    VectorDB->>Embedding: compute embeddings(documents chunks)
    Embedding->>Embedding: compute
    Embedding-->>VectorDB: computed embeddings
    VectorDB->>VectorDB: store
    VectorDB-->>RAG: storage ok

    User->>UI: <br><br><br><br>Enter Query
    UI->>RAG: Submit Query
    RAG->>Embedding: Generate Query Embedding
    Embedding-->>RAG: Query Vector

    RAG->>VectorDB: Semantic Search
    VectorDB->>VectorDB: Similarity function
    VectorDB-->>RAG: Relevant Document Extracts

    RAG->>LLM: Generate Response (Query + Context + (history))
    LLM-->>RAG: Generated Response
    RAG-->>UI: Final Answer
    UI-->>User: Displayed answer

    User-->>UI: <br>Provide feedback
    UI-->>UI: Store feedback



    User->>UI: <br><br><br>Logout / Timeout
    UI->>UI: Kill Session

    Note over DocStore,Chunker: Document Processing Pipeline
    Note over RAG,VectorDB: Retrieval Pipeline
    Note over RAG,LLM: Generation Pipeline
```

## Main code dependencies

```mermaid
flowchart TB
   wattelse["wattelse"]

   subgraph ML["Machine Learning"]
       torch["torch"]
       sklearn["scikit-learn"]
       accelerate["accelerate"]
       scipy["scipy"]
       numpy["numpy"]
   end

   subgraph LLM["LLM & RAG"]
       langchain["langchain"]
       langchain_comm["langchain-community"]
       langchain_chroma["langchain-chroma"]
       langchain_openai["langchain-openai"]
       llama_index["llama-index-core"]
       openai["openai"]
       chromadb["chromadb"]
       sent_trans["sentence-transformers"]
       vllm["vllm"]
       fschat["fschat"]
       tiktoken["tiktoken"]
   end

   subgraph Web["Web Framework"]
       django["django"]
       fastapi["fastapi"]
       streamlit["streamlit"]
       uvicorn["uvicorn"]
   end

   subgraph Doc["Document Processing"]
       docxtpl["docxtpl"]
       python_docx["python-docx"]
       python_pptx["python-pptx"]
       pymupdf["pymupdf"]
       unstructured["unstructured"]
       mammoth["mammoth"]
       xlsx2html["xlsx2html"]
   end

   subgraph Data["Data Processing"]
       pandas["pandas"]
       plotly["plotly"]
       seaborn["seaborn"]
       bs4["bs4"]
   end

   wattelse --> ML
   wattelse --> LLM
   wattelse --> Web
   wattelse --> Doc
   wattelse --> Data
```

