Metadata-Version: 2.4
Name: radlab-llm-router-plugins
Version: 0.1.1
Summary: Plugins for the LLM Router (guardrails, maskers, etc.)
Home-page: https://github.com/radlab-dev-group/llm-router-plugins
Author: RadLab.dev Team
Author-email: hello@radlab.dev
License: Apache-2.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: radlab-pii-classification>=0.1.0
Provides-Extra: ml
Requires-Dist: faiss-cpu; extra == "ml"
Requires-Dist: sentence-transformers; extra == "ml"
Requires-Dist: numpy; extra == "ml"
Requires-Dist: scipy; extra == "ml"
Provides-Extra: ml-gpu
Requires-Dist: faiss-gpu; extra == "ml-gpu"
Requires-Dist: sentence-transformers; extra == "ml-gpu"
Requires-Dist: numpy; extra == "ml-gpu"
Requires-Dist: scipy; extra == "ml-gpu"
Provides-Extra: rag
Requires-Dist: langchain; extra == "rag"
Requires-Dist: Pillow; extra == "rag"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

## Overview

The **LLM‑Router** project ships with a modular plugin system that lets you plug‑in **anonymizers** (also called
*maskers*) and **guardrails** into request‑processing pipelines.  
Each plugin implements a tiny, well‑defined interface (`apply`) and can be composed in an ordered list to form a *
*pipeline**. Pipelines are instantiated by the `MaskerPipeline` and `GuardrailPipeline` classes and are driven
automatically by the endpoint logic in `endpoint_i.py`.

---

## 1. Anonymizers (Maskers)

### 1.1 What they do

* **Goal** – Remove or replace personally‑identifiable information (PII) from a payload before it reaches the LLM or an
  external service.
* **Typical strategy** – Run a pipeline of maskers that locate spans corresponding to IDs, emails, IPs, etc., and
  replace each span with a placeholder such as `{{MASKED_ITEM}}`.

### 1.2 Built‑in anonymizer plugins

| Plugin                                         | Description                                                                                                                                       | Technical notes                                                                                                                                                            |
|------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **FastMaskerPlugin** (`fast_masker_plugin.py`) | Thin wrapper around the `FastMasker` utility class. Receives a JSON‑compatible payload and returns the same payload with all detected PII masked. | Implements `PluginInterface`. The heavy lifting is delegated to `FastMasker.mask_payload(payload)`. No extra I/O; the `FastMasker` instance is created once in `__init__`. |

### 1.3 How a masker is used

1. The endpoint (e.g. `EndpointI._do_masking_if_needed`) checks the global flag `FORCE_MASKING`.
2. If enabled, it creates a `MaskerPipeline` with the list of masker plugin identifiers (e.g. `["fast_masker"]`).
3. The pipeline calls each plugin’s `apply` method sequentially, feeding the output of one as the input of the next.
4. The final payload – now stripped of PII – proceeds to the rest of the request flow (guardrails, model dispatch,
   etc.).

---

## 2. Guardrails

### 2.1 What they do

* **Goal** – Verify that a request (or its response) complies with policy rules (e.g. no hateful, illegal, or unsafe
  content).
* **Typical strategy** – Split the payload into manageable text chunks, run a pipeline of guardrails, aggregate
  per‑chunk scores, and decide whether the overall request is safe.

### 2.2 Built‑in guardrail plugins

| Plugin                                             | Description                                                                                                                                                                                                                  | Technical notes                                                                                                                                                                                                                       |
|----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **NASKGuardPlugin** (`nask_guard_plugin.py`)       | HTTP‑based guardrail that forwards the payload to the external NASK guardrail service (`/nask_guard` endpoint) and returns a boolean *safe* flag together with the raw response.                                             | Inherits from `HttpPluginInterface`. The `apply` method calls `_request(payload)` (provided by the base class) and extracts `results["safe"]`. Errors are caught and logged; on failure the plugin returns `(False, {})`.             |
| **SojkaGuardPlugin** (`sojka_guard_plugin.py`)     | HTTP‑based guardrail that forwards the payload to the **Sójka** guardrail service (`/sojka_guard` endpoint) and returns a safety flag.                                                                                       | Mirrors the design of `NASKGuardPlugin`. The `endpoint_url` is built from the `LLM_ROUTER_GUARDRAIL_SOJKA_GUARD_HOST` environment variable. On success it returns `(True, response)`, otherwise `(False, {})`.                        |
| **(Implicit) GuardrailProcessor** (`processor.py`) | Core logic used by the internal NASK guardrail Flask route (`nask_guardrail`). Tokenises the payload, creates overlapping chunks, runs a Hugging‑Face `text‑classification` pipeline, and produces a detailed safety report. | Handles model loading (`AutoTokenizer`, `pipeline("text‑classification")`), chunking (`_chunk_text`), and scoring thresholds (`MIN_SCORE_FOR_SAFE`, `MIN_SCORE_FOR_NOT_SAFE`). Returns a dict: `{"safe": <bool>, "detailed": [...]}`. |

### 2.3 How a guardrail is used

1. The endpoint calls `_is_request_guardrail_safe(payload)` (or the analogous response guardrail).
2. If `FORCE_GUARDRAIL_REQUEST` is true, a `GuardrailPipeline` is built from the configured plugin IDs (e.g.
   `["nask_guard", "sojka_guard"]`).
3. The pipeline iterates over each guardrail plugin; each `apply` returns `(is_safe, message)`.
4. The first plugin that reports `is_safe=False` short‑circuits the pipeline and the request is rejected with a 400/500
   error payload.

---

## 2.5 ML-Based PII Classification

For cases where regex patterns alone are insufficient (e.g. context-dependent PII detection), the project integrates
with the **anonymizer-model** repository:

**Repository**: [radlab-dev-group/anonymizer-model](https://github.com/radlab-dev-group/anonymizer-model)

| Feature               | Description                                                                                     |
|-----------------------|-------------------------------------------------------------------------------------------------|
| **Approach**          | NER (Named Entity Recognition) model based on Hugging Face `AutoModelForTokenClassification`    |
| **What it does**      | Identifies PII entities in Polish text with context-aware detection (not just pattern matching) |
| **Training pipeline** | Configurable via JSON, logs to W&B, exports best model (F1 macro) to `final_model/`             |
| **REST API**          | Flask service supporting multiple model versions with optional dynamic quantization             |
| **CLI**               | `pii-classifier convert` / `generalise` / `report` for data prep and analysis                   |
| **Inference**         | Sub-token merging, punctuation cleaning, gap preservation for human-readable entity spans       |
| **Web tester**        | HTML/JS interface for real-time PII detection testing                                           |

### How it complements regex maskers

| Approach                              | Strength                                                           | Best for                                                    |
|---------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------------|
| **Regex maskers** (FastMasker)        | Deterministic, zero false negatives for known formats, fast        | Structured IDs (KRS, NIP, PESEL, NRB, REGON, VIN, etc.)     |
| **PII classifier** (anonymizer-model) | Context-aware, handles unknown formats, generalizes across domains | Free-text entities, names, addresses, context-dependent PII |

### Quick start

```bash
# Clone and install
git clone https://github.com/radlab-dev-group/anonymizer-model.git
cd anonymizer-model
pip install .

# Run the API (serves multiple model versions)
python3 -m pii_classification.api.app

# API endpoints
#   GET  /models   — list available models + default
#   POST /predict   — { "text": "...", "model": "optional_name" }
```

---

## 2.6 Polish Identification Regex Patterns

The **FastMaskerPlugin** ships with a comprehensive set of rules for detecting Polish business and personal identifiers.
These rules use regex matching + checksum/form validation to minimize false positives.

### Available Polish rules

| Rule                | Placeholder        | Format                                            | Validation                                           |
|---------------------|--------------------|---------------------------------------------------|------------------------------------------------------|
| **KrsRule**         | `{{KRS}}`          | `1234567890` or `123-456-78-90` / `123 456 78 90` | 10 digits (format-only)                              |
| **NrbRule**         | `{{NRB}}`          | `PL58105012981000009062923173` (26 digits)        | 26 digits                                            |
| **NipRule**         | `{{NIP}}`          | `1234567890` or `123-456-78-90`                   | Checksum with weights `[6,5,7,2,3,4,5,6,7]` mod 11   |
| **PeselRule**       | `{{PESEL}}`        | 11 digits                                         | Checksum with weights `[1,3,7,9,1,3,7,9,1,3]` mod 10 |
| **RegonRule**       | `{{REGON}}`        | 9 or 14 digits                                    | Checksum (different weights per form)                |
| **BankAccountRule** | `{{BANK_ACCOUNT}}` | `PL58 1050 1298 1000 0090 6292 3173`              | Polish IBAN (28 chars), supports masked `XX` groups  |

### How they work

Each rule follows the same pattern:

1. **Regex match** — detects candidate strings (e.g. 10-digit sequences for KRS)
2. **Checksum validation** — validates the candidate against the official algorithm
3. **Placeholder replacement** — valid matches are replaced with `{{PLACEHOLDER}}`; invalid ones are left untouched
4. **Optional anonymizer** — if an `anonymizer_fn` is provided, it's called with `(original, tag_type)` and its result (
   wrapped in `{}`) is used instead

### Adding rules to the pipeline

To enable Polish rules in your masking pipeline, add their plugin IDs to `MASKING_STRATEGY_PIPELINE`:

```python
MASKING_STRATEGY_PIPELINE = ["fast_masker"]
```

All Polish rules are included in the FastMasker plugin by default. For a full list of available rules, see
the [fast_masker README](llm_router_plugins/maskers/fast_masker/README.md).

---

## 2.7 Semantic Routing (Model Selection)

Four routing plugins are available for model selection. The two semantic plugins
(`simple_semantic_routing`, `semantic_biencoder_routing`) activate when `payload["model"] == "auto"`; the Codex plugin
(`agentic_routing_codex`) activates on `"auto_codex"`, routing the OpenAI-Responses-style requests emitted by the Codex
CLI agent by request class and work mode. The Claude Code plugin (`agentic_routing_claude_code`) decides by model name
alone, mapping the model IDs Claude Code asks for onto the models this router serves.

### 2.7.1 Simple Semantic Routing (Heuristic)

The **Simple Semantic Routing plugin** (`simple_semantic_routing`, `utils/routing/simple_semantic/`) activates on
`payload["model"] == "auto"` and rewrites it in three dependency-free stages: **intent** scoring over the last user
message — weighted keywords (case-insensitive substrings), phrases with a `:weight` suffix and regex patterns (+3.0
each) — giving `code` / `math` / `creative` / `general`, or `none` when nothing scores; **complexity** from a word-count
estimate (`int(words * 1.25)`) against the `simple` / `medium` thresholds; and **model selection** from an ordered pool
where index 0 is the cheapest and index `n-1` the strongest. `intent_adjustment` only ever escalates the intents that
declare one; an intent with an empty adjustment is demoted a single pool step. Only `payload["model"]` is written, so
the decision lives in the log:
`SimpleSemanticRouting: intent=code, complexity=simple (10 tokens) -> qwen3.6:35b`.

Configuration ships in
[simple_semantic.json](llm_router_plugins/resources/routing/simple_semantic.json). Overrides:
`LLM_ROUTER_ROUTING_COMPLEXITY_THRESHOLDS` (`simple|medium`), `LLM_ROUTER_ROUTING_MODELS` (the pool),
`LLM_ROUTER_ROUTING_DEFAULT_MODEL` and `LLM_ROUTER_ROUTING_INTENT_<name>` — that last one **replaces** an intent and
clears its patterns and weights, and there is no `..._CONFIG` variable for this plugin.

**Full documentation** — stage-by-stage mechanics, measured intent × complexity → model matrices for two- and
three-model pools, configuration reference, tuning, gotchas and a runnable verification snippet:
[Simple Semantic Routing README](llm_router_plugins/utils/routing/simple_semantic/README.md).

### 2.7.2 Bi-Encoder Semantic Routing (Model Selection)

The **Bi-Encoder routing plugin** (`semantic_biencoder_routing`, `utils/routing/semantic_biencoder/`) also answers
`payload["model"] == "auto"`, but decides by **embedding similarity**. Each routing target is indexed at startup as
`"Target: {name}. {description}"` plus its `examples`, split into overlapping token chunks, embedded by a
sentence-transformers BiEncoder (`google/embeddinggemma-300m` by default), L2-normalized and stored in a
`faiss.IndexFlatIP`; a docstore maps doc IDs back to target names. At request time the last user message is embedded
(text longer than the model's `max_seq_length` is windowed and averaged, capped at `MAX_QUERY_WINDOWS = 4`), the
`top_k` hits are **averaged per target**, and the winning target's `model_name` is selected when its cosine reaches
`similarity_threshold`; the payload gains `routing.target_name` and `routing.similarity`. The index is persisted as
`index.faiss` + `docstore.pkl` and reloaded on the next start, rebuilt automatically when the embedding model outputs a
different dimension. Needs the `[ml]` / `[ml-gpu]` extras — and it fails hard rather than passing traffic through.

Configuration ships in [semantic_biencoder.json](llm_router_plugins/resources/routing/semantic_biencoder.json).
Overrides: `LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_CONFIG` (inline JSON *or* a path), `_MODEL`, `_TARGETS`,
`_CHUNK_SIZE`, `_CHUNK_OVERLAP`, `_PERSIST_DIR`; `similarity_threshold` and `top_k` are config-only. The shipped
`similarity_threshold: 0.0` reroutes **every** request — measured on the shipped targets, real matches score 0.70–0.82
while unrelated or nonsense text still scores ≈0.35–0.37, so `0.5`–`0.6` is the usual production band.

**Full documentation** — startup and query mechanics, measured similarities per target, configuration reference, failure
modes, tuning, gotchas (persisted-index staleness included) and a verification recipe:
[Bi-Encoder Routing README](llm_router_plugins/utils/routing/semantic_biencoder/README.md).

### 2.7.3 Codex CLI routing (`auto_codex`)

The **Codex Routing plugin** (`agentic_routing_codex`, `utils/routing/agentic_routing/codex/`) routes the
OpenAI-Responses-style requests emitted by the **Codex CLI** coding agent. It activates only when `payload["model"]`
equals its configured trigger — `"auto_codex"` by default — so it never intercepts the `"auto"` traffic owned by the
two semantic plugins. It rewrites `payload["model"]` and adds `payload["agent_mode"]` plus a `payload["routing"]`
block, and nothing else; routing **fails open**, so a payload it cannot classify is passed through untouched.

A work mode is resolved by a deterministic cascade — an explicit `agent_mode` override, the request class
(`compaction`, `aux_title`), the `<collaboration_mode>` Plan block injected by the CLI, PL+EN keyword scoring, then an
optional embedding cosine-similarity lookup — and falls back to `implement`:

| Mode         | Model                     | Routed by                                      |
| ------------ | ------------------------- | ---------------------------------------------- |
| `plan`       | `qwen/Qwen3.8-Flash-Next` | `<collaboration_mode>` Plan Mode block         |
| `implement`  | `qwen/Qwen3.8-Flash-Next` | Fallback for a plain main turn                 |
| `test`       | `qwen/Qwen3.8-27B`        | Keywords in the classified user text           |
| `git_review` | `qwen/Qwen3.8-27B`        | Keywords in the classified user text           |
| `review`     | `qwen/Qwen3.8-Flash-Next` | Keywords in the classified user text           |
| `debug`      | `qwen/Qwen3.8-Flash-Next` | Keywords in the classified user text           |
| `aux_title`  | `qwen/Qwen3.8-27B`        | Request class — system-thread title generation |
| `compaction` | `qwen/Qwen3.8-27B`        | Request class — context compaction             |

Modes ship in
[agentic_routing_codex.json](llm_router_plugins/resources/routing/agentic_routing_codex.json) and are overridable with
`LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_*` environment variables (`…_CONFIG`, `…_TRIGGER`, `…_MODEL_<MODE>`,
`…_MODELS`, `…_MODES`, `…_FALLBACK_MODE`, `…_HEURISTIC_ENABLED`, `…_HEURISTIC_MIN_SCORE`, `…_CLASSIFY_MAX_CHARS`,
`…_MODEL`, `…_SEMANTIC_ENABLED`, `…_SIMILARITY_THRESHOLD`, `…_TOP_K`, `…_CHUNK_SIZE`, `…_CHUNK_OVERLAP`,
`…_PERSIST_DIR`, `…_MODE_<name>_KEYWORDS`). `MODES` filters the mode list but does not move the fallback: if the
whitelist excludes the configured `fallback_mode` (`implement`), set `FALLBACK_MODE` as well — otherwise configuration
validation fails at startup.

**Full documentation** — the cascade and scoring model, installation, loading the plugin into the router, model
selection and its pitfalls, tuning, a verification recipe and a troubleshooting table — lives in the
[Codex CLI Routing README](llm_router_plugins/utils/routing/agentic_routing/codex/README.md).

### 2.7.4 Claude Code Model Swap (`claude-*` → configured models)

The **Claude Code Model Swap plugin** (`agentic_routing_claude_code`, `utils/routing/agentic_routing/claude_code/`)
rewrites the model Claude Code asks for into the model the operator serves. Claude Code resolves its `fable` / `opus` /
`sonnet` / `haiku` aliases client-side and sends a versioned ID, so pointing it at your own backend means overriding one
environment variable per tier (`ANTHROPIC_DEFAULT_FABLE_MODEL`, `ANTHROPIC_DEFAULT_OPUS_MODEL`,
`ANTHROPIC_DEFAULT_SONNET_MODEL`, `ANTHROPIC_DEFAULT_HAIKU_MODEL`, `CLAUDE_CODE_SUBAGENT_MODEL`) on every host. This
plugin moves that mapping into one config file: a **tier** names the model IDs it answers and the model to serve them
with, and a matching request gets the target written into `payload["model"]` (and `payload["model_name"]`, when the
payload carries it) plus a `payload["routing"]` block.

| Tier     | Model IDs Claude Code sends                        | Setting it replaces             |
| -------- | -------------------------------------------------- | ------------------------------- |
| `fable`  | `claude-fable-5-1`, `claude-fable-5`, `best`       | `ANTHROPIC_DEFAULT_FABLE_MODEL` |
| `opus`   | `claude-opus-5-5` … `claude-opus-4-6`              | `ANTHROPIC_DEFAULT_OPUS_MODEL`  |
| `plan`   | `opusplan`                                         | the `model: "opusplan"` phase   |
| `sonnet` | `claude-sonnet-5`, `claude-sonnet-4-6`, `-4-5`     | `ANTHROPIC_DEFAULT_SONNET_MODEL`|
| `haiku`  | `claude-haiku-4-5`, `claude-3-5-haiku-*`           | `ANTHROPIC_DEFAULT_HAIKU_MODEL` |

Matching is content-free and deterministic: the requested name is normalized (window suffix `[1m]`, provider version
`:0`, `@YYYYMMDD` revision, `us.anthropic.` / `anthropic/` prefix, `models/` path and `-YYYYMMDD` release date stripped)
and resolved through four layers, most specific first — literal, normalized exact, longest wildcard prefix
(`claude-opus-*`), then family token (`claude-3-5-haiku-20241022` → Haiku). Configuration ships in
[agentic_routing_claude_code.json](llm_router_plugins/resources/routing/agentic_routing_claude_code.json) and is
overridable with `LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CLAUDE_CODE_*` (`…_CONFIG`, `…_ENABLED`, `…_MATCH_FAMILIES`,
`…_FIELDS`, `…_MODEL_<TIER>`, `…_MODELS`, `…_MODES`, `…_MODE_<tier>_MODELS`). Routing **fails open and silent**: a model
no tier claims comes back as the same object it arrived as, with no log line, while an inconsistent config (duplicate tier
names, an embedded or bare wildcard, one exact name claimed twice) raises at startup. No ML dependencies.

**Full documentation** — the normalization and match layers, installation, router and CLI wiring, tier configuration,
linting, gotchas, a verification recipe and a troubleshooting table — lives in the
[Claude Code Model Swap README](llm_router_plugins/utils/routing/agentic_routing/claude_code/README.md).

---

## 3. Pipelines

Both masker and guardrail pipelines share the same design pattern:

| Class                                                     | Purpose                                                                            |
|-----------------------------------------------------------|------------------------------------------------------------------------------------|
| **MaskerPipeline** (`pipeline.py` – masker version)       | Executes a list of masker plugins in order, transforming the payload step‑by‑step. |
| **GuardrailPipeline** (`pipeline.py` – guardrail version) | Executes guardrail plugins sequentially, stopping on the first failure.            |

### 3.1 Registration

* Plugins are registered lazily via `MaskerRegistry.register(name, logger)` or
  `GuardrailRegistry.register(name, logger)`.
* The registry maps a string identifier (e.g. `"fast_masker"`) to a concrete plugin class, allowing pipelines to resolve
  the classes at runtime.

### 3.2 Configuration

All plugin identifiers are stored in environment variables or constants such as:

```python
MASKING_STRATEGY_PIPELINE = ["fast_masker"]
GUARDRAIL_STRATEGY_PIPELINE_REQUEST = ["nask_guard", "sojka_guard"]
```

These lists are consumed by the endpoint initialization (`EndpointI._prepare_masker_pipeline`,
`EndpointI._prepare_guardrails_pipeline`).

---

## 4. Adding a New Plugin

1. **Create a subclass** of either `PluginInterface` (for maskers) or `HttpPluginInterface` / a custom guardrail base.
2. **Define a `name` class attribute** – this is the identifier used in pipeline configuration.
3. **Implement `apply(self, payload: Dict) -> Dict`** (masker) **or `apply(self, payload: Dict) -> Tuple[bool, Dict]`
   ** (guardrail).
4. **Register the plugin** – either automatically via the registry’s `register` call in the pipeline constructor, or
   manually by calling `MaskerRegistry.register(name=MyPlugin.name, logger=logger)`.

*Example stub for a new masker:*

```python
# my_custom_masker.py
from llm_router_plugins.maskers.plugin_interface import PluginInterface
import logging
from typing import Dict, Optional


class MyCustomMasker(PluginInterface):
    name = "my_custom_masker"

    def __init__(self, logger: Optional[logging.Logger] = None):
        super().__init__(logger=logger)
        # Load any heavy resources here (e.g., a spaCy model)

    def apply(self, payload: Dict) -> Dict:
        # Perform your masking logic and return the modified payload
        return payload
```

After placing the file in `llm_router_plugins/maskers/plugins/`, enable it by adding `"my_custom_masker"` to
`MASKING_STRATEGY_PIPELINE`.

---

## 5. Retrieval‑Augmented Generation (RAG) Support

The project now includes a **LangChain‑based RAG plugin** that enables semantic search over user‑provided documents. The
implementation lives in `llm_router_plugins/utils/rag/langchain_plugin.py` and is driven by the helper CLI scripts
located in `scripts/`.

### 5.1 What the plugin does

| Feature           | Description                                                                                                                                                                                                                             |
|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Indexing**      | Reads a directory of text‑like files (`.txt`, `.md`, `.html`, `.js`, …), splits them into token‑based windows, embeds each chunk with a configurable transformer model, and stores the vectors in a FAISS (or compatible) vector store. |
| **Searching**     | Given a user query, retrieves the most similar chunks and injects them into the payload (e.g., appends to the last user message) so that downstream LLM calls can use the retrieved context.                                            |
| **Configuration** | All parameters (collection name, embedder model, device, chunk size, overlap, persistence directory) are driven by environment variables prefixed with `LLM_ROUTER_`. See the table below for the full list.                            |
| **CLI helpers**   | Two ready‑to‑use scripts: `scripts/llm-router-rag-langchain-index.sh` (indexes a repository) and `scripts/llm-router-rag-langchain-search.sh` (runs a search or starts an interactive REPL).                                            |

### 5.2 Environment variables

| Variable                                 | Default                                                                        | Meaning                                                     |
|------------------------------------------|--------------------------------------------------------------------------------|-------------------------------------------------------------|
| `LLM_ROUTER_LANGCHAIN_RAG_COLLECTION`    | *must be set*                                                                  | Name of the FAISS collection (e.g. `sample_collection`).    |
| `LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER`      | `/mnt/data2/llms/models/community/google/embeddinggemma-300m`                  | Path or Hugging‑Face identifier of the embedding model.     |
| `LLM_ROUTER_LANGCHAIN_RAG_DEVICE`        | `cuda:2`                                                                       | Torch device (`cpu`, `cuda:0`, …).                          |
| `LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE`    | `1024`                                                                         | Number of tokens per chunk.                                 |
| `LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP` | `100`                                                                          | Number of overlapping tokens between consecutive chunks.    |
| `LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR`   | `./workdir/plugins/utils/rag/langchain/${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION}` | Directory where the FAISS index and docstore are persisted. |

#### Example export block (add to your shell profile or a `.env` file)

```shell script
export LLM_ROUTER_LANGCHAIN_RAG_COLLECTION="${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION:-sample_collection}"
export LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER="${LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER:-/mnt/data2/llms/models/community/google/embeddinggemma-300m}"
export LLM_ROUTER_LANGCHAIN_RAG_DEVICE="${LLM_ROUTER_LANGCHAIN_RAG_DEVICE:-cuda:2}"
export LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE="${LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE:-1024}"
export LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP="${LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP:-100}"
export LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR="${LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR:-./workdir/plugins/utils/rag/langchain/${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION}}"
```

### 5.3 Using the CLI scripts

**Index a repository** (example for the documentation site):

```shell script
scripts/llm-router-rag-langchain-index.sh
# Internally runs:
# llm-router-rag-langchain index --path "../.github/pages/llmrouter.cloud/" --ext .html .js .md
```

**Search** (interactive REPL):

```shell script
scripts/llm-router-rag-langchain-search.sh
# Internally runs:
# llm-router-rag-langchain search
# (you will be prompted for a query, type “exit” to quit)
```

**One‑shot search**:

```shell script
llm-router-rag-langchain search --query "What is Retrieval‑Augmented Generation?" --top_n 5
```

The CLI returns the raw matching chunks together with similarity scores. The `LangchainRAGPlugin` automatically formats
the retrieved text and appends it to the user’s last message, prefixed with:

```
If the context below will help answer the above question, use it.
Context separated with double enter
```
