Metadata-Version: 2.4
Name: Pythonaibrain-NLP
Version: 0.2.0
Summary: Advanced neural (non-transformer, non-rule-based) NLU + NLG: joint intent/slot understanding and SC-LSTM generation, both driven by nlg_intents.json, with RAG retrieval and a dialogue Context Manager. Part of the Pythonaibrain ecosystem.
Author: Divyanshu Sinha
License: MIT
Keywords: nlg,nlp,sc-lstm,rag,dialogue-systems,pythonaibrain
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: torch>=2.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"

# Pythonaibrain-NLP

Advanced neural language **understanding and generation** — **not
rule-based, not transformer-based** — both driven by `nlg_intents.json`,
grounded with Retrieval-Augmented Generation (RAG) on the generation
side, and aware of multi-turn dialogue context. NLG turns (intent,
slots) into text with a Semantically-Controlled LSTM; NLU turns text
back into (intent, slots) with a joint BiLSTM intent-classifier/
slot-tagger. Each half works fully standalone, or chained together
via `NLUNLGPipeline`. Part of the
[Pythonaibrain](https://pypi.org/project/pyaitk/) offline AI toolkit
family.

```
pip install -e .
python -m pythonaibrain_nlp demo
```

## Why this exists

Most "NLG" in the wild is one of two things:
- **Rule-based / templated**: `f"Sure, I can book a flight from {origin} to {destination}."`
  — brittle, doesn't scale past a handful of phrasings, no learning.
- **Transformer-based**: fine-tuned GPT-style models — powerful, but
  heavy, hard to run fully offline/from-scratch, and overkill for a
  closed-domain, slot-driven generation task.

This package is neither. It's a **Semantically-Controlled LSTM
(SC-LSTM)** — a genuinely learned, recurrent neural decoder purpose-built
for exactly this problem (intent + slots -> fluent text), implemented
as a real `torch.nn.Module` and trained with `torch.autograd` +
`torch.optim.Adam`. No attention stack, no pretrained weights, no
templates — real gradients through a real (if unfashionable, by
2026 standards) recurrent architecture.

## Architecture

```
nlg_intents.json                  raw text corpus
       │                                │
       ▼                                ▼
 IntentSet (schema.py)          RAGRetriever (retriever.py)
       │                                │
       │            ┌───────────────────┘
       ▼            ▼
 DialogueActVectorizer      ContextManager (context_manager.py)
   (neural/model.py)          - turn history
       │                      - persistent slot memory
       │  d0 (dialogue-act        │
       │   vector)                │ context vector
       ▼                          ▼
            SCLSTMCell (neural/sclstm.py)
     token embedding + context vector -> hidden state
     dialogue-act vector decays via a learned "reading gate"
     as each word is generated
                    │
                    ▼
          output projection -> softmax over vocabulary
                    │
                    ▼
        postprocess.finalize() (relexicalize + cleanup)
                    │
                    ▼
              generated text
```

### Module map

| Module | Responsibility |
|---|---|
| `schema.py` | Typed data model for `nlg_intents.json` (`Intent`, `Slot`, `Example`, `RetrievalConfig`) |
| `intents_loader.py` | Load/validate/save `nlg_intents.json` |
| `intents_builder.py` | **Build** `nlg_intents.json` from text + RAG + ContextManager (see below) |
| `embeddings.py` | Dependency-free TF-IDF + co-occurrence (SVD) embeddings — no transformer |
| `retriever.py` | RAG: chunk + TF-IDF index + cosine retrieval, function mapping, save/load |
| `external_sources.py` | Load RAG sources from files, directories, or URLs |
| `functions.py` | Sandboxed example functions (create/list/read/delete a file) to register via `map_function` |
| `context_manager.py` | Turn history, persistent slot memory, fixed-size context vector |
| `tokenizer.py` | Word-level vocab, delexicalization/relexicalization |
| `neural/sclstm.py` | The SC-LSTM cell — a `torch.nn.Module` (forward only; autograd handles backward) |
| `neural/model.py` | `NLGModel`: embedding + context + SC-LSTM + output projection, `torch.nn.Module`; `DialogueActVectorizer`; `.save()`/`.load()` checkpointing |
| `neural/trainer.py` | Teacher-forced training loop with `torch.optim.Adam` + gradient clipping |
| `generator.py` | `NLGGenerator` — the public, one-call generation API |
| `nlu/tagging.py` | BIO tag alignment/extraction, `IntentVocabulary`, `TagVocabulary` |
| `nlu/dataset.py` | Builds NLU training data from the same `nlg_intents.json` examples |
| `nlu/model.py` | `NLUModel` — joint BiLSTM intent-classifier + slot-tagger |
| `nlu/trainer.py` | `NLUTrainer` — batched joint training (same large-corpus machinery as NLG) |
| `nlu/parser.py` | `NLUParser` — the public, standalone understanding API |
| `pipeline.py` | `NLUNLGPipeline` — chains NLU + NLG, dispatches mapped functions, RAG fallback |
| `postprocess.py` | Relexicalization + text cleanup |

## `nlg_intents.json`: what it is and why it's shaped this way

Each **intent** bundles everything the model needs to condition one
kind of response:

```json
{
  "id": "book_flight",
  "description": "user wants to book a flight",
  "dialogue_act": "inform",
  "slots": [
    {"name": "origin", "type": "string", "required": true},
    {"name": "destination", "type": "string", "required": true},
    {"name": "date", "type": "string", "required": false}
  ],
  "context_requirements": ["previous_intent"],
  "retrieval": {"enabled": true, "corpus_ref": "policy", "top_k": 3},
  "examples": [
    {
      "slots": {"origin": "Delhi", "destination": "Mumbai", "date": "tomorrow"},
      "context": {"previous_intent": null},
      "reference_text": [
        "Sure, I can book a flight from Delhi to Mumbai for tomorrow.",
        "Booking your flight from Delhi to Mumbai for tomorrow now.",
        "Got it — a flight from Delhi to Mumbai for tomorrow is on its way."
      ],
      "pattern": [
        "book me a flight from Delhi to Mumbai for tomorrow",
        "I want to fly from Delhi to Mumbai for tomorrow"
      ]
    }
  ]
}
```

- **`slots`** become the dialogue-act vector `d0` fed into the SC-LSTM
  (via `DialogueActVectorizer`) — this is what the reading gate decays
  as it "uses up" each slot during generation.
- **`retrieval`** points at a named corpus (`corpus_ref`) so the
  `RAGRetriever` knows which text source to search for this intent.
- **`context_requirements`** documents which context keys this intent's
  examples actually depend on (populated for real by `ContextManager`,
  not hand-guessed).
- **`reference_text`** is a *list*, not a single string — several valid
  ways NLG could phrase this same (intent, slots). Training on all of
  them (see `neural/dataset.py`) is what lets generation vary its
  output instead of memorizing one fixed template per slot combination;
  pair it with `NLGGenerator.generate(..., temperature=0.7)` (see
  below) to actually sample a different phrasing across calls, not
  just train on several. Slot values are delexicalized to placeholders
  (`__destination__`) before tokenization so the network learns
  structure, not memorized entities. A single string is still accepted
  and auto-wrapped into a one-item list, for files written before this
  field became a list.
- **`pattern`** is the NLU-facing counterpart to `reference_text` —
  example *user* utterances for this same intent+slots ("book me a
  flight from Delhi to Mumbai", not "Sure, I can book a flight..."),
  written the way someone would actually ask rather than the way the
  system would answer. NLU trains on both `reference_text` and
  `pattern` (see `nlu/dataset.py`); `pattern` is what actually teaches
  it to recognize real commands and questions instead of only
  NLG-style confirmations. Optional — an intent set without it still
  trains an NLU, just narrower (see honest limitations below).

### How to build `nlg_intents.json`: text + RAG + Context Manager

Hand-writing this file for a real domain doesn't scale. Use
`IntentsBuilder`, which is exactly "some text + RAG + Context Manager"
turned into an API:

```python
from pythonaibrain_nlp import IntentsBuilder, save_intents

builder = IntentsBuilder(domain="travel_support")

# 1. "Some text" — raw reference docs become your RAG corpus
builder.add_source_text("policy", open("flight_policy.txt").read())

# 2. Declare an intent + its slots, pointing at that corpus
builder.define_intent(
    "book_flight", "user wants to book a flight", "inform",
    slots=[("origin", "string", True), ("destination", "string", True), ("date", "string", False)],
    corpus_ref="policy",
)

# 3. Add examples in conversational order — a real ContextManager runs
#    underneath and records actual dialogue state into each example's
#    "context" block (not a hand-typed guess). reference_text accepts
#    a single string or a list of phrasing variants; pattern is optional
#    genuine user-query phrasing for the same slots.
builder.add_example(
    "book_flight",
    {"origin": "Delhi", "destination": "Mumbai", "date": "tomorrow"},
    reference_text=[
        "Sure, I can book a flight from Delhi to Mumbai for tomorrow.",
        "Booking your flight from Delhi to Mumbai for tomorrow now.",
    ],
    pattern=["book me a flight from Delhi to Mumbai for tomorrow"],
)

intent_set = builder.build()          # validated IntentSet
save_intents(intent_set, "nlg_intents.json")
```

See `examples/build_intents_demo.py` for a runnable version and
`examples/nlg_intents.json` for the file it produces.

### External data sources

RAG sources don't have to be inline Python strings — `external_sources.py`
loads them from files, directories, or URLs, feeding into the same
`{source_name: text}` shape `RAGRetriever.index()` and
`IntentsBuilder.add_source_text()` already expect:

```python
from pythonaibrain_nlp import load_text_file, load_directory, load_url

text = load_text_file("policies/refunds.txt")
sources = load_directory("policies/")          # one source per .txt/.md/.rst file, named by filename stem
web_text = load_url("https://example.com/faq")  # HTML gets tag-stripped automatically
```

Or via `IntentsBuilder` directly:

```python
builder.add_source_file("policies/refunds.txt")
builder.add_source_directory("policies/")
builder.add_source_url("https://example.com/faq")
```

`RAGRetriever` also supports incremental indexing and persistence, so
a large external corpus doesn't need re-chunking on every startup:

```python
retriever.add_document("new_policy", text)     # add one document to an existing index
retriever.save("index.json")                    # persist raw source texts + settings
retriever = RAGRetriever.load("index.json")     # re-derives the TF-IDF index, no re-chunking needed
```

Stays dependency-free on purpose — stdlib `urllib` only, no
`requests`/`beautifulsoup4` — consistent with the rest of the
package. HTML gets a lightweight regex tag-strip, not a full parser;
fine for policy pages and docs, not a general-purpose scraper.

### Ready-made large corpus

`examples/build_large_corpus.py` builds a bigger, realistic example:
a `customer_support` domain with **17 intents** — 12 transactional
(`book_flight`, `cancel_flight`, `refund_status`, `track_order`,
`reset_password`, `update_address`, `business_hours`,
`file_complaint`, `cancel_subscription`, `payment_failed`,
`contact_support`, `check_balance`), each backed by its own RAG
policy document, plus **5 general-purpose / chit-chat intents**
(`greeting`, `farewell`, `thanks`, `smalltalk`, `help`) with no RAG
grounding — there's nothing factual to retrieve for "hello" or
"thanks". **200 examples each (3,400 total)**. Every example carries
several `reference_text` phrasings *and* several `pattern` query
phrasings — genuine lexical diversity on both the NLG-output side and
the NLU-input side, not one template with values swapped in.

```
python examples/build_large_corpus.py --n-per-intent 200 --out nlg_intents.json
```

The output ships pre-built at `examples/nlg_intents_large.json` (17
intents, 3,400 examples, expanding to 12,440 `reference_text`
variants and 12,912 `pattern` variants, ~2.3 MB) so you can start
training against a real-sized corpus immediately:

```python
from pythonaibrain_nlp import load_intents, NLGGenerator, ContextManager
from pythonaibrain_nlp.retriever import RAGRetriever
from pythonaibrain_nlp.neural.model import DialogueActVectorizer, NLGModel
from pythonaibrain_nlp.neural.trainer import Trainer
from pythonaibrain_nlp.tokenizer import Vocabulary, delexicalize
from examples.build_large_corpus import SOURCES  # the same RAG policy docs used to build it

intent_set = load_intents("examples/nlg_intents_large.json")
retriever = RAGRetriever()
retriever.index(SOURCES)

# every reference_text variant becomes its own training target — see neural/dataset.py
texts = [delexicalize(t, ex.slots) for i in intent_set.intents for ex in i.examples for t in ex.reference_text]
vocab = Vocabulary().build(texts)             # ~221 words, 12,440 training texts
da_vectorizer = DialogueActVectorizer().fit(intent_set)  # 31-dim dialogue-act space (17 unique acts + slot bits)

model = NLGModel(vocab, da_vectorizer, embed_dim=64, hidden_dim=128, context_dim=64)
model.sentence_embedder.fit(texts)

Trainer(model, lr=0.005).fit(intent_set, retriever, epochs=25, batch_size=128)
```

**Verified, not just schema-validated**: this was actually trained
end to end — 12,440 training texts, 98 batches/epoch, ~3s/epoch on
CPU. 25 epochs (loss plateaus ~0.156) generates correctly and
slot-faithfully across all 17 intents, no confusion between the new
general-purpose intents and the 12 transactional ones:

```
greeting      : Nice to meet u Divyanshu.
smalltalk     : I'm good thank you what can i do for you.
help          : I can book flights check refunds track orders reset passwords and handle account questions.
book_flight   : Booking your flight from Delhi to Mumbai for tomorrow now.
refund_status : I can confirm the refund for order ORD90210 is in progress.
cancel_flight : Booking BK55221 is now cancelled.
farewell      : Goodbye take care.
```

The NLU side was trained the same way, on both `reference_text` and
`pattern` (25,352 combined training texts, 10 epochs, ~90s on CPU —
NLU converges much faster than NLG since classification/tagging is an
easier task than token-by-token generation) and tested against
genuine **query-style** phrasing it only ever saw via `pattern`, not
`reference_text`:

```
"Hi I'm Divyanshu!"                                   -> greeting, {'name': 'Divyanshu'}
"how are you"                                          -> smalltalk, {}
"what can you do"                                       -> help, {}
"book me a flight from Delhi to Mumbai for tomorrow"    -> book_flight, {'origin': 'Delhi', 'destination': 'Mumbai', 'date': 'tomorrow'}
"where is my refund for order ORD90210"                 -> refund_status, {'order_id': 'ORD90210'}
"please cancel booking BK55221"                         -> cancel_flight, {'booking_id': 'BK55221'}
"thanks a lot"                                           -> thanks, {}
"bye"                                                    -> farewell, {}
```

All correct — no confusion between the 5 new general-purpose intents
and the 12 transactional ones, and this is the direct, measured payoff
of `pattern` existing at all: without it, NLU only ever sees
`reference_text`'s system-confirmation style, and generalizes
noticeably worse to real user commands (see honest limitations below
for what that gap looked like before `pattern` was added).

Two honest, known rough edges visible in that output, both from the
word-level tokenizer rather than the SC-LSTM itself: punctuation like
em dashes gets dropped (tokenizer only keeps alphanumerics/apostrophes/
underscores), and `postprocess.finalize()` only capitalizes the first
letter of the whole sentence, not a mid-sentence "I". Neither affects
slot correctness or intent discrimination — both are straightforward
tokenizer-level fixes if you need cleaner surface punctuation.

## Training and generating

### Small corpus (quick iteration)

```python
from pythonaibrain_nlp import (
    ContextManager, NLGGenerator, load_intents,
)
from pythonaibrain_nlp.neural.model import DialogueActVectorizer, NLGModel
from pythonaibrain_nlp.neural.trainer import Trainer
from pythonaibrain_nlp.tokenizer import Vocabulary, delexicalize
from pythonaibrain_nlp.intents_builder import IntentsBuilder

builder = IntentsBuilder(domain="travel_support")
# ... add_source_text / define_intent / add_example calls ...
intent_set = builder.build()
retriever = builder.retriever

texts = [delexicalize(t, ex.slots)
         for i in intent_set.intents for ex in i.examples for t in ex.reference_text]
vocab = Vocabulary().build(texts)
da_vectorizer = DialogueActVectorizer().fit(intent_set)

model = NLGModel(vocab, da_vectorizer, embed_dim=64, hidden_dim=128, context_dim=64)
model.sentence_embedder.fit(texts)

Trainer(model, lr=0.02).train_intent_set(intent_set, retriever, epochs=200)

gen = NLGGenerator(intent_set, retriever, model, ContextManager())
print(gen.generate("book_flight", {"origin": "Delhi", "destination": "Mumbai"}))

# save / reload the trained model (torch checkpoint + vocab + DA-vectorizer + fitted embedder)
model.save("nlg_model.pt")
from pythonaibrain_nlp.neural.model import NLGModel
reloaded = NLGModel.load("nlg_model.pt")
```

Runs on GPU for free — pass `device="cuda"` to `NLGModel(...)` if one's available; everything else (RAG retrieval, context vectors, DA-vectorizer) stays CPU/numpy since those aren't backprop targets.

### Large corpus (thousands–millions of examples)

Use `Trainer.fit()` directly instead of the `train_intent_set` alias —
it exposes batching, gradient accumulation, mixed precision, and
checkpoint/resume:

```python
trainer = Trainer(model, lr=0.01)
history = trainer.fit(
    intent_set, retriever,
    epochs=20,
    batch_size=128,          # real minibatches through the SC-LSTM, not one example at a time
    grad_accum_steps=4,      # effective batch size 512 without needing 512-worth of GPU memory
    num_workers=4,           # parallel batch collation
    checkpoint_path="nlg_checkpoint.pt",
    checkpoint_every=1,      # save after every epoch — long runs can be interrupted safely
    resume=True,             # picks back up from checkpoint_path if it already exists
)
```

What makes this different from the small-corpus path, and why it
matters once the corpus stops being tiny:

- **Retrieval and context vectors are computed once, not every epoch.**
  `Trainer.fit()` calls `build_dataset()` internally, which walks the
  intent set through a real `ContextManager` exactly one time and
  caches (dialogue-act vector, context vector, target token ids) per
  example. The original per-example trainer re-ran RAG retrieval and
  context-vector construction on *every* epoch — for a large corpus
  trained over many epochs that dominates total training time. Now it
  doesn't happen again after the first pass.
- **Real minibatches.** `SCLSTMCell` and `NLGModel.forward_sequence_batch`
  operate on `(B, ...)` tensors — the whole point of a GPU.
- **Sparse, not dense, retrieval indexing.** `RAGRetriever` /
  `TfidfVectorizer` build `scipy.sparse` matrices, not a dense
  `(n_chunks x vocab)` numpy array — a corpus with tens of thousands
  of chunks and a large vocabulary would otherwise not fit in memory.
  `max_vocab` caps pathological vocabulary growth (typos, IDs, noise
  tokens) to the most frequent terms.
- **Truncated SVD, not full dense SVD, for context embeddings.**
  `CooccurrenceEmbedder` uses `scipy.sparse.linalg.svds` over a sparse
  PPMI matrix once the vocabulary passes a few hundred words — the
  naive dense `numpy.linalg.svd` over a `(V, V)` matrix is `O(V³)`
  time and `O(V²)` memory and simply doesn't run on a real large
  vocabulary. Small vocabularies still get a fast dense fallback,
  since ARPACK's setup cost isn't worth paying on a toy corpus.
- **Mixed precision + gradient accumulation.** `torch.autocast` +
  `GradScaler` activate automatically when `model.device.type == "cuda"`
  (no-op on CPU); `grad_accum_steps` lets you reach a large effective
  batch size on limited GPU memory.
- **Checkpoint/resume.** Long training runs on a large corpus can get
  interrupted — `checkpoint_path` + `resume=True` saves model +
  optimizer state and picks back up rather than restarting from
  scratch.

`train_intent_set(...)` is kept around as a thin backward-compatible
alias (`fit(..., batch_size=8)`) for existing small scripts and tests.

## NLU: understanding text, not just generating it

`nlg_intents.json` also supervises the reverse direction. Each
example's `(reference_text, slots)` pair is simultaneously "generate
this text from these slots" (NLG) and "recover these slots from this
text" (NLU) — same facts, opposite direction, so the same intents
file trains both sides. No separate annotation schema.

**Architecture** — a joint BiLSTM intent-classifier + slot-tagger
(`nlu/model.py`), matching the rest of the package's rules: neural end
to end (no keyword rules, no regex intent matching), not a transformer
(a bidirectional `torch.nn.LSTM`, no self-attention). One shared
encoder feeds two heads: an intent classifier over the concatenated
final forward/backward hidden states, and a per-token BIO slot tagger
(`B-origin`, `I-origin`, `O`, ...) over every timestep's output. Both
train jointly with a combined loss.

### Standalone (no NLG involved at all)

```python
from pythonaibrain_nlp import load_intents, NLUParser
from pythonaibrain_nlp.nlu.model import NLUModel
from pythonaibrain_nlp.nlu.tagging import IntentVocabulary, TagVocabulary
from pythonaibrain_nlp.nlu.trainer import NLUTrainer
from pythonaibrain_nlp.tokenizer import Vocabulary

intent_set = load_intents("nlg_intents.json")
# train the NLU vocab on both reference_text and pattern text — pattern is
# what actually teaches recognition of real query phrasing, not just system text
texts = [
    t for i in intent_set.intents for ex in i.examples for t in (*ex.reference_text, *ex.pattern)
]

nlu_vocab = Vocabulary().build(texts, min_freq=2)
intent_vocab = IntentVocabulary().fit(intent_set)
tag_vocab = TagVocabulary().fit(intent_set)

nlu_model = NLUModel(nlu_vocab, intent_vocab, tag_vocab, embed_dim=64, hidden_dim=128)
NLUTrainer(nlu_model, lr=0.005).fit(intent_set, epochs=15, batch_size=128)

parser = NLUParser(nlu_model)
result = parser.parse("please cancel booking BK12345")   # genuine query phrasing, from pattern
print(result.intent_id, result.slots, result.confidence)
# cancel_flight  {'booking_id': 'BK12345'}  1.0

nlu_model.save("nlu_model.pt")   # weights + vocab + intent/tag vocabularies
```

`NLUTrainer.fit()` uses the same large-corpus machinery as the NLG
side — batching, `DataLoader`, mixed precision on CUDA, checkpoint/resume
(`checkpoint_path`, `resume=True`). On the bundled 4,800-example corpus
(`examples/nlg_intents_large.json`), 15 epochs (~30s on CPU) is enough
to converge — NLU trains noticeably faster than NLG, since classifying
and tagging is an easier task than generating fluent text token by token.

### Combined: text in, response out, one call

```python
from pythonaibrain_nlp import NLUNLGPipeline

pipeline = NLUNLGPipeline(nlu_parser, nlg_generator)  # both trained independently, as above
result = pipeline.respond("Sure, I can book a flight from Chennai to Pune.")

print(result.intent_id)       # "book_flight"
print(result.slots)           # {"origin": "Chennai", "destination": "Pune"}
print(result.response_text)   # NLG's generated reply
```

`NLUNLGPipeline` is a thin wrapper — `NLUParser.parse()` and
`NLGGenerator.generate()` don't know it exists, and both remain fully
usable on their own. Pass `min_confidence` to reject low-confidence
parses before they reach NLG:

```python
pipeline = NLUNLGPipeline(nlu_parser, nlg_generator, min_confidence=0.6)
```

## Trial AI: a real interactive assistant

`examples/trial_ai.py` wires everything in this README into one
runnable assistant: the bundled 21-intent corpus (12 transactional +
5 general-purpose: `greeting`, `farewell`, `thanks`, `smalltalk`,
`help`, + 4 function-mapped: `create_file`, `list_files`, `read_file`,
`delete_file`), a trained NLU, a trained NLG, RAG grounding,
`ContextManager`, and `NLUNLGPipeline`.

```bash
python examples/trial_ai.py            # interactive chat
python examples/trial_ai.py --demo     # scripted sample conversation, no typing needed
python examples/trial_ai.py --retrain  # ignore cached checkpoints, retrain from scratch
```

Trained checkpoints ship pre-built in `examples/trial_models/` (NLG
856 KB, NLU 1.4 MB), so a normal run loads instantly — `--retrain`
takes a few minutes on CPU.

```
You: Hi I'm Divyanshu!
AI : [greeting] Nice to meet you Divyanshu.

You: book me a flight from Delhi to Mumbai for tomorrow
AI : [book_flight] Booking your flight from Delhi to Mumbai for tomorrow now.

You: create a file test.txt
AI : [create_file, ran a real function] I've created the file test.txt.

You: show list of files
AI : [list_files, ran a real function] I found these files test.txt.

You: how many kilograms of baggage can I bring on an economy flight
AI : [from RAG] Flight changes made more than 24 hours before departure are free of charge. Cancellations within 24 hours of booking are fully refundable. Flights cancelled closer to departure are subject to a cancellation fee based on fare type. Baggage allowance is 20 kilograms for economy class and 30 kilograms for business class.

You: I want to book a flight
AI : Sorry, I couldn't help with that — missing required slots for intent 'book_flight': ['origin', 'destination']

You: thanks a lot
AI : [thanks] You're welcome.
```

`create a file test.txt` and `show list of files` are **real filesystem
operations**, sandboxed to `examples/trial_workspace/` — see "Function
mapping and RAG fallback" below. `how many kilograms of baggage...`
isn't a defined intent at all — it's answered directly from the
indexed flight policy text.

### Two real bugs this surfaced, both fixed

Building an actual multi-turn, multi-intent assistant — not just
single-shot generation calls — surfaced two genuine issues that
single-intent testing never would have:

1. **Context corruption across an intent switch.** `build_dataset()`
   trains NLG by walking through examples grouped by intent (every
   `book_flight` example, then every `cancel_flight` example, ...), so
   the model only ever saw *same-intent* context during training. A
   shared `ContextManager` across a genuine topic switch — book a
   flight, then ask about a refund — kept generating book_flight-style
   text for the second turn, driven by leftover context, even with the
   correct intent+slots passed to `generate()`. Fixed in
   `NLUNLGPipeline`: it now resets the NLG side's conversation whenever
   NLU's parsed intent differs from the previous turn's
   (`auto_reset_on_intent_change=True` by default — see `pipeline.py`
   for how to disable it). Covered by
   `test_pipeline_handles_intent_switch_without_context_corruption`.

2. **A shared `dialogue_act` collapses the DA vector's discriminating
   signal.** The bundled corpus originally gave every intent the same
   `dialogue_act="inform"`, so `DialogueActVectorizer` produced only
   *one* shared "act" bit across all 12 intents — the only real
   intent-discriminating signal came from which slots were filled. For
   an intent with no required slots and an unfilled optional one
   (`business_hours` with no branch mentioned), the DA vector carried
   almost no information, and generation defaulted to whatever pattern
   was best-represented in training rather than the actual intent.
   Fixed by giving each intent a unique `dialogue_act`
   (`inform_book_flight`, `inform_refund_status`, ...) in
   `build_large_corpus.py`, guaranteeing every intent gets its own
   identifying bit in the DA vector regardless of slot fill state.

Both are the kind of bug that only shows up once you actually chain
multiple intents together in one conversation — worth knowing if
you build your own multi-intent corpus: give every intent a distinct
`dialogue_act`, and don't assume a shared `ContextManager` is safe
across an intent switch unless your training data was built to teach
that.

3. **Narrow date-phrasing coverage broke slot extraction on realistic
   input.** `book_flight` training only ever paired dates with the
   preposition "for" ("for next Monday") and only ever used
   Monday/Tuesday as example weekdays — a real user typing "on next
   sunday" (different preposition, unseen weekday) broke destination
   extraction outright: `"Book a flight from Patna to Rachi on next
   sunday."` returned `destination: 'next'` instead of the actual
   city, and that wrong value then correctly-by-design persisted via
   `ContextManager` into the next turn's response too, compounding the
   error. Fixed by training `book_flight` on both "for" and "on"
   (where "on" is grammatical — weekday and ordinal dates only, not
   "on two weeks from now") and the full week of weekday names, not
   just two of them. Verified: date extraction went from silently
   dropping "next sunday" entirely to 100% correct regardless of
   preposition or casing.

   **What this fix did *not* solve, and why**: the same test with a
   *misspelled* city ("Rachi" instead of "Ranchi", which is in the
   training vocabulary) still fails to extract a destination at all —
   correct spelling ("Ranchi") extracts perfectly every time, typo'd
   spelling doesn't. This is a different, harder problem than phrasing
   coverage: NLU here is word-level with no character/subword
   modeling, so a token it's never seen has no path to being
   recognized as "probably a city name" — its embedding is
   indistinguishable `<unk>` from any other unknown word. Broadening
   training phrasing can't fix this; it would need either fuzzy
   matching before tagging, or a genuine architecture change
   (character-level or subword embeddings) to generalize from partial
   spelling similarity. Worth knowing if your domain has
   frequently-misspelled proper nouns.

### General-purpose / chit-chat intents

The 5 non-transactional intents (`greeting`, `farewell`, `thanks`,
`smalltalk`, `help`) apply bug #2's lesson directly: each gets its own
unique `dialogue_act` from the start, which matters *more* here than
for the transactional intents — `farewell`, `thanks`, `smalltalk`, and
`help` all have zero slots, so the DA vector's only discriminating
signal for any of them is that unique `act` bit; there's no slot
presence to fall back on. `greeting` is the exception with one
required slot (`name`) — a genuine test of the same single-slot
extraction already proven for emails and order IDs, just with a
first-name pool instead. `ContextManager` carries a captured name
across turns of the same intent as expected (a repeated "hi" without
restating your name still gets addressed by name) — verified this
doesn't happen on a *fresh* conversation with no prior name; NLG
correctly raises the missing-required-slot error instead of
hallucinating one, which `NLUNLGPipeline.respond()` surfaces as a
`ValueError` for the caller to handle.

### Function mapping and RAG fallback

Two capabilities live in `RAGRetriever` (not a separate framework) and
get used automatically by `NLUNLGPipeline.respond()`:

**Function mapping** — `RAGRetriever.map_function(intent_id, func)`
registers a plain Python function to run when that intent is
recognized, *before* generation. No parameter schema to declare
separately: whatever the function's own keyword arguments are named
(`filename`, `content`, ...) is exactly what gets filled in from
matching slots, via `inspect.signature` — see `functions.py`'s
`FileSystemFunctions` for a real sandboxed example (create/list/
read/delete a file, with path-traversal rejected via a plain
`PermissionError` before touching disk):

```python
from pythonaibrain_nlp import RAGRetriever, FileSystemFunctions

retriever = RAGRetriever()
retriever.index(SOURCES)
FileSystemFunctions("./workspace").register_all(retriever)  # maps create_file/list_files/read_file/delete_file

# ... build NLGGenerator(intent_set, retriever, nlg_model, ContextManager()), NLUParser as usual ...
pipeline = NLUNLGPipeline(nlu_parser, nlg_gen)
result = pipeline.respond("create a file test.txt")
# result.source == "function" — a real file was created; its output ("status": "created")
# merged into the slots NLG generated the confirmation from
```

**RAG fallback** — `min_confidence`/required-slot checks used to just
raise. Now, when a parse looks untrustworthy (low confidence, an
unrecognized intent, or required slots NLU couldn't fill), `respond()`
searches the retriever's full text index for the raw query *before*
raising — so a question whose answer lives in an indexed policy doc
but was never turned into a defined intent still gets answered:

```python
pipeline.respond("how many kilograms of baggage can I bring on an economy flight")
# no such intent exists — answered directly from the indexed flights policy text
# result.source == "rag_fallback", result.intent_id is None
```

A `ValueError` is still raised if RAG has nothing good either — set
`rag_fallback=False` to disable this and always raise immediately, or
tune `rag_fallback_min_score` (default 0.25) if a fallback answer is
firing on too-weak a match. That default came from a real measurement,
not a guess: testing found `"I want to book a flight"` (missing
required slots, should prompt for them) scoring 0.21 against the
flights policy purely from sharing the word "flight", versus 0.37 for
a genuine question — without a floor, the weak match would win over a
more useful "please give me origin/destination" prompt just for
scoring above zero.

Honest limitation: a fallback answer is the *entire* matching chunk,
not just the sentence that answers the question — visible above,
where the baggage question's answer comes back wrapped in three
unrelated sentences about flight changes and cancellations. There's no
LLM here to extract just the relevant part; `chunk_text()` (see
retriever.py) already keeps chunks short (~60 words, sentence-aligned)
specifically to limit how much irrelevant text rides along, but it
doesn't eliminate it. Chunking your source documents more finely (one
fact per chunk) is the practical mitigation if this matters for your
domain.

**Softmax overconfidence, and why two signals catch it where one
doesn't**: testing found NLU's confidence score alone isn't a reliable
"this doesn't belong to any known intent" signal — genuinely
nonsensical input (`"asdkj qwerty nonsense gibberish"`) still scored
0.70-0.97 confidence, always landing on whichever intent had the most
open-ended slot (`file_complaint`'s free-text `topic`), never
dropping low enough to trigger any reasonable threshold. Requiring the
top intent's required slots to actually be fillable turned out to be
a second, more reliable signal — every one of those overconfident
misfires also had empty extracted slots for an intent that requires
one. `respond()` checks both together rather than confidence alone.

Honest scope note: none of this is the actual Model Context Protocol
(no JSON-RPC 2.0 messages, no stdio/SSE transport, no connecting to
external MCP servers) — it's a local function-calling layer living
inside the RAG retriever, sized for "a recognized intent triggers a
real action," not for connecting to arbitrary external tool servers.

## CLI

```
pythonaibrain-nlp validate path/to/nlg_intents.json
pythonaibrain-nlp demo
```

## Testing

```
pip install -e ".[dev]"
pytest tests/ -q
```

## Design notes / honest limitations

- **`pattern` closes the "NLU only sees system text" gap, but only
  when you actually populate it.** Before this field existed, NLU had
  no choice but to train on `reference_text` alone — confirmations
  like "Booking X is now cancelled" — which generalizes noticeably
  worse to real user commands ("please cancel my booking X") than to
  text in that same confirmation style. `pattern` gives NLU genuine
  query-style examples to train on directly (verified: 5/5 correct on
  held-out query phrasing in the large-corpus test above). The gap
  doesn't disappear on its own, though — an `IntentsBuilder.add_example()`
  call that skips `pattern` still leaves NLU with only `reference_text`
  to learn from for that example, same limitation as before.
- Even batched, the SC-LSTM is a sequential (per-timestep) recurrent
  decoder — throughput scales with batch size and sequence length,
  but it's not going to match a fully parallelizable architecture on
  raw tokens/sec. Batching + gradient accumulation + AMP get you real
  GPU utilization; they don't remove the recurrence.
- `RAGRetriever` uses sparse TF-IDF, not dense/transformer embeddings,
  to stay offline and dependency-minimal; `CooccurrenceEmbedder`
  (truncated SVD over a sparse co-occurrence matrix, plain numpy/scipy
  — not trained via backprop) fills the "dense vector" role instead
  where one's needed for the context vector.
- The SC-LSTM decodes greedily at inference; beam search would be a
  straightforward addition to `NLGModel.generate()` if higher-quality
  decoding is needed later. Inference is still single-example, not
  batched — batching matters for training throughput, not for serving
  one generation request at a time.
- `NLGModel.load()` / `Trainer.load_checkpoint()` use `weights_only=False`
  since checkpoints intentionally bundle non-tensor state (vocab,
  fitted embedder, optimizer state) — only load checkpoints from a
  source you trust.
- `max_vocab` caps (default 50k for TF-IDF, 20k for the co-occurrence
  embedder) bound memory on a pathologically large or noisy
  vocabulary; raise them if your domain genuinely needs a larger
  vocabulary and you have the memory for it.
