LoCAL2 — Plan 2: Multimodal, Tools, RAG

Plan 1 (Phases 1–6) established the core: LLM-native tool calling, episodic memory, pairwise evaluation, and per-participant observability UI. Plan 2 extends Gemma's capabilities with file attachments, new tools, and a persistent knowledge base.

Phase 7 — File Attachments + Multimodal

Supported file types

UI changes — MainWindow

Attachment button

A paperclip button (⌁) sits to the left of the query input. Clicking it opens a QFileDialog. Drag-and-drop onto the input area also works — dragEnterEvent / dropEvent on the input container.

Attachment chips

Attached files appear as small chips above the query input. Each chip shows the filename and an ✕ to remove it. Multiple files are allowed. Chips are cleared after the query is sent.

┌─────────────────────────────────────────────────┐
│ 📎 diagram.png ✕   📎 notes.pdf ✕               │  ← attachment chips
├─────────────────────────────────────────────────┤
│ What does this architecture diagram show?  Send │  ← query input
└─────────────────────────────────────────────────┘

Response card

The existing StreamingResponseWidget shows a small attachment summary line: "[attached: diagram.png, notes.pdf]" below the query badge.

Attachment processing

Processing happens in the UI thread at send time (files are small, extraction is fast). No background worker needed unless PDFs are very large.

def _process_attachment(path: str) -> dict:
    ext = Path(path).suffix.lower()
    if ext in {".jpg", ".jpeg", ".png", ".gif", ".webp"}:
        data = base64.b64encode(Path(path).read_bytes()).decode()
        return {"type": "image", "name": Path(path).name, "data": data}
    elif ext == ".pdf":
        text = _extract_pdf_text(path)   # pypdf
        return {"type": "text", "name": Path(path).name, "data": text}
    elif ext in {".txt", ".md", ".py", ".js", ...}:
        text = Path(path).read_text(errors="replace")
        return {"type": "text", "name": Path(path).name, "data": text}
    else:
        return {"type": "error", "name": Path(path).name}

Bus protocol

query.received payload gains an attachments field:

{
  "query": "What does this architecture diagram show?",
  "session_id": "...",
  "query_id": "...",
  "attachments": [
    {"type": "image", "name": "diagram.png", "data": "<base64>"},
    {"type": "text",  "name": "notes.pdf",   "data": "extracted text..."}
  ]
}

GeneratorAgent — _build_messages

When attachments are present, the user message is built differently:

• Text attachments: prepended as a fenced block before the query:

  [Attached: notes.pdf]
  <extracted text (truncated to max_attachment_chars)>
  
  <user query>

• Image attachments: the message dict gets an images list alongside the content string. Ollama passes these to Gemma's vision encoder.

  {
    "role": "user",
    "content": "What does this architecture diagram show?",
    "images": ["<base64>"]
  }

• Mixed: text context prepended to content string, images in images list.

Config — generator.yaml

max_attachment_chars: 8000   # truncation limit per text attachment

New dependency

pypdf>=4.0

New files

Modified files

Build order

1. Add pypdf to requirements, install
2. Build AttachmentBar widget (chips, file picker, drag-drop)
3. Wire AttachmentBar into MainWindow input area
4. Extend query.received payload with attachments
5. Update _build_messages to handle image and text attachments
6. Smoke test: attach an image, ask "what do you see?"
7. Smoke test: attach a PDF, ask a question about it
8. Story S9: multimodal acceptance test

Phase 8 — Date/Time + Location Tools

DateTimeTool

Returns current local date, time, timezone, and day of week. Single system call.

Tool name: get_datetime
Schema: no parameters required
Result: "Tuesday 2026-06-03 09:17:42 PDT (UTC-7)"

No config file needed. Follows the existing tool pattern: announces schema on startup, responds to tool.request.get_datetime.

LocationTool

Reads from config/location.yaml and returns a structured location string. No live geolocation — user sets their own context.

# config/location.yaml
city: "Cupertino"
state: "California"
country: "United States"
timezone: "America/Los_Angeles"
coordinates: "37.3230° N, 122.0322° W"   # optional, for distance queries

Tool name: get_location
Result: "Cupertino, California, United States (America/Los_Angeles, 37.3230° N, 122.0322° W)"

New files

Phase 9 — Semantic Scholar Tool

Tool schema

Tool name: search_papers
Parameters:
  query  (string, required) — research topic or keywords
  limit  (integer, optional, default 5) — max papers to return

API

Uses the Semantic Scholar Graph API (https://api.semanticscholar.org/graph/v1/paper/search). Free, no API key required for basic use (rate limited to 100 req/5min). Optional key for higher limits, stored in .env.

Result format

[2026-06-03] Papers: "transformer attention mechanisms"

1. Attention Is All You Need (2017) — Vaswani et al.
   Citations: 98,432  |  https://semanticscholar.org/paper/...
   Transformers dispense with recurrence entirely, relying instead on...

2. ...

Config — semantic_scholar.yaml

max_results: 5
timeout: 15
fields: "title,authors,year,abstract,citationCount,url"

Phase 10 — RAG: Persistent Document Knowledge Base

Design

A separate ChromaDB collection (collective.documents) holds chunked, embedded document passages. A search_documents tool lets Gemma query it semantically, same pattern as search_memory.

Ingestion

Two paths:

• CLI ingest script: python scripts/ingest.py path/to/file.pdf — chunks, embeds, stores. Runs offline, no stack needed.
• UI "Add to knowledge base" button: appears in the MemoryWindow or a new DocumentsWindow. Attaches a file, triggers ingestion in a background thread.

Chunking strategy

Fixed-size chunks (512 tokens) with 64-token overlap. Each chunk stored with metadata: source_file, chunk_index, page (for PDFs), ingested_at.

Tool schema

Tool name: search_documents
Parameters:
  query  (string, required) — what to look for in the knowledge base

New files

Phase 11 — Documentation

• Architecture overview (bus topology, participant roles, message flow diagrams)
• Developer guide: adding a new tool, adding a new agent
• Story authoring guide
• Config reference (all YAML files, all keys)

Build Order Summary