Metadata-Version: 2.5
Name: fds-bot
Version: 0.2.0
Summary: A chatbot for the FDS course at UZH
Project-URL: Homepage, https://www.ifi.uzh.ch/en/dast.html
Author-email: Christoph Mayer <mayer@ifi.uzh.ch>, Giuseppe Doda <giuseppe.doda@uzh.ch>
License-Expression: MIT
License-File: LICENSE
Keywords: UZH,chatbot,education,fds,machine-learning
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Education
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Education
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: <3.15,>=3.10
Requires-Dist: cryptography<49; sys_platform == 'darwin' and platform_machine == 'x86_64'
Requires-Dist: faiss-cpu<1.15,>=1.12; python_version >= '3.14'
Requires-Dist: faiss-cpu==1.10.0; python_version < '3.14'
Requires-Dist: fastapi[standard-no-fastapi-cloud-cli]<0.136.0,>=0.135.2
Requires-Dist: google-genai<2,>=1.68.0
Requires-Dist: pydantic-settings<3,>=2.13.1
Requires-Dist: pymupdf<1.28,>=1.27.2.2
Requires-Dist: rapidfuzz<4,>=3.14.5
Requires-Dist: sqlmodel<0.0.38,>=0.0.37
Requires-Dist: sse-starlette<4,>=3.3.4
Requires-Dist: structlog>=25.4.0
Description-Content-Type: text/markdown

# FDS Bot

A locally deployable RAG study assistant for the **Foundations of Data Science** course at UZH.

## Overview

FDS Bot gives students 24/7 access to an AI tutor grounded exclusively in course materials. Course content is preprocessed centrally (LaTeX lecture notes parsed, chunked, enriched, and embedded) into a single **course bundle**, which is distributed to enrolled students through the course's own channel and imported by the app on first launch. Students run the application locally with free-tier [Gemini API](https://ai.google.dev/).

Key capabilities:

- **Hybrid retrieval**: vector similarity (FAISS) + keyword search (SQLite FTS5) with reciprocal rank fusion; a deterministic query router classifies each message (`section_overview` / `exact_lookup` / `question` / `anaphoric` / `default`) and pins exact identifier matches (named definitions/exercises, section titles, "chapter N") to the top ranks. A leading summarize imperative naming a section ("summarize the section on ridge") routes `section_overview`: retrieval serves that section's entire subtree in document order and the answer is generated under a dedicated summarize-mode system prompt.
- **Streaming chat**: Gemini-powered responses over Server-Sent Events, with LaTeX math rendering and code highlighting.
- **Citation system**: every answer front-loads citations (chunk id, rank, source title + section + snippet) before the first token.
- **Grounded quotes**: after the answer, each cited source carries the verbatim quotes the assistant pulled from it, each verified locally against the source text (`rapidfuzz`); only quotes that could NOT be confirmed are flagged (unverifiable).
- **Rate limiting**: two per-model-family `RateLimiter`s: the user-configured `gemini_rpm` governs generation calls (stream, list-models, verify-key); embeddings run on their own fixed budget (Gemini embedding quota is separate and higher).
- **Conversation history**: persistent sessions with messages, citations, and user preferences stored in a local SQLite database (`~/.fds-bot/fds-bot.db`).
- **Student feedback**: students select answers and package them (with app + model metadata) into a clipboard copy or `.md` download to email the course maintainer; entirely client-side, no server-side collection.

## Architecture

| Component | Technology | Why |
|---|---|---|
| Web framework | FastAPI | Async JSON + SSE API mounted at root (no `/api` prefix); also serves the built SPA as static assets |
| Frontend | React + Vite + TypeScript + Tailwind + shadcn/ui | Static SPA built at dev time and served by FastAPI — **no Node in the student runtime** |
| Streaming | `text/event-stream` via `StreamingResponse` (POST) | Unidirectional fits chat; consumed client-side by `fetch` + `eventsource-parser` (the POST stream rules out native `EventSource`) |
| Vector search | FAISS (faiss-cpu, `IndexFlatIP`) | Fast, pre-built indices serialize to files |
| Keyword search | SQLite FTS5 (BM25 ranking) | Built-in to Python |
| Fusion | Reciprocal Rank Fusion (`k=60`) | Standard hybrid-retrieval blend |
| Embeddings | Gemini `gemini-embedding-001` | Same model at index-build and query time |
| LLM | Gemini API (`gemini-3.1-pro-preview`) | Free tier available |
| Chat database | SQLite via SQLModel + SQLAlchemy 2.0 | `~/.fds-bot/fds-bot.db` (preferences, conversations, messages, citations) |
| Bundle database | Read-only SQLite imported at first launch | `chunks.db` + FAISS indices + `notes.pdf`, downloaded from the course page and installed under `~/.fds-bot/bundles/` — never part of the wheel |
| PDF source viewer | PyMuPDF (`fitz`) | Serves the cited page of `notes.pdf` with all of the citation's verified quotes highlighted server-side at `GET /chunks/{chunk_id}/pdf` (repeatable `quote` param; page derived from chunk metadata, never client-supplied) |
| Logging | structlog | Errors only by default, so the terminal stays readable; structured output with sensitive-data scrubbing, rendered tracebacks included |
| LaTeX parsing | Regex-based | Pipeline-only (not shipped to students) |

## Project Structure

```
src/fds_bot/
├── cli.py                   # Console-script entry point (fds-bot) — Ctrl-C guard around main.py's import
├── main.py                  # FastAPI app factory, lifespan, CLI command handling
├── config.py                # Startup Settings (host, port, log level, data dir)
├── __init__.py              # __version__, __commit_sha__ (derived from hatch-vcs)
│
├── domain/                  # Pure domain layer — Pydantic records + Protocol ports + authored prompts
│   ├── chat.py              # Conversation/Message records, ChatStore port, SSE event types
│   ├── chat_manager.py      # ChatManager — user-message → retrieval → streaming pipeline
│   ├── prompts.py           # SYSTEM_PROMPT (pedagogical contract) + build_system_instruction (pure builder)
│   ├── query_router.py      # route_query — deterministic intent classification + lookup extraction
│   ├── retrieval.py         # RetrievalEngine port, RetrievalResult types
│   ├── pdf.py               # PdfHighlighter/ChunkPageReader/BundleFingerprintReader ports, ChunkPdfService, compute_etag
│   ├── text_normalize.py    # normalize_for_match — shared LaTeX-strip/casefold (quote verifier + PDF search text)
│   ├── quote_verifier.py    # Local quote verification (rapidfuzz) + trailing-Sources strip + inline-citation audit
│   ├── macro_expansion.py   # compile_macro_expander — glossary LaTeX-macro expansion for the retrieval query
│   ├── llm.py               # LLMClient port, GenerationRequest, TokenDelta
│   ├── preferences.py       # UserPreferences model, PreferencesStore + PreferencesReader ports, PreferencesManager
│   ├── errors.py            # Domain exception taxonomy
│   └── _types.py            # Shared literal / alias types
│
├── db/                      # Chat database schema + lifecycle
│   ├── manager.py           # DatabaseManager — engine, pragmas, schema init, migrations
│   ├── models.py            # SQLModel tables (Preferences, Conversation, Message, Citation)
│   └── sql_helpers.py       # ISO-8601 timestamp helpers shared by stores
│
├── repositories/            # Concrete chat-DB port adapters
│   ├── chat_store.py        # SQLiteChatStore
│   └── preferences_store.py # SQLitePreferencesStore
│
├── adapters/                # External-dep adapters
│   ├── retrieval/           # Hybrid retrieval over the packaged bundle
│   │   ├── engine.py        # SQLiteFaissRetrievalEngine orchestrator + RRF
│   │   ├── vector.py        # Pure FAISS search
│   │   ├── keyword.py       # Pure FTS5 search with BM25 normalization
│   │   └── models.py        # Chunk SQLModel (read-only bundle schema)
│   ├── pdf/                 # PyMuPDF PDF adapters (PdfHighlighter, BundleChunkPageReader, ManifestFingerprintReader)
│   └── llm/                 # Gemini SDK adapter
│       ├── client.py        # GeminiLLMClient (uses domain.prompts.build_system_instruction)
│       └── rate_limiter.py  # Async RateLimiter with dynamic rpm_fn
│
├── api/                     # HTTP layer
│   ├── routes/              # FastAPI routers (health / preferences / conversations / messages / chunks / pics)
│   ├── schemas/             # Pydantic request + response DTOs per resource
│   ├── dependencies.py      # Depends() factories reading from app.state
│   ├── errors.py            # RFC 7807 handlers + ProblemHTTPException
│   ├── spa.py               # Static-SPA serving (Option A): /assets mount + index.html catch-all, no-op without a build
│   └── mappers.py           # Pure domain-record → API-DTO converters
│
├── data/                    # Course bundle in a dev checkout (chunks.db, FAISS, glossary.json, manifest.json, notes.pdf, pics/) — gitignored, never in the wheel; students import theirs at runtime
├── static/                  # Built SPA (index.html + hashed assets) — emitted by `make build-frontend`; gitignored
├── pipeline/                # Offline preprocessing (excluded from wheel)
├── eval/                    # Offline retrieval-eval harness + LLM judge (excluded from wheel)
└── utils/                   # structlog wiring + shared log-level types

frontend/                    # React + Vite SPA source — chat, citations, figure lightbox (Node is a build-time-only dev dependency)
tests/                       # Mirrors source; includes contract/, e2e/, unit/integration
docs/                        # Maintainer-local notes (design record, course-channel hand-off) — not distributed
```

Dependency arrows: `api → domain`, `repositories → domain`, `adapters → domain`. `domain/` imports nothing from `db/`, `repositories/`, `adapters/`, or `api/`. `repositories/` imports from `db/` (SQLModel tables + engine). `pipeline/` is offline-only and not imported at runtime.

## Quick Start

### Students

**Requires Python 3.10–3.14** — check with `python3 --version`. The app and its
web UI install from PyPI; the course content comes from your course page.

**1. Install the app**

```bash
pip install fds-bot
```

**2. Get the course bundle**

Download `fds-bot-data-YYYY-MM-DD.tar.gz` (about 65 MB) from the FDS course page.
It holds the lecture notes and the search indices built from them —
course material shared only with enrolled students, which is why it comes from
the course page and not from PyPI. Leave it packed; the app takes the file
exactly as it downloaded.

**3. Run it, and import the bundle**

```bash
fds-bot       # serves the app at http://127.0.0.1:8000 and opens it in your browser
```

The terminal stays almost silent, and that is the healthy state rather than a
stalled start:

```text
fds-bot is ready — open http://127.0.0.1:8000/ in your browser.

Press Ctrl+C to stop.
```

Only errors are logged, so anything printed alongside those lines is worth
reading. On a first launch that means one short block for each thing still
missing, just above the banner.

The first start asks for two things, right where you would chat:

- **The course bundle** — drag the downloaded `.tar.gz` onto the page, or use
  the file picker. (Safari sometimes unpacks the download into a folder on
  arrival; there is a folder picker for that.) The app copies it into its own
  data folder under `~/.fds-bot/`, so you can delete the download afterwards.
  Nothing is sent anywhere — the import happens on this machine. No browser
  handy? `fds-bot --import-bundle path/to/fds-bot-data-YYYY-MM-DD.tar.gz` does
  the same from the terminal and then carries on serving.
- **A Gemini API key** — grab a free
  [Gemini API key](https://aistudio.google.com/apikey), paste it in, and you are
  going. The terminal prints the same instructions. If you prefer,
  `export GEMINI_API_KEY=...` before the *first* launch and it is picked up
  automatically. Your key stays on your machine.

The browser opens by itself once the server is ready. The address is printed on
every launch either way — it is ordinary terminal output rather than a log line,
so quieting the logs never hides it — and you can always open it yourself. Pass
`--no-browser` to keep it from opening (it is also skipped automatically on a
machine with no desktop).

When the notes are updated mid-semester, download the new file and import it
from **Settings → Course material → Replace** — no reinstall needed. That
settings card shows the build date of the bundle you have, and
`fds-bot --version` prints it next to the app version — the two things a support
request needs.

<details>
<summary>Platform requirements</summary>

Pre-built wheels cover all supported platforms, so no compiler is needed:

| Platform | Requirement |
|---|---|
| Linux | x86_64 or arm64, glibc ≥ 2.28 (Ubuntu 20.04+, Debian 10+, RHEL 8+) |
| macOS (Apple Silicon) | macOS ≥ 11 on Python 3.10–3.13; macOS ≥ 14 on Python 3.14 |
| macOS (Intel) | macOS ≥ 10.14 on Python 3.10–3.13; macOS ≥ 13 on Python 3.14 |
| Windows | AMD64 |

If `pip` starts *compiling* anything, you are on an unsupported combination —
an older Python (3.10–3.13) is the quickest fix.

</details>

## Support

Found a wrong answer or a bug? The app has a built-in feedback flow: select the
answers in question and use **Copy** or **Download** to package them — with the
app and model metadata already filled in — and email the result to the course
maintainer.

## Development

Source access is internal to the UZH DaST group. Contact
[mayer@ifi.uzh.ch](mailto:mayer@ifi.uzh.ch) or
[giuseppe.doda@uzh.ch](mailto:giuseppe.doda@uzh.ch). Contributors: see
`CONTRIBUTING.md` for the development workflow and `openapi.yaml` for the
authoritative HTTP + SSE contract. The design and decision record
(`docs/DESIGN.md`) is kept maintainer-local and is not distributed with the
repository — ask a maintainer if you need it.
