System Architecture

Queries are embedded with nomic-embed-text and stored as high-dimensional vectors in ChromaDB. On each new query, the embedding is computed and the top-k nearest neighbors are fetched by cosine similarity. If the closest match exceeds the 0.92 threshold, the stored answer is returned instantly — no LLM call.

• Threshold tunable via CACHE_SIMILARITY_THRESHOLD env var
• Async embedding via thread pool — never blocks FastAPI event loop
• Cache grows passively; every answered query seeds future hits
• ChromaDB persists to disk across restarts (cache/data/chromadb/)

Classification is a two-layer cascade. Layer 1 runs 80+ regex rules in ~0.01 ms and handles roughly 80% of queries — no LLM token burn. Only when confidence falls below CLASSIFIER_DUAL_THRESHOLD does Layer 2 fire an Ollama LLM as a second opinion.

• Rules cover: notes, code, math, creative, general, complex, vision, retrieval
• Each rule has a weight and minimum confidence to trigger Layer 2
• Dual opinion aggregated by highest confidence wins
• Semaphore limits Ollama calls — no VRAM saturation under bursty load

Whenever a new answer displaces an existing cached one (the user ran the same query again and got a different model), the Elo rating of each model is updated. No explicit rating calls, no annotations — the act of displacement is the signal.

• Winner: the model whose answer was chosen to replace the old one
• Loser: the model whose answer was evicted from cache
• Elo K-factor: 32 (standard competitive rating)
• Leaderboard visible at GET /api/leaderboard
• Tier selection uses Elo rankings to pick highest-rated model at each price tier

A library of 200+ deterministic answers for queries where an LLM adds zero value. Pattern-matched in pure Python before the pipeline even starts — under 0.1 ms, zero network, zero cost.

• Dates/time: "what day is it", "what year", "current time"
• Math: arithmetic, powers, factorials via eval() sandbox
• Philosophy quotes: Stoics, Socrates, Nietzsche, Kant
• Music theory: chord names, intervals, scales
• HTTP codes: 200, 404, 418, 500, 503 explained
• Identity: "who are you", "what model" — always honest about being muLLM

Multi-part prompts (e.g. "explain X, write code for Y, and summarize Z") are decomposed into independent sub-tasks, each routed to the cheapest model that can handle its complexity, then executed in parallel with asyncio.gather().

• Decomposition via local LLM — one cheap classification call
• Sub-tasks can mix tiers: simple sub-task = local free, complex = cloud cheap
• Total cost = sum of sub-task costs, often cheaper than routing the whole thing cloud
• Response recombined with section headers per sub-task

The router never fires cloud unless complexity demands it. Tier selection uses the intent complexity score (1–5), Elo rankings, daily budget remaining, and spend velocity to pick the cheapest capable model.

• Daily hard cap: default $5/day, configurable via DAILY_BUDGET_CAP
• Auto-approve ceiling: cloud calls below threshold proceed without confirmation
• Ask-approval ceiling: calls above threshold prompt user in the Unblock panel
• Spend velocity: if cost/minute spikes, tier ceiling lowers automatically

Every query result publishes a cost event to an SSE event bus. The chat UI receives real-time token counts and dollar amounts without polling. Session cost accumulates in memory and is queryable at any time.

• GET /query/stream — SSE endpoint, each message includes cost field
• GET /api/session/cost — current session total
• Multiple dashboard subscribers each get their own asyncio.Queue
• Queue overflow protection: put_nowait silently drops if subscriber is slow