# Self-hosted generative AI

Run HuggingFace models on **your own hardware** — no external API, no data
leaving your servers. The `tempest_fastapi_sdk.genai` module ships in
slices; this page covers the **first**: knowing, *before* you download
gigabytes of weights, whether the machine can handle the model.

!!! info "Module roadmap"
    - **v0.96:** `genai.hardware` — probing + `can_run` / `recommend`.
    - **v0.97:** `genai.rag` — RAG context (SearXNG web search + PDF
      reading) to inject into LLMs (this page, [RAG context](#rag-context)).
    - **v0.98:** `TextGenerator` — local LLM + int4/int8 quantization
      (section [Generate text](#generate-text-with-a-local-llm)).
    - **v0.99:** `Embedder`, `BatchScheduler`, `ModelRegistry` — embeddings
      + scale (section [Embeddings and scale](#embeddings-and-scale)).
    - **v0.102:** `SpeechToText` / `TextToSpeech` — audio (section
      [Audio (voice)](#audio-voice)).

The `[genai]` extra (transformers + torch + accelerate) is only needed to
**run** models. The capacity functions **import without the extra** —
`torch` is only used (when present) to read real GPU VRAM.

## "Can the machine handle it?"

Loading a too-large model ends in an OOM minutes into the download.
`can_run` answers first:

```python
from tempest_fastapi_sdk.genai import can_run, ModelDtype

report = can_run(model_id="Qwen/Qwen2.5-7B-Instruct", dtype=ModelDtype.BFLOAT16)

if report.fits:
    print(f"OK on {report.device} — {report.headroom_pct:.0f}% headroom")
else:
    print(report.reason)
    print("Suggestion:", report.suggestion)   # e.g. "Quantize to int4 ..."
```

`CapacityReport` carries: `fits`, `device` (`cuda`/`mps`/`cpu`),
`estimated_bytes` vs `available_bytes`, `headroom_pct`, `reason`, and a
concrete `suggestion` when it doesn't fit (quantize, offload to CPU, or
pick a smaller model).

!!! tip "Let the SDK pick the precision"
    `recommend(...)` tries `bfloat16` → `int8` → `int4` on the best
    available device and returns the **first** config that fits:

    ```python
    from tempest_fastapi_sdk.genai import recommend

    best = recommend(model_id="meta-llama/Llama-3.1-8B")
    print(best.device, best.dtype, best.fits)   # e.g. cuda int8 True
    ```

## Probing the hardware

```python
from tempest_fastapi_sdk.genai import probe_hardware

hw = probe_hardware()
print(hw.cpu_cores, hw.ram_available_bytes)
print(hw.has_cuda, [g.name for g in hw.gpus])   # per-GPU VRAM when CUDA is present
```

`HardwareInfo` reports CPU, total/available RAM, CUDA GPUs (name +
total/free VRAM), MPS (Apple), and free disk space. Without `psutil` or
`torch` installed, the matching fields fall back to safe defaults (`0` /
`False` / empty list) — nothing breaks.

## Estimate without downloading weights

The math is `params × bytes-per-param × overhead`. Bytes per param come
from the precision (`float32`=4, `float16`/`bfloat16`=2, `int8`=1,
`int4`≈0.6); the overhead (×1.25) covers activations, KV cache and
runtime context.

```python
from tempest_fastapi_sdk.genai import estimate_model_bytes, ModelDtype

gb = estimate_model_bytes(7_000_000_000, ModelDtype.INT4) / 1e9
print(f"~{gb:.1f} GB")   # 7B in int4
```

The parameter count can be passed explicitly (`num_params=`) or read from
the Hub by `model_id` (via `huggingface_hub`, without downloading the
weights — safetensors metadata).

## Generate text with a local LLM

`TextGenerator` loads a HuggingFace causal LM **once** and generates on
your hardware. It resolves device and precision itself, supports int4/int8
quantization, loads the weights lazily (on first call), and frees VRAM
when idle. Needs `[genai]` (and `[genai-quant]` to quantize).

```python
from tempest_fastapi_sdk.genai import TextGenerator

gen = TextGenerator(
    "Qwen/Qwen2.5-7B-Instruct",
    quantization="int4",            # fits a modest GPU; None = full precision
    idle_unload_seconds=300,        # free VRAM after 5 min idle
)

text = await gen.generate("Explain PIX in one sentence.", max_new_tokens=128)

# chat with a role template:
reply = await gen.chat([
    {"role": "system", "content": "You answer in English."},
    {"role": "user", "content": "What is PIX?"},
])

# token-by-token streaming:
async for piece in gen.stream("Write a haiku about rain."):
    print(piece, end="", flush=True)

gen.unload()                        # free the memory now
```

Blocking generation runs in `asyncio.to_thread` — it never blocks the
event loop. `device="auto"` picks CUDA → MPS → CPU; `dtype="auto"` uses
bf16 on GPU and fp32 on CPU.

!!! tip "Check before loading"
    Pair it with the [capacity check](#can-the-machine-handle-it): run
    `can_run` / `recommend` to pick a `quantization`/`device` that **fits**
    before instantiating the `TextGenerator`.

!!! tip "Free VRAM between bursts"
    With `idle_unload_seconds` set, call `gen.unload_if_idle()` periodically
    (e.g. in a `@tq.interval(60)` [TaskQueue](queue-tasks.md) task) — it
    unloads only once past the idle threshold, no background-thread magic.
    `unload()` frees immediately.

## Embeddings and scale

### Generate embeddings

`Embedder` turns text into vectors on your hardware (semantic search, RAG,
clustering). It loads the model once, batches, and (optionally) caches a
vector per text — a cache hit never touches the model.

```python
from tempest_fastapi_sdk.genai import Embedder, InMemoryEmbeddingCache

emb = Embedder(
    "sentence-transformers/all-MiniLM-L6-v2",
    cache=InMemoryEmbeddingCache(),     # or a Redis wrapper (get/set)
)

vectors = await emb.embed(["what is pix?", "how to refund?"])   # list[list[float]]
```

`cache` is any object with `get(key)->list|None` and `set(key, val)` —
pass a wrapper over `AsyncRedisManager` to share across workers.
`device`/`dtype`/`unload`/`unload_if_idle` work as on `TextGenerator`.

For semantic search, use `normalize=True` (unit vectors) + the
`cosine_similarity` function:

```python
from tempest_fastapi_sdk.genai import cosine_similarity

emb = Embedder("sentence-transformers/all-MiniLM-L6-v2", normalize=True)
q, *docs = await emb.embed(["question", "doc a", "doc b"])
ranked = sorted(docs, key=lambda d: cosine_similarity(q, d), reverse=True)
```

### Batch concurrent inference

On a GPU, one item at a time wastes the device. `BatchScheduler` coalesces
concurrent calls into a single batch — each caller still `await`s its own
result:

```python
from tempest_fastapi_sdk.genai import BatchScheduler

sched = BatchScheduler(emb._embed_many, max_batch=32, max_wait_ms=10)

# N concurrent requests become 1 forward pass:
vector = await sched.submit("text")
await sched.aclose()
```

It forms a batch once `max_batch` items are queued **or** `max_wait_ms`
has elapsed since the first — whichever comes first. A handler error
propagates to every caller in that batch.

### Share loaded models

`ModelRegistry` keeps loaded models by id (LRU) — two call sites asking
for the same model reuse the instance, and the least-recently-used one is
unloaded (`unload()`) once over `max_models`:

```python
from tempest_fastapi_sdk.genai import Embedder, ModelRegistry

registry = ModelRegistry(max_models=2)

def get_embedder(model_id: str) -> Embedder:
    return registry.get(model_id, lambda: Embedder(model_id))
```

## Audio (voice)

Interpret and generate voice on your hardware — no external API. Needs the
`[genai-audio]` extra (faster-whisper + Coqui TTS); the engines import
lazily.

### Interpret audio (STT)

`SpeechToText` transcribes with **faster-whisper** (Whisper via
CTranslate2, fast on CPU/GPU). Loads once, runs in a worker thread,
serializes calls through a semaphore.

```python
from tempest_fastapi_sdk.genai.audio import SpeechToText

stt = SpeechToText("base", device="auto")     # tiny…large-v3
result = await stt.transcribe("meeting.wav")
print(result.text, result.language, result.duration)
for seg in result.segments:                    # per-span timestamps
    print(seg.start, seg.end, seg.text)
```

Accepts a path or `bytes`. `device`/`compute_type` resolve automatically
(`float16` on GPU, `int8` on CPU).

### Generate voice (TTS)

`TextToSpeech` synthesizes with **Coqui TTS** (WAV). Same discipline
(lazy + thread + semaphore).

```python
from tempest_fastapi_sdk.genai.audio import TextToSpeech

tts = TextToSpeech("tts_models/multilingual/multi-dataset/xtts_v2")
wav = await tts.synthesize("Hello, world.", language="en")   # -> WAV bytes
# voice cloning (XTTS): pass a reference clip
wav = await tts.synthesize("Hi!", language="en", speaker_wav="ref.wav")
```

`synthesize` returns the WAV `bytes`; pass `out_path=` to also write it to
disk.

### Language (PT-BR / EN-US)

No need to know the Whisper code or pick a TTS model: use the `Language`
enum. It resolves the code (`pt`/`en`) for STT and a good TTS model per
language:

```python
from tempest_fastapi_sdk.genai.audio import Language, SpeechToText, TextToSpeech

# STT: force the language without memorizing the code
await SpeechToText("base").transcribe("audio.wav", language=Language.PT_BR)

# TTS: picks the language's default model automatically
tts = TextToSpeech.for_language(Language.EN_US)     # en-US model
wav = await tts.synthesize("Hello, world.")
```

`preset_for(Language.PT_BR)` exposes the preset (`whisper_language`,
`tts_model`, `tts_language`) to inspect/override. `language=` on
`transcribe`/`synthesize` also accepts a raw code (``"pt"``) or ``None``
(auto-detect for STT).

!!! tip "Full voice loop"
    Chain with the LLM: **STT** transcribes speech → `TextGenerator`/RAG
    answers → **TTS** speaks the reply. All local, nothing leaves the box.

## RAG context

A local LLM only knows what it trained on. For current, grounded answers,
inject context: `tempest_fastapi_sdk.genai.rag` searches the web
(self-hosted SearXNG), extracts page bodies, reads PDFs, and assembles a
prompt-ready block — without sending data outside. Needs the `[genai-rag]`
extra (httpx + trafilatura + pymupdf); the pieces import lazily.

### Web search (SearXNG)

```python
import httpx
from tempest_fastapi_sdk.genai.rag import SearxngBackend, WebSearch, build_context

client = httpx.AsyncClient()
search = WebSearch(SearxngBackend("http://localhost:8080", http_client=client))

results = await search.search("what is PIX?", max_results=5)   # list[SearchResult]
context = build_context("what is PIX?", results, long_text=False, max_chars=2000)
# -> a string ready to inject into your TextGenerator prompt
```

The backend is a `Protocol` (`WebSearchBackend`) — swap SearXNG for another
provider without touching call sites. The `httpx.AsyncClient` is injected
(pool reuse; wire it in the FastAPI lifespan).

!!! tip "From question to context in one call"
    `WebSearch.retrieve` does search → (optional) parallel body extraction
    → `build_context`, all at once:

    ```python
    from tempest_fastapi_sdk.genai.rag import ContentExtractor

    extractor = ContentExtractor(http_client=client)
    context = await search.retrieve("what is PIX?", extractor=extractor, max_results=5)
    answer = await gen.generate(context)
    ```

    Without `extractor` it uses snippets only. `ContentExtractor.extract_many`
    fetches N pages concurrently (bounded by `concurrency`).

### Extract page bodies

Search snippets are thin. To give the LLM ground truth, fetch each page
and extract the clean text (via `trafilatura`):

```python
from tempest_fastapi_sdk.genai.rag import ContentExtractor

extractor = ContentExtractor(http_client=client)
for result in results:
    outcome = await extractor.extract(result.url)
    result.content = outcome.text          # "" on failure; outcome.failed marks it
context = build_context("what is PIX?", results)   # now with full bodies
```

Failures (timeout, 4xx/5xx, empty page) **never** raise — they come back
as `ExtractionResult(text="", failed=True)`, so no source is silently
dropped.

### Read PDFs (knowledge base)

`PdfReader` (PyMuPDF — detailed, reading-order extraction) turns PDF paths
into text and prompt/embedding-ready chunks:

```python
from tempest_fastapi_sdk.genai.rag import PdfReader, build_context

reader = PdfReader()
doc = reader.read("/kb/manual.pdf")             # Document: text + pages + metadata
chunks = reader.chunks("/kb/manual.pdf", max_chars=2000, overlap=200)

context = build_context("how to refund?", chunks)   # cites "file (page N)"
```

`chunks(..., overlap=200)` shares characters between neighbors so a fact on
a boundary isn't cut in half; `per_page=True` (default) keeps each chunk on
a single page and records its page number.

!!! tip "Mix web + PDF in one context"
    `build_context` accepts `SearchResult` and `Chunk` in the same list —
    it delimits each source with `---` and labels the origin (URL or `file
    (page N)`) so the LLM can cite. Pass `long_text=False` to truncate each
    source to `max_chars`.

### RAG over your own corpus (vector store)

Web search is one source; the other is **your own knowledge** (PDFs, docs).
Instead of re-embedding everything each request, index once into a **vector
store** and retrieve by similarity. `Retriever` ties `Embedder` → store →
`build_context`:

```python
from tempest_fastapi_sdk.genai import Embedder
from tempest_fastapi_sdk.genai.rag import InMemoryVectorStore, PdfReader, Retriever

rag = Retriever(Embedder("sentence-transformers/all-MiniLM-L6-v2", normalize=True),
                InMemoryVectorStore())

await rag.index(PdfReader().chunks("/kb/manual.pdf"))     # once
context = await rag.retrieve("how to refund?", top_k=5)   # cheap, afterwards
answer = await gen.generate(context)
```

- **`VectorStore`** is a `Protocol` — `InMemoryVectorStore` (dev/tests,
  cosine scan) or `PgVectorStore` (production).
- **`PgVectorStore`** uses **pgvector** in the Postgres the service already
  has (no new infra): creates the table on demand, searches with the cosine
  distance operator `<=>`. Needs `[genai-rag]` + `CREATE EXTENSION vector`.

```python
from tempest_fastapi_sdk.genai.rag import PgVectorStore

store = PgVectorStore(db, dim=384)          # db = AsyncDatabaseManager
rag = Retriever(embedder, store)
```

`rag.search(query, top_k=)` returns the `Chunk`s with a `score` (similarity);
`rag.retrieve(...)` builds the context for you. Need Qdrant/Weaviate later?
Implement `VectorStore` (2 methods) and inject it — `Retriever` doesn't
change.

## Ergonomics: typed config, router and Redis cache

### Typed `GenerationConfig`

Instead of scattering `**kwargs` (`max_new_tokens=...`, `temperature=...`)
across every call, build a validated, reusable `GenerationConfig` and
pass it via `config=`:

```python
from tempest_fastapi_sdk.genai import GenerationConfig, TextGenerator

gen = TextGenerator("Qwen/Qwen2.5-7B-Instruct", quantization="int4")
config = GenerationConfig(max_new_tokens=512, temperature=0.2, top_p=0.9)

await gen.generate("Explain PIX in one sentence.", config=config)
await gen.chat([{"role": "user", "content": "Hi"}], config=config)
```

Only the set fields layer over the defaults; explicit `**kwargs` still
win over the config (`gen.generate(prompt, config=config,
temperature=0.9)` uses `0.9`).

### `make_genai_router` — ready endpoints

Inject the objects you have loaded and the router mounts **only** the
matching endpoints:

```python
from fastapi import FastAPI
from tempest_fastapi_sdk.genai import Embedder, TextGenerator, make_genai_router

app = FastAPI()
app.include_router(
    make_genai_router(
        text_generator=TextGenerator("Qwen/Qwen2.5-7B-Instruct"),
        embedder=Embedder("sentence-transformers/all-MiniLM-L6-v2"),
    )
)
```

| Object | Endpoints |
| --- | --- |
| `text_generator` | `POST /generate`, `POST /generate/stream` (token-by-token SSE), `POST /chat` |
| `embedder` | `POST /embed` |
| `retriever` | `POST /rag` (query → context block) |
| `speech_to_text` | `POST /transcribe` (audio upload) |
| `text_to_speech` | `POST /tts` (returns `audio/wav`) |

!!! tip "Streaming"
    `/generate/stream` returns `text/event-stream`: each token becomes an
    SSE event, ending with a `done` event. It reuses the SDK's
    `sse_response` — a client with `EventSource` receives tokens live.

### `RedisEmbeddingCache` — cache shared across workers

`Embedder` accepts a synchronous cache (`InMemoryEmbeddingCache`) **or**
an async one. Swap in `RedisEmbeddingCache` to share vectors across
processes with no call-site change:

```python
from redis.asyncio import Redis
from tempest_fastapi_sdk.genai import Embedder, RedisEmbeddingCache

cache = RedisEmbeddingCache(Redis.from_url("redis://localhost"), ttl_seconds=86400)
embedder = Embedder("sentence-transformers/all-MiniLM-L6-v2", cache=cache)

await embedder.embed(["text"])  # first call computes; other workers reuse it
```

`Embedder` awaits `get`/`set` when the cache is async and calls them
directly when it is sync — the same code serves both.

## Recap

- **`GenerationConfig`** — typed, reusable generation parameters instead
  of `**kwargs`.
- **`make_genai_router`** — mounts only the endpoints of the injected
  objects; `/generate/stream` streams tokens over SSE.
- **`RedisEmbeddingCache`** — a shared vector cache; `Embedder` accepts a
  sync or async cache at the same call site.
- **`can_run` / `recommend`** — answer whether the host runs the model
  and what to do if not, **before** the download.
- **RAG over a corpus** — `Retriever` + `VectorStore` (`InMemoryVectorStore`
  / `PgVectorStore` pgvector): index chunks once, retrieve top-k by
  similarity.
- **Audio** — `SpeechToText` (faster-whisper) transcribes; `TextToSpeech`
  (Coqui TTS) synthesizes — local voice, end to end.
- **RAG** — `WebSearch`/`SearxngBackend` (search), `ContentExtractor`
  (page bodies), `PdfReader` (PDF → text/chunks) and `build_context`
  (prompt block), all under the `[genai-rag]` extra.
- **`probe_hardware`** — a CPU/RAM/GPU/disk snapshot; degrades without the
  extras.
- **`estimate_model_bytes` / `bytes_per_param`** — the estimation math,
  testable and reusable.
- Everything imports without the `[genai]` extra; install it to actually
  run models (upcoming module slices).
