# ADR 0006: Hand-written provider adapters, and embeddings that stay Google-only

**Status:** accepted

## Context

Every generation path in the kit ran on Gemini. `SCI_RAG_LLM_MODEL` named a
model with no way to say whose model it was, so pointing answers at Claude or
Grok meant forking the kit.

Three things pushed against that. Adopters standardizing on Google Cloud can
reach Claude, Grok, Llama, Mistral, and DeepSeek as Vertex partner models
without a new vendor relationship. The evaluation harness grades answers with
the same model family that wrote them, which flatters the score. And the kit
is a template: an adopter who cannot change the backend has to change the
code.

The interface for this already existed. `LLMClient` has three methods, every
consumer takes it by injection, and it had exactly one factory with a handful
of call sites. The open question was not where the seam goes, but what fills
it.

The obvious alternative was a translation layer such as LiteLLM: one
dependency, roughly a hundred providers, far less code to write.

## Decision

**Hand-written adapters, one module per provider, behind the existing
`LLMClient` ABC.** Three ship: `google`, `anthropic`, and
`openai-compatible`. You select one with a `provider:model` spec, so a bare
model id keeps resolving to `SCI_RAG_LLM_PROVIDER` and configurations written
before this change keep working unedited.

We rejected a translation layer because the provider differences that matter
here are precisely the ones it would smooth over. JSON-mode calls set
`thinking_budget=0` on Gemini for a documented reason: without it, extraction
and judging spend their whole output budget on thought and return empty. The
Claude equivalent is `output_config={"effort": "low"}`. That is a different
knob with different semantics, *not* a translation of the same one. Disabling
thinking on current Claude models can leak reasoning tags into the JSON these
call sites parse. Those models also removed the sampling parameters outright,
so the adapter drops `temperature` rather than forwarding it. Reaching all of
that through a normalizing layer is harder than writing the hundred lines
directly, and writing them keeps the supported set visible in one file, which
is the same trade-off `get_embedder()` already makes.

`openai-compatible` is the third adapter rather than a dedicated OpenAI one
because Vertex serves its non-Google partner models behind an
OpenAI-compatible endpoint instead of native APIs. One adapter therefore
covers Grok, Llama, Mistral, DeepSeek, every future partner model on that
endpoint, OpenAI itself, and a self-hosted vLLM or Ollama server.

**Embeddings stay Google-only.** Anthropic ships no embedding API. On Vertex
the only managed text embeddings are Google's; everything else requires
deploying and paying for a GPU endpoint in Model Garden. More decisively, an
embedder is not a runtime-swappable choice in this system. A migration bakes
`SCI_RAG_EMBEDDING_DIM` into the pgvector column (ADR 0002), and every chunk
stores the version that produced it. Switching therefore means a migration, a
full re-embed, and an index rebuild. A provider flag would advertise a
configuration change for what is really a data migration.

## Consequences

* Roles resolve independently, so a run can extract with cheap Gemini Flash,
  answer with Claude, and judge with a third provider. Cross-provider judging
  is now a config change, and reports name the answering and grading models so
  the choice is auditable rather than merely available.
* Adding a fourth provider means writing an adapter, not registering a plugin.
  The capability table in [extend.md](../extend.md) is the checklist; the
  `temperature` and effort rows are where a new adapter is most likely to get
  it wrong.
* The SDKs are optional extras (`--extra anthropic`, `--extra openai`) that
  the kit imports lazily, so an offline or Google-only install carries neither.
* `retry_async()` holds the one retry policy, and the kit builds the provider
  SDKs with `max_retries=0`, so there is one backoff to reason about instead of
  three that compound.
* Bring-your-own-key stays meaningful per provider. It has no meaning against
  Vertex, which authenticates with the operator's Google credentials, so that
  combination raises rather than silently ignoring the caller's key.
