LLM & embedding models¶

tdfs4ds decouples three model concerns:

Setting block	Used by
`INSTRUCT_MODEL_*`	Process documentation, dataset documentation, intent classification, answer synthesis, query optimizer
`EMBEDDING_MODEL_*`	Consumer agent vector index — client-side embeddings (Chroma backend; default for the Teradata backend)
`CHROMA_*`	Where the Chroma vector store persists
`VECTOR_STORE_BACKEND`	`'chroma'` (default) or `'teradata'` — selects where the vector index lives
`TERADATA_VECTOR_STORE_*`	Naming for the Teradata Enterprise Vector Store collections
`TERADATA_AI_*`	Opt-in in-database embeddings via `teradatagenai.TeradataAI` (Teradata backend only)

If EMBEDDING_MODEL_* is not set, it falls back to INSTRUCT_MODEL_*.

Supported providers¶

Provider	`INSTRUCT_MODEL_PROVIDER`	Notes
OpenAI	`'openai'`	`INSTRUCT_MODEL_URL` optional
Azure OpenAI	`'azure'`	`INSTRUCT_MODEL_URL` required (deployment endpoint)
vLLM / TGI	`'vllm'`	OpenAI-compatible HTTP endpoint
AWS Bedrock	`'bedrock'` (alias `'aws'`)	No URL / API key — uses boto3 default credentials (Converse API). With a `.../openai/v1` URL it uses Bedrock's OpenAI-compatible endpoint instead, see below

The same provider keys apply to embeddings via EMBEDDING_MODEL_PROVIDER.

OpenAI-compatible example¶

tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'openai'
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'gpt-4o'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = os.getenv('OPENAI_API_KEY')

tdfs4ds.EMBEDDING_MODEL_PROVIDER = 'openai'
tdfs4ds.EMBEDDING_MODEL_MODEL    = 'text-embedding-3-small'
tdfs4ds.EMBEDDING_MODEL_DIM      = 1536

LiteLLM: proxy or SDK?¶

The two LiteLLM topologies need different providers — picking the wrong one is the most common configuration mistake here.

Your setup	Provider	Needs
A LiteLLM proxy server (an HTTP endpoint, typically `:4000`)	`'openai-compatible'` (or `'vllm'`)	nothing extra — the proxy speaks the OpenAI API, so `ChatOpenAI` talks to it natively
The LiteLLM Python SDK, in-process, no proxy	`'litellm'`	`pip install langchain-community litellm` — neither is a tdfs4ds dependency

If you are pointing INSTRUCT_MODEL_URL at a URL, you have a proxy — use 'openai-compatible':

tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'openai-compatible'
tdfs4ds.INSTRUCT_MODEL_URL      = 'http://litellm.example.svc.cluster.local:4000'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = os.getenv('LITELLM_VIRTUAL_KEY')
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'us-anthropic-claude-sonnet-4-5-...'   # a proxy model name

The 'litellm' provider routes through langchain_community.chat_models.ChatLiteLLM, which calls the litellm library directly with provider-prefixed model strings ('bedrock/amazon.nova-pro-v1:0', 'ollama/llama3', …). Recent langchain-community releases have also dropped ChatLiteLLM, so 'openai-compatible' is the more durable choice whenever a proxy exists.

JSON constraints: which syntax does your endpoint accept?¶

There is no single industry parameter for constrained decoding. Each serving stack named the same idea differently, and a gateway accepts only its own:

Syntax	Serving stack
`response_format={"type": "json_schema"}`	OpenAI, Azure, Mistral, and gateways speaking their API
`tools=[...]` + `tool_choice`	anything with tool calling
`response_format={"type": "json_object"}`	OpenAI "JSON mode", widely copied
`extra_body.guided_json`	vLLM / SGLang
`extra_body.nvext.guided_json`	NVIDIA NIM
`extra_body.grammar`	HuggingFace TGI
`extra_body.format`	Ollama

tdfs4ds asks for typed JSON along two independent paths, and they are controlled by different settings — which is why a problem with one is invisible from the other:

Path	Used by	Mechanism	Setting
langchain `with_structured_output`	consumer agent, query optimizer	`json_schema` -> `function_calling` -> `json_mode`	`STRUCTURED_OUTPUT_METHOD`
provider-branched wire parameter	documentation generator	one row of the table above	`INSTRUCT_MODEL_PROVIDER`

The second one matters more than it looks. If the backend rejects your provider's parameter, the documentation path retries once unconstrained and carries on — producing plausible output at strictly lower reliability, with nothing reporting it.

Finding out¶

check_configuration() probes both paths, and prints one row for them:

[ OK ] JSON constraints   agent: structured output enforced end to end;
                          documentation: response_format enforced - the reply is bare JSON

If either reports trouble, it then runs a nine-request wire-level sweep that tries every syntax above and identifies the stack behind the gateway from the accept/reject pattern. A healthy endpoint never pays for that sweep. You can run it on its own at any time:

python -m tdfs4ds.agent.json_constraints

report = tdfs4ds.probe_json_constraints()
report['engine']       # e.g. 'vLLM / SGLang'
report['recommend']    # e.g. {'STRUCTURED_OUTPUT_METHOD': 'function_calling'}

The common case: a relaying gateway¶

LiteLLM in front of Bedrock or Mistral accepts response_format and does not enforce it. The model answers with markdown-fenced JSON, the strict parser rejects it at line 1 column 1, and the consumer agent dies on its very first step (intent classification). tdfs4ds recovers automatically by falling back to function_calling, but that costs a failed request on every typed call until the method is pinned:

tdfs4ds.STRUCTURED_OUTPUT_METHOD = 'function_calling'

Recording the answer¶

check_configuration(pin=True) applies what it discovered to the session and writes it into your .env via patch_dotenv, stamped with the endpoint and date it was probed against — so it is not re-derived every session, and a pin that stops being true is visible rather than silent:

TDFS4DS_STRUCTURED_OUTPUT_METHOD=function_calling  # pinned by check_configuration on 2026-09-11, probed against https://gw.example/v1

A pinned method travels into a deployment

STRUCTURED_OUTPUT_METHOD is ordinary config to the AgentOps packager, so a pinned value ends up in a deployed pod's config.yaml. That is right when the pod uses the same gateway and wrong when it does not — which is what the date stamp is for.

TLS: internal certificate authorities¶

A gateway on an internal host often presents a certificate the system CA store does not trust. Two settings cover it, for the instruct model only — embedding clients are never affected:

# Preferred: verification stays ON, anchored to your internal CA
tdfs4ds.INSTRUCT_MODEL_CA_BUNDLE  = '/etc/ssl/certs/internal-ca.pem'

# Last resort: disable verification entirely
tdfs4ds.INSTRUCT_MODEL_VERIFY_SSL = False

INSTRUCT_MODEL_VERIFY_SSL=False is the equivalent of ChatOpenAI(..., http_client=httpx.Client(verify=False)). It works, and it is sometimes the only option — but the API key then travels over a connection whose peer was never authenticated, so tdfs4ds logs a warning whenever it is active and check_configuration() reports TLS verification DISABLED in the instruct check. Prefer the CA bundle whenever the certificate can be obtained.

With neither set, no custom HTTP client is built at all — identical to the behaviour before these options existed.

What `check_configuration()` reports¶

A dedicated LLM transport (TLS) check runs before the model is contacted and without sending any credential, so its answer frames everything after it:

[ OK ] LLM transport (TLS)   https://api.example.com/v1 - TLSv1.3 -
                             cert api.example.com, issued by Internal Root CA -
                             expires Nov 21 01:57:54 2026 GMT (70 days)

It separates failures that look identical in a traceback but have different fixes:

Reported	Fix
untrusted issuer	`INSTRUCT_MODEL_CA_BUNDLE` — and if the system store does accept the certificate, it says so, because then the bundle is simply the wrong one
hostname mismatch	the host in `INSTRUCT_MODEL_URL`; the certificate's real name is reported
expired	renewal on the server — nothing here can work around it
expires in under 30 days	nothing yet, which is the point of saying so now
plain `http://`	acceptable in-cluster; the API key is unencrypted

When verification is off it still reads the certificate (without validating it) so you can see which CA to obtain and turn verification back on.

Only the instruct model

These settings reach the OpenAI-family clients only. Bedrock and Vertex secure their transport through their own SDKs, and the embedding client is deliberately never affected by the instruct model's TLS posture — a proxy fallback cannot weaken security, but a verification fallback could. If an embedding endpoint sits behind an internal CA, install that CA in the system trust store, point EMBEDDING_MODEL_URL at a publicly trusted route, or use EMBEDDING_MODEL_PROXY.

Error code: 302 and 'str' object has no attribute 'model_dump'

Two symptoms, one cause, and neither is a TLS problem despite appearing in the same environments: something in front of the gateway — an ingress, an SSO login page — answered instead of the model.

They differ only in whether the redirect was followed. httpx rewrites a redirected POST into a GET, so following it lands on the target (an HTML page: success status, non-JSON body, which the OpenAI SDK returns as raw text for LangChain to mis-parse), while not following it surfaces the bare 302.

check_configuration() reports both with the same hint and adds a probe showing where the URL actually leads — GET … -> 302 -> https://…. Check that INSTRUCT_MODEL_URL's path is right (the SDK appends /chat/completions), and if the target is a login page, look for a route that bypasses the ingress, such as a cluster-local service URL.

vLLM behind LiteLLM example¶

tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'vllm'
tdfs4ds.INSTRUCT_MODEL_URL      = 'https://api.example.com/v1'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = os.getenv('FS_API_KEY')
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'mistral-7b-instruct'

tdfs4ds.EMBEDDING_MODEL_PROVIDER = 'vllm'
tdfs4ds.EMBEDDING_MODEL_URL      = 'https://api.example.com/v1/e5'
tdfs4ds.EMBEDDING_MODEL_API_KEY  = os.getenv('FS_API_KEY')
tdfs4ds.EMBEDDING_MODEL_MODEL    = 'bge-m3'
tdfs4ds.EMBEDDING_MODEL_DIM      = 1024

AWS Bedrock example¶

tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'bedrock'
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'anthropic.claude-3-sonnet-20240229-v1:0'
# Authentication: boto3 default chain (env vars, ~/.aws/credentials, IAM role)

Bedrock's OpenAI-compatible endpoint¶

Give the provider (aws or bedrock) the endpoint URL and tdfs4ds speaks the OpenAI protocol to it. Two ways to authenticate; neither needs anything else in the file:

# A) the environment's role - nothing secret on disk
TDFS4DS_INSTRUCT_MODEL_PROVIDER=aws
TDFS4DS_INSTRUCT_MODEL_URL=https://bedrock-runtime.eu-central-1.amazonaws.com/openai/v1
TDFS4DS_INSTRUCT_MODEL_MODEL=mistral.devstral-2-123b
TDFS4DS_INSTRUCT_MODEL_CLIENT_MODE=ambient

# B) a Bedrock API key, sent as the bearer token
TDFS4DS_INSTRUCT_MODEL_PROVIDER=aws
TDFS4DS_INSTRUCT_MODEL_URL=https://bedrock-runtime.eu-central-1.amazonaws.com/openai/v1
TDFS4DS_INSTRUCT_MODEL_MODEL=mistral.devstral-2-123b
TDFS4DS_INSTRUCT_MODEL_API_KEY=...

CLIENT_MODE=ambient (or simply no key) signs every request with the AWS identity of the environment - the standard credential chain: an instance or task role, SSO, a profile, the environment variables. The region comes from the URL.
The route is chosen by the URL alone: a bedrock configuration without a .../openai/v1 URL is the Converse client it has always been.
A role needs no key, so check_configuration() no longer treats a missing one as a gap for bedrock / aws / gcp; the report says how the call authenticated (never a value).
An AccessDenied is a permission problem, not a missing key: allow bedrock:InvokeModel on the model (or on its inference profile, eu. / us. / apac. prefix) for the role.
Embeddings still use the Converse route; set TDFS4DS_EMBEDDING_MODEL_MODEL to a Bedrock embedding model (for example amazon.titan-embed-text-v2:0).

Listing available models¶

When the endpoint exposes /v1/models (OpenAI-compatible), discover models without leaving Python:

from tdfs4ds.agent import list_instruct_models, list_embedding_models

list_instruct_models()
list_embedding_models(sub_paths=['e5', 'code'])

Vector store backend¶

The consumer agent's vector index is backend-agnostic. Pick where it lives via VECTOR_STORE_BACKEND:

tdfs4ds.VECTOR_STORE_BACKEND = 'chroma'     # default — local or remote ChromaDB
# or
tdfs4ds.VECTOR_STORE_BACKEND = 'teradata'   # Teradata Enterprise Vector Store

build_vector_index() and search_vector_index() dispatch on this setting; all callers (chatbot, search/definition skills) are unchanged.

Chroma backend¶

tdfs4ds.CHROMA_MODE = 'local'                # persist to disk
tdfs4ds.CHROMA_PATH = './tdfs4ds_chroma'

# OR

tdfs4ds.CHROMA_MODE = 'server'               # connect to a Chroma server
tdfs4ds.CHROMA_HOST = 'localhost'
tdfs4ds.CHROMA_PORT = 8000

The Chroma backend holds three collections (objects, sections, columns) populated by build_vector_index() from the business dictionary. Embeddings come from the EMBEDDING_MODEL_* (or INSTRUCT_MODEL_*) configuration.

Teradata backend¶

The Teradata Enterprise Vector Store persists the same three collections directly in Vantage as named vector stores:

tdfs4ds.VECTOR_STORE_BACKEND         = 'teradata'
tdfs4ds.TERADATA_VECTOR_STORE_PREFIX = 'tdfs4ds'   # → tdfs4ds_objects / _sections / _columns

By default the Teradata backend reuses the LangChain EMBEDDING_MODEL_* setup — your existing vLLM / TEI / LiteLLM / OpenAI configuration keeps working unchanged because TeradataVectorStore accepts a LangChain Embeddings object at its embedding= argument.

To opt into in-database embeddings via teradatagenai.TeradataAI, set TERADATA_AI_API_TYPE:

# Example: AWS Bedrock embeddings produced inside Vantage
tdfs4ds.TERADATA_AI_API_TYPE   = 'aws'
tdfs4ds.TERADATA_AI_MODEL_NAME = 'amazon.titan-embed-text-v2:0'
tdfs4ds.TERADATA_AI_REGION     = 'us-west-2'
tdfs4ds.TERADATA_AI_ACCESS_KEY = os.getenv('AWS_ACCESS_KEY')  # or via .env

# Example: NIM / OpenAI-compatible in-database client
tdfs4ds.TERADATA_AI_API_TYPE   = 'nim'
tdfs4ds.TERADATA_AI_MODEL_NAME = 'nvidia/nv-embedqa-e5-v5'
tdfs4ds.TERADATA_AI_API_BASE   = 'https://nim.example.com/v1'

Supported TERADATA_AI_API_TYPE values: aws, azure, gcp, nim, hugging_face, onnx. azure / gcp / hugging_face rely on the standard provider env vars (e.g. AZURE_OPENAI_*, GOOGLE_APPLICATION_CREDENTIALS).

Incremental updates

The Enterprise Vector Store API does not yet guarantee per-id upsert, so a non-force_rebuild build will try add_documents and fall back to a destroy + rebuild if it is not available.

Performance knobs¶

tdfs4ds.INSTRUCT_MODEL_TIMEOUT     = 60       # seconds per LLM call
tdfs4ds.INSTRUCT_MODEL_MAX_RETRIES = 3        # transient-error retries

For very small-context models (≤ 16 k tokens), cap the candidate generation length so the query optimizer does not exhaust its output budget on verbose reasoning text:

tdfs4ds.QUERY_OPTIMIZER_MAX_CANDIDATE_TOKENS = 4096