tdfs4ds.agent.structured_output¶
Choosing how structured (Pydantic-typed) output is requested from a model.
llm.with_structured_output(schema) defaults, in langchain_openai, to
method="json_schema" — OpenAI's Structured Outputs API. Its contract is
that the server guarantees a schema-valid reply via constrained decoding, so
the SDK parses the content verbatim (model_validate_json), with no
tolerance for anything around it.
An OpenAI-compatible gateway accepts response_format but cannot always
honour it. LiteLLM in front of Bedrock/Anthropic degrades it to an instruction
in the prompt, and Claude answers the way it answers when asked for JSON in
prose::
```json
{ "intent": ... }
```
...with high confidence.
model_validate_json then fails at line 1, column 1 — a backtick — and the
whole call raises. The model was right; only the wrapper is unparseable.
It is the GATEWAY, not the model. This was first diagnosed as a per-model problem, on a report that Mistral worked where Anthropic failed. The raw content proves otherwise — both come back fenced, from the same LiteLLM endpoint::
anthropic : '```json\n{\n "intent": ...t with high confidence.'
ministral : '```json\n{\n "intent": ...ull\n }\n ]\n}\n```'
So a model-name heuristic would fix one deployment and miss the next. The deciding factor is whether the endpoint enforces the schema, and nothing in the configuration states that — the only way to know is to ask it and see.
Hence: try, then fall back. The default path is attempted unchanged; if it
raises, the call is retried with function_calling, where the structure
travels as tool-call arguments that a relaying gateway passes through intact.
This is backward compatible in the strong sense — the fallback is reachable
only after a call that already failed, so a deployment that works today
issues a byte-identical request and never reaches it.
Blanket-switching everyone to function_calling was rejected: it changes the
wire payload for every non-OpenAI deployment, and payload shape is demonstrably
gateway-sensitive here — v0.3.1.145 moved openai-compatible onto
response_format precisely because vLLM-style guided_json was returning
422 from a LiteLLM gateway. That would trade one deployment's crash for
another's regression.
The working method is remembered per process once learned, so the double request is paid once rather than on every classification — this sits on an interactive path.
include_raw=True was investigated as a gentler fix and does not work.
In BaseChatOpenAI.with_structured_output the response format is bound to the
model itself and the wrapper is RunnableMap(raw=llm) | parser_with_fallback:
the LLM runs first, and the strict parse happens inside _generate, so the
exception is raised before the fallback parser is ever reached.
The explicitly configured method, or None to auto-detect.
tdfs4ds.STRUCTURED_OUTPUT_METHOD pins the behaviour for a deployment
whose endpoint is known: it skips the probe-and-fall-back entirely, so a
gateway known not to enforce schemas never pays the failed first attempt.
Record method as the one this process should use from now on.
The probe below and the runtime fallback discover the same fact, but only
the fallback used to record it — so check_configuration() could
establish that function_calling works and the very next intent
classification would still pay for the failed json_schema attempt
first. This is the setter that closes that gap.
Not a configuration change: it sets exactly what a first successful
fallback would have set, lives for one process, and is cleared by
:func:reset_learned_method. Pinning it durably is
tdfs4ds.STRUCTURED_OUTPUT_METHOD.
An unknown method is refused rather than stored, because a stored bad value
would be passed straight to with_structured_output on every later call.
llm.with_structured_output(schema), resilient to a gateway that does
not enforce the schema it was asked for.
The single seam every structured-output call site goes through — one in the classifier, one in EXPLAIN, seven in the query optimizer, all of which fail identically on a relaying gateway. A new call site added directly against the model would silently miss this (there is a test that forbids it).
An explicit method= from the caller always wins, and a configured
STRUCTURED_OUTPUT_METHOD short-circuits the probe.
Ask the endpoint for a tiny typed answer and report what came back.
Answering a prompt proves nothing about structured output — the failure this exists for happens only on the typed path, and it takes down the consumer agent's very first step (intent classification) plus all seven query-optimizer call sites, while a plain hello-world check stays green. Same reasoning as the MCP check's second stage: listing tools proves nothing about calling them.
Returns {"default_ok", "fallback_ok", "detail"}. fallback_ok is
None when the default worked and the fallback was never needed;
default_ok is None when the probe could not run at all, which is NOT a
capability verdict — see below.
Wrap fallback so a success records which method actually worked.