AI capabilities — overview

tdfs4ds ships with four LLM-powered features layered on top of the feature store. They are independent — you can turn any one of them on without enabling the others.

Process documentation

LLM business-logic description + per-feature column docs + EXPLAIN score 1–5.

Dataset documentation

Lineage-aware dataset docs: bottom-up walk through every process and source.

Business dictionary

Three temporal tables, no LLM required — feed the agent with structured business descriptions.

Consumer agent

Conversational chatbot over the dictionary: SEARCH, DEFINITION, USAGE, FRESHNESS, SUMMARY, LINEAGE, EXPLAIN, ALGORITHM (MCP).

Query optimizer

9-step pipeline: simplify → EXPLAIN → candidates → compare → FilterManager proposal.

EXPLAIN scoring

1–5 USER / GLOBAL scores plus deterministic counters parsed from the raw plan.

Common configuration

All four features share the same LLM config:

tdfs4ds.INSTRUCT_MODEL_PROVIDER = 'openai'       # 'openai' | 'azure' | 'vllm' | 'bedrock'
tdfs4ds.INSTRUCT_MODEL_MODEL    = 'gpt-4o'
tdfs4ds.INSTRUCT_MODEL_API_KEY  = os.getenv('OPENAI_API_KEY')

See Configuration / LLM & embedding models for full provider details.

Lazy loading

tdfs4ds.genai and tdfs4ds.agent import LangChain, LangGraph, Chroma, and Gradio — together a few seconds at first access. The package defers both behind a module __getattr__, so plain feature-store use stays fast.

import tdfs4ds                           # ~ 400 ms
tdfs4ds.genai.document_process(...)     # langchain* loaded here
tdfs4ds.agent.launch_chatbot(...)       # langgraph + gradio loaded here

Serve the agents over HTTP

The consumer agent and the query optimizer can be exposed as an OpenAI-compatible API (two models, one per agent) and shipped in Docker — see Deployment / OpenAI-compatible API server.