Consumer Agent Field Guide

What we cover

Six sections. The first three describe what the agent knows and how it knows it; the last three describe how we prove it.

A read layer over the catalogs, not a chatbot bolted on top

The consumer agent answers questions about the feature store using the store's own catalogs, business dictionary and lineage graph. Every object name it states is checked against the catalog before it reaches the user.

# tdfs4ds/agent/graph.py
from tdfs4ds.agent import consumer_agent

answer = consumer_agent("What is provider_z_score?", thread_id="demo")

# the shape evaluation uses: answer + trace + accumulated context graph
answer, trace, ctx = consumer_agent(
    "and is it up to date?",
    thread_id="demo",
    return_trace=True,
    return_context=True,
    mcp_enabled=False,
)

Bilingual: French and English questions route through the same classifier.

Where the scope stops

Knowing what the agent refuses to do is what makes an evaluation set honest — half of these boundaries only show up as a scenario that should fail.

One LangGraph pass per turn

Three nodes resolve what is being asked about before any skill runs. That ordering is why a pronoun in turn 2 still lands on the right feature.

classify

Structured-output intent classification over a Pydantic model built from the installed skills. Also extracts object_name, entity_name, a confidence, and candidate interpretations.

detect_domain

Resolves which DATA_DOMAIN owns the named object, and keeps it sticky across turns so an ambiguous name does not silently flip project.

recognize_entity

Pins the entity when the question named one, so a feature that exists under two entities is not sent to a clarification prompt unnecessarily.

route

Conditional edge. A pending clarification short-circuits straight to synthesize; a compound question goes to skill_compound, which runs each sub-intent in order; EXPLAIN passes through pre_skill_explain first.

synthesize

Most skills return their final answer directly — only SEARCH pays for a second LLM pass. That is a latency decision, not an oversight.

Checkpointing: MemorySaver, keyed on thread_id. The compiled graph is a module singleton — rebuilding it per call would discard all cross-turn state.

The skill roster

Fourteen skills ship as ca-*/SKILL.md files; two more are injected in code; two require MCP. The classifier's vocabulary is assembled from whatever is installed — there is no hard-coded intent list.

The certainty boundary

The agent's honesty rests on one distinction, and your evaluation set has to encode it or the judge will reward confident guessing.

What survives between turns

Multi-turn evaluation is only meaningful because these carry forward. Each one is a thing a follow-up question can break.

resolved_data_domain

The active project. Sticky — an ambiguous feature name does not flip it; only an explicit object that resolves elsewhere does.

resolved_feature_triplet

feature name, entity, process id, view name for the last single feature resolved.

resolved_entity_name

The entity in focus for this turn. Cleared when the turn names no entity; the sticky cross-turn one lives in the context graph.

resolved_feature_list

The group currently under discussion — what "are these features up to date?" refers to.

resolved_column_sources

Column to source-table map written after an EXPLAIN turn, so "what is that column?" can drill down.

accumulated_context_graph

The cross-turn knowledge graph: features, datasets, objects, columns and their edges. It powers the fast paths, the visualisation panel — and it is the second grounding source the evaluation judge reads.

Three ways to put it in front of people

from tdfs4ds.agent import consumer_agent

consumer_agent(
  "Which datasets expose total_amount?",
  thread_id="notebook",
)

from tdfs4ds.agent import (
    build_vector_index,
    launch_chatbot_with_index,
)

launch_chatbot_with_index(
    port=7860,
    force_rebuild=False,
)

python -m tdfs4ds.serve

report = tdfs4ds.check_configuration()   # names the setting to fix

Each skill has a hard prerequisite in the store

A skill whose prerequisite is missing does not degrade gracefully into a worse answer — it degrades into no answer. Check this table before blaming the model.

The functional log is what makes freshness answerable

There is no "last updated" column on a feature. Freshness is reconstructed from the execution history of the process that produces it — so the log is not telemetry, it is the answer.

# tdfs4ds/process_store/process_followup.py -- FS_FOLLOW_UP
RUN_ID         VARCHAR(255)   # one run
PROCESS_TYPE   VARCHAR(255)
PROCESS_ID     VARCHAR(255)   # joins back to the process catalog
START_DATETIME TIMESTAMP(6) WITH TIME ZONE
END_DATETIME   TIMESTAMP(6) WITH TIME ZONE
VALIDTIME_DATE TIMESTAMP(6) WITH TIME ZONE   # the as-of date the run wrote
APPLIED_FILTER JSON(2000)     # which FilterManager segment
STATUS         VARCHAR(2048)  # 'COMPLETED' / 'FAILED,...' / 'RUNNING'

The resolution chain is feature catalog → PROCESS_ID → process catalog → follow-up rows. Read it directly with:

tdfs4ds.follow_up_report(process_id="550e8400-...").to_pandas()

Temporal catalogs, and temporal data

Time travel needs both halves. Temporal catalogs alone tell you what was registered on a date; temporal feature data tells you what the values were. Freshness questions sit between the two.

# None = current state
tdfs4ds.FEATURE_STORE_TIME = None

# a timestamp = point-in-time read
tdfs4ds.FEATURE_STORE_TIME = "2024-01-01 00:00:00"

The business dictionary is the agent's vocabulary

Three valid-time tables, three levels, designed so retrieval never needs chunking: rows are already the right grain.

# populate, then index -- in that order
tdfs4ds.rebuild_documentation(force_update=False, all_data_domains=True)

from tdfs4ds.agent import build_vector_index
build_vector_index(force_rebuild=False)

Backends: Chroma (local, default) or the Teradata Enterprise Vector Store (VECTOR_STORE_BACKEND = 'teradata').

Drilling from a feature down to the Python that produced it

When a feature is produced by tdstone2 — in-database Python running in a Script Table Operator — EXPLAIN can only show you the SQL wrapper. ALGORITHM walks the chain into the model itself.

algorithm

e.g. IsolationForest, named from the model metadata

hyperparameters

parsed from the ARGUMENTS JSON

model_inputs

column_names_X — and their business descriptions, resolved from the dictionary so inputs are explained, not just listed

packages

extracted from the source, only when the question asks about code

n_trained_partitions

one model per partition — how many were actually trained

sto_invocation

the real FROM Script(...) block, quoted verbatim

Registering tdstone2 output as governed features: tdfs4ds.upload_tdstone2_scores(model).

A skill is a Markdown file with frontmatter

The file is the source of truth for the intent name, the classifier's description of it, and the few-shot examples. Editing the prompt means editing this file.

# tdfs4ds/skills/ca-freshness/SKILL.md
---
name: "Consumer Agent: Freshness"
intent: "FRESHNESS"
description: "user asks whether a feature/dataset is up to date or reliable"
argument-hint: "[feature or dataset name]"
allowed-tools: "Read, Grep, Glob"
---

Check whether a feature or dataset is current and reliable by inspecting
the execution history of its registered process.

## Queries      # the resolution chain, documented
## Returns      # every key the skill puts on its result dict
## Error cases  # every `error` value a caller must handle
## Notes        # the RUNNING trap, per-segment freshness
## Classifier Examples
- "Cette feature est-elle a jour ?" -> FRESHNESS

Taking the skills somewhere else

The skills are ordinary files shipped inside the package. Nothing about them is specific to our agent runtime.

Export them

tdfs4ds.list_skills()
# ['ca-algorithm', 'ca-build-dataset', ..., 'fs-upload', 'td-explain']

tdfs4ds.skill_path("ca-freshness")          # path to the SKILL.md

tdfs4ds.install_skills()                    # -> ~/.claude/skills/
tdfs4ds.install_skills("/srv/my-agent/skills",
                       skills=["ca-freshness", "ca-explain", "ca-lineage"],
                       overwrite=False)

# or from a shell, after pip install:
#   tdfs4ds-install-skills

Extend the agent with your own

# point the agent at a folder of extra skills
export TDFS4DS_SKILLS_FOLDER=/srv/my-agent/skills

# each subdirectory needs a SKILL.md with name/intent/description,
# and optionally a skill.py defining a single entry point:
#     def node(state: dict) -> dict: ...

Inspect what the agent will actually load: tdfs4ds.consumer_agent_skill_catalog() — bundled skills plus plugin ones, the latter tagged plugin: "true".

What we are actually protecting against

The agent has no test suite in the usual sense, because its output is prose. What it has is a harness that scores prose against catalog facts — and it exists because four things move underneath it.

Five dimensions, scored 1 to 5, per turn

Each turn gets all five. Scenario and category averages are rolled up from them; nothing is collapsed into a single number before you see the breakdown.

Reported alongside them: intent_match_rate — a deterministic measurement, not a judged one.

The judge reads two sources of truth, not one

This is the single most important thing to understand before you write expected_facts by hand — because it determines what a missing fact means.

Deterministic signals computed before the judge is asked

• Intent match — compared, then stated in the prompt. A mismatch caps robustness at 2 regardless of what the judge says.
• Fact hits and misses — which expected names literally appear in the answer, case-insensitively, listed for the judge.
• Counts — feature_count and the per-entity breakdown are catalog-derived. A correct count registers as a hit, so the judge cannot flag a right number as fabricated.
• The faithfulness cap fires only on a genuine fabrication signal: the focal fact absent and no verified context at all. It fails open.

Anatomy of a single-turn scenario

One scenario, one thread, one turn. This is the shape generate_scenarios emits and the shape you hand-write.

{
  "id": "S01",
  "category": "standard",
  "description": "Single-turn DEFINITION",
  "multi_turn": false,
  "skills": ["DEFINITION"],
  "uses_pronouns": false,
  "thread_id": "eval_definition_0",
  "turns": [
    {
      "turn": 1,
      "user": "What is provider_z_score?",
      "expected_intent": "DEFINITION",
      "expected_skill": "skill_definition",
      "expected_facts": {
        "domain": "claims_anomaly_detection",
        "entity_name": "provider_id",
        "feature_name": "provider_z_score",
        "process_id": "550e8400-e29b-41d4-a716-446655440000",
        "view_name": "FS_PROVIDER_FLAGS",
        "sibling_features": ["provider_z_score", "pharmacy_z_score"]
      }
    }
  ]
}

thread_id

Isolates the conversation. Give every scenario its own, or turn 1 of one scenario will be visible to turn 1 of the next.

expected_facts

Fills itself from the catalog when generated. When you write one by hand, the keys the judge checks literally are feature_name, entity_name, domain, view_name, object_name, column_name, plus the lists feature_names and lineage_neighbors.

expected_intent

Drives the deterministic intent-match signal — get this wrong and you have written a robustness failure, not a test.

Anatomy of a multi-turn scenario

Turn 2 deliberately omits the feature name. It is a test of memory, and it is the only way to catch a regression in domain stickiness or triplet resolution.

{
  "id": "M01",
  "category": "multi_turn",
  "description": "Multi-turn DEFINITION -> FRESHNESS (pronoun)",
  "multi_turn": true,
  "skills": ["DEFINITION", "FRESHNESS"],
  "uses_pronouns": true,
  "thread_id": "eval_mt_definition",
  "turns": [
    { "turn": 1,
      "user": "What is provider_z_score?",
      "expected_intent": "DEFINITION",
      "expected_skill": "skill_definition",
      "expected_facts": { "...": "as before" } },

    { "turn": 2,
      "user": "and is it up to date?",          // no name -- on purpose
      "expected_intent": "FRESHNESS",
      "expected_skill": "skill_freshness",
      "expected_facts": { "...": "same feature, resolved from memory" } }
  ]
}

Both turns run against the same thread_id in order, so turn 2 sees whatever turn 1 left in the checkpointer. The four follow-up pairs generated automatically are:

Set use_pronouns=False to regenerate the same pairs with the name restated — run both and the difference isolates memory from retrieval.

Fact-first generation: harvest the answer, then write the question

Skill-first templating guarantees that every skill is exercised. It does not guarantee breadth — it will ask about the same two features forever. Fact-first fixes that by starting from the catalog instead of the template.

The two sets are unioned and deduplicated by question text — skill-first guarantees per-skill coverage, fact-first guarantees grounded breadth.

tdfs4ds.generate_scenarios(fact_first=True, facts_per_kind=2)
# ids prefixed FO.. for objects, FC.. for columns

Step 0 — the prerequisite that is not negotiable

Scenario generation reads the live catalogs. Against an empty or undocumented store it does not fail — it produces a small, plausible-looking scenario file that measures nothing. This is the failure to guard against, because it is silent.

print(tdfs4ds.feature_catalog(all_data_domains=True).shape)
print(tdfs4ds.process_catalog(all_data_domains=True).shape)
print(tdfs4ds.dataset_catalog().shape)
print(tdfs4ds.follow_up_report().to_pandas().shape)

Step 1 — connect, and configure the model that will judge

The judge is the same instruct model the agent uses. It is called through the agent's own seam, so if the agent can talk, the judge can score.

import teradataml as tdml
import tdfs4ds

tdml.create_context(host=HOST, username=USER, password=PWD)

tdfs4ds.SCHEMA      = "MY_FEATURE_STORE"
tdfs4ds.DATA_DOMAIN = "claims_anomaly_detection"

tdfs4ds.connect(database=tdfs4ds.SCHEMA, create_if_missing=False)

tdfs4ds.INSTRUCT_MODEL_PROVIDER = "openai-compatible"
tdfs4ds.INSTRUCT_MODEL_URL      = "https://gateway.internal/v1"
tdfs4ds.INSTRUCT_MODEL_API_KEY  = "..."
tdfs4ds.INSTRUCT_MODEL_MODEL    = "ministral-8b"

report = tdfs4ds.check_configuration()
# every check carries a hint naming the setting to fix

Step 2 — generate the grounded baseline

One call. It reads the live catalogs, fills the templates with real feature, entity and domain names, and attaches the ground truth each skill needs.

scenarios = tdfs4ds.generate_scenarios(
    skills=["DEFINITION", "EXPLAIN", "FRESHNESS", "SUMMARY",
            "DATA_DOMAINS", "FEASIBILITY", "LINEAGE", "USAGE", "DATASET"],
    multi_turn=True,      # also emit the four follow-up pairs
    use_pronouns=True,    # turn 2 omits the name -- tests memory
    per_skill=1,          # single-turn scenarios per skill
    fact_first=True,      # harvest objects/columns from the dictionary
    facts_per_kind=2,     # how many objects, how many columns
)

tdfs4ds.save_scenarios(scenarios, "golden_scenarios.json")

skills

Omit it to cover every templated skill. Pass a list to iterate on one.

categories

["standard"] or ["multi_turn"] to restrict; both by default.

per_skill

Raise it and each skill gets a different feature — the templates walk the catalog rather than repeating one name.

facts_per_kind

Governs breadth over documented objects and columns. This is the knob that grows the set without touching the templates.

Step 3 — read what it actually produced

Do this every time. A generated set is grounded in the catalog, which is not the same as being a good test of anything.

print(scenarios["meta"])
# {'scenario_version': '20260915_104233',
#  'scenario_hash': 'a1b2c3d4e5f6',      <- content fingerprint
#  'generated_at': '20260915_104233',
#  'tdfs4ds_version': '0.3.1.208',
#  'n_scenarios': 17,
#  'domains': ['claims_anomaly_detection'],
#  'fact_first': True}

for s in scenarios["scenarios"]:
    print(f'{s["id"]:5} {s["category"]:11} {len(s["turns"])}t  {s["turns"][0]["user"]}')

What you are looking for

• Placeholder names. A question containing the literal word feature means the feature catalog read came back empty. Stop and go back to Step 0.
• Repetition. The same feature in every scenario means per_skill=1 and a small catalog — raise it, or add hand-written scenarios.
• Empty ground truth. A DATASET scenario with "dataset_names": [] is unanswerable; drop it or register the dataset.
• The domains list. If it holds one domain, you are not testing domain stickiness at all — that needs two.

Keep the hash. Two results files with different scenario_hash values are not comparable, however similar the averages look.

Step 4 — curate: add what generation cannot reach

Templates exist for nine intents. The rest of the roster has no template at all, and those are exactly the intents where a regression is most embarrassing.

Blue: generated for you. Red: you write these, or they are never tested.

hand = {
  "scenarios": [
    {
      "id": "H01",
      "category": "standard",
      "description": "OUT_OF_SCOPE must decline, not improvise",
      "multi_turn": False,
      "skills": ["OUT_OF_SCOPE"],
      "uses_pronouns": False,
      "thread_id": "eval_hand_oos",
      "turns": [{
        "turn": 1,
        "user": "What is the weather in Paris tomorrow?",
        "expected_intent": "OUT_OF_SCOPE",
        "expected_skill": "skill_out_of_scope",
        "expected_facts": {}
      }]
    },
    {
      "id": "H02",
      "category": "standard",
      "description": "FRESHNESS on a segmented process",
      "multi_turn": False,
      "skills": ["FRESHNESS"],
      "uses_pronouns": False,
      "thread_id": "eval_hand_segments",
      "turns": [{
        "turn": 1,
        "user": "Is provider_z_score up to date for every segment?",
        "expected_intent": "FRESHNESS",
        "expected_skill": "skill_freshness",
        "expected_facts": {
          "feature_name": "provider_z_score",
          "entity_name": "provider_id",
          "domain": "claims_anomaly_detection"
        }
      }]
    }
  ],
  "meta": {}
}

merged = tdfs4ds.merge_scenarios(scenarios, hand, mode="append")
tdfs4ds.save_scenarios(merged, "golden_scenarios.json")

mode="append" unions by id, incoming wins on conflict. mode="replace" returns the incoming set untouched.

Step 5 — run the agent and judge every turn

Each scenario runs on its own thread, turns in order. Each turn is judged immediately, with the agent's own context graph passed to the judge.

results = tdfs4ds.run_evaluation(
    merged,
    mcp_enabled=False,     # True to also exercise ALGORITHM / DATA_QUERY
    progress=print,        # emits 'PROGRESS k/N' lines
)

Or the whole pipeline from a file in one call:

out = tdfs4ds.evaluate_agent(
    "golden_scenarios.json",
    out_path="agent_eval_report.html",
    fmt="html",
    results_path="agent_eval_results.json",
    progress=print,
)
print(out["report_path"], out["results_path"])

What lands in the results file

per turn

question, the agent's answer, expected vs actual intent, fact_hits / fact_misses, the five scores, the judge's rationale per metric, the trace and the context graph

per scenario

the five averages plus avg_score

stamped

model and provider, tdfs4ds version, scenario version and hash — so a result can always be tied back to what produced it

Step 6 — report, then triage

html = tdfs4ds.build_report(results, fmt="html")   # or fmt="md"

The report opens with the global averages and the intent-match rate, then breaks down by category and by expected intent, then lists the worst turns, then every turn in full with its trace and judge rationale.

Reading it in the right order

# the eval-inspect skill ships a stdlib reader -- no DB, no LLM
python inspect.py agent_eval_results.json --worst 5
python inspect.py agent_eval_results.json --scenario S17
python inspect.py agent_eval_results.json --intent-mismatch

Regeneration discipline

A golden set is only golden if it stays still. Regenerating it against a changed catalog produces a different test, and comparing across that boundary is how a regression gets explained away.

What this does not prove — and what to do next

Next steps for the team

1. Run Step 0's four checks against the store you intend to evaluate, and fix what is empty.
2. Generate a baseline, read it, and commit it with its hash.
3. Write the six missing-intent scenarios by hand — SEARCH, HELP, RECAP, BUILD_DATASET, OUT_OF_SCOPE, and one ALGORITHM if you run tdstone2 features.
4. Run it, record the numbers, and treat that as the line every prompt change must hold.