tdfs4ds.agent.eval_scenarios¶

tdfs4ds.agent.eval_scenarios ¶

Scenario generation for the agent evaluation, from the store (0.3.1.240).

Questions are chosen from data, not from position: a feature is picked because it fits the skill (documented for DEFINITION / EXPLAIN / SEARCH, produced by a process that ran for FRESHNESS, read by a dataset for USAGE / DATASET, registered under several entities for disambiguation), deterministically for a seed. Each turn carries a fact_spec; its expected_facts are built from it by :func:tdfs4ds.agent.eval_facts.build_facts, and time_context says which valid time the turn is about, following the agent's own date rules (consumer_agent._resolve_as_of).

Time scenarios (category temporal) are built from the store's history: a date between two registrations (so the store at that date differs from now), a date before anything was registered, a date kept by a follow-up question and cleared by "now", freshness and lineage at a past date, a dataset's own data time, and a year alone (which the agent must not take as a date).

logger `module-attribute` ¶

logger = getLogger(__name__)

ALL_SKILLS `module-attribute` ¶

ALL_SKILLS = ['DEFINITION', 'EXPLAIN', 'FRESHNESS', 'SUMMARY', 'DATA_DOMAINS', 'FEASIBILITY', 'LINEAGE', 'USAGE', 'DATASET', 'SEARCH', 'RECAP', 'HELP', 'OUT_OF_SCOPE', 'ALGORITHM', 'BUILD_DATASET']

_Q `module-attribute` ¶

_Q = {'DEFINITION': ['What is {f}?', 'Que signifie {f} ?'], 'EXPLAIN': ['How is {f} computed?', 'Comment {f} est-il calculé ?'], 'FRESHNESS': ['Is {f} up to date?', 'Est-ce que {f} est à jour ?'], 'FRESHNESS2': ['Are {f} and {f2} up to date?'], 'SUMMARY': ['What features are available in the {d} domain?', 'Quelles features sont disponibles dans le domaine {d} ?'], 'DATA_DOMAINS': ['What data domains are available?', 'Quels domaines de données existent ?'], 'DOMAIN_DETAIL': ['How many features and processes are in the {d} domain?'], 'FEASIBILITY': ['I want to analyse {w} - what does the store already offer for that?'], 'LINEAGE': ['Where does {f} come from?', "D'où vient {f} ?"], 'USAGE': ['How do I use {f} in a query?'], 'DATASET': ['Which datasets expose {f}?'], 'SEARCH': ['Find features about {w}.'], 'HELP': ['What is the difference between a feature and a dataset?'], 'OUT_OF_SCOPE': ['What will the weather be in Paris tomorrow?']}

_FR_MONTHS `module-attribute` ¶

_FR_MONTHS = ['janvier', 'février', 'mars', 'avril', 'mai', 'juin', 'juillet', 'août', 'septembre', 'octobre', 'novembre', 'décembre']

_EN_MONTHS `module-attribute` ¶

_EN_MONTHS = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']

_Gen ¶

store `instance-attribute` ¶

store = store

store_at `instance-attribute` ¶

store_at = store_at

rng `instance-attribute` ¶

rng = Random(seed)

use_pronouns `instance-attribute` ¶

use_pronouns = use_pronouns

include_writes `instance-attribute` ¶

include_writes = include_writes

mcp_enabled `instance-attribute` ¶

mcp_enabled = mcp_enabled

feats `instance-attribute` ¶

feats = features_index(store)

freq `instance-attribute` ¶

freq = word_frequency((f['description']) for f in (feats))

scenarios `instance-attribute` ¶

scenarios: List[dict] = []

coverage `instance-attribute` ¶

coverage: Dict[str, str] = {}

n `instance-attribute` ¶

n = 0

init ¶

__init__(store, store_at, seed, use_pronouns, include_writes, mcp_enabled)

store_for ¶

store_for(t)

pick ¶

pick(pred, k=1, exclude=())

turn ¶

turn(user, intent, spec, as_of=None, accept=None)

fq ¶

fq(f, spec=None)

How a question names a feature: its name, plus its data domain when the name exists in several domains (the question is then not ambiguous).

add ¶

add(prefix, category, desc, skills, turns, pronouns=False, as_ofs=None)

fspec `staticmethod` ¶

fspec(kind, f, **kw)

single ¶

single(skill, k_index)

skip ¶

skip(skill, reason)

multi_turn ¶

multi_turn()

temporal ¶

temporal()

_say_date ¶

_say_date(t: str, french: bool = False) -> str

A valid time as a question would write it: a day in words, a timestamp in ISO.

_time_contexts ¶

_time_contexts(turns: List[dict]) -> None

time_context of each turn, following the agent's own date rules (consumer_agent._resolve_as_of): a date in the turn sets the valid time, a later turn without a date keeps it, "now" clears it, a year alone does not set it. The facts of a dated turn are labelled with the same instant.

generate ¶

generate(skills=None, *, multi_turn=True, use_pronouns=True, per_skill=1, categories=None, fact_first=True, facts_per_kind=2, temporal=True, seed=0, include_writes=False, mcp_enabled=False, store=None, store_at: Optional[Callable] = None, object_facts: Optional[dict] = None) -> dict

See :func:tdfs4ds.agent.evaluation.generate_scenarios.

tdfs4ds.agent.eval_scenarios¶