tdfs4ds.agent.plot_skill¶
PLOT skill: draw a feature or a dataset in the database with TD_PLOT.
The request is read by code, not by an LLM: the dataset or feature must be named, and every column the chart uses is checked against the real object.
- A dataset named - its columns named in the question choose the chart: a
numeric column is the value (
y); a second numeric column after "against / versus / vs / en fonction de" is the x axis of a scatter; a text column after "by / par" is the category of a bar chart; "distribution / histogram" gives a histogram. Whatever is left is chosen by :func:tdfs4ds.plotting.plot_dataset(time axis, entity as series). - A feature named - its distribution at the valid time, or its history when the question says "over time / history / evolution / trend" (the entity and data domain of the turn decide between same-named features).
- A documented source object named (a table or view of the business dictionary or the lineage graph, never a feature-store internal) - a historian (deep) table is drawn tag by tag from its documented layout ("NVDA close": tag CLOSE, one line per id NVDA); a wide table or view like a dataset. A tag name alone finds its historian table.
- A feature's period as the window - "during its last stress episode", or a date/timestamp/PERIOD feature named next to the series: the window is the period the feature stores for the series' id value. "With its episodes" adds a panel of every period the feature has held over its history.
The answer is written by code. The image is drawn by TD_PLOT; an interactive
version is added only for small requests (tdfs4ds.PLOT_INTERACTIVE_MAX_POINTS).
module-attribute
¶
_PLOT_RE = compile("\\b(plot|plotting|chart|histogram|scatter\\s*plot|bar\\s*chart|line\\s*chart|graph\\s+of|graph\\s+(?:the|it|this|them)|visuali[sz]e|visuali[sz]ation|draw|trace[rz]?\\s+(?:le|la|les|l')|graphique|courbe|histogramme|diagramme|dessine[rz]?|repr[ée]sente[rz]?\\s+graphiquement|visualise[rz]?|zoom\\w*|heat\\s*map|carte\\s+de\\s+chaleur|correlation\\s+(?:matrix|chart|plot)|matrice\\s+de\\s+corr\\w*|superimpos\\w*|super-impos\\w*|superpos\\w*|overlay|rebase\\w*|base\\s*100)\\b", IGNORECASE)
module-attribute
¶
_NOT_PLOT_RE = compile("\\b(lineage|lignage|upstream|downstream|d[ée]pendanc\\w*|dependenc\\w*|provenance|where\\s+does\\s+\\S+\\s+come\\s+from|d'o[uù]\\s+vient)\\b", IGNORECASE)
module-attribute
¶
_HIST_RE = compile('\\b(distribution|histogram\\w*|r[ée]partition|spread|dispersion)\\b', IGNORECASE)
module-attribute
¶
_HISTORY_RE = compile('\\b(over\\s+time|history|historique|evolution|[ée]volution|trend|tendance|au\\s+fil\\s+du\\s+temps|through\\s+time|dans\\s+le\\s+temps)\\b', IGNORECASE)
module-attribute
¶
_SCATTER_RE = compile('\\b(against|versus|vs\\.?|en\\s+fonction\\s+de|scatter|nuage)\\b', IGNORECASE)
module-attribute
¶
module-attribute
¶
_CORR_RE = compile('\\b(correlat\\w*|corr[ée]l\\w*|heat\\s*map|carte\\s+de\\s+chaleur)\\b', IGNORECASE)
module-attribute
¶
_DURING_RE = compile("\\b(during|within|throughout|pendant|durant|lors\\s+d[eu']|au\\s+cours\\s+d[eu'])\\b", IGNORECASE)
module-attribute
¶
_PERIODS_RE = compile('\\b(episodes?|[ée]pisodes?|periods?|p[ée]riodes?|intervals?|intervalles?)\\b', IGNORECASE)
module-attribute
¶
_ALL_TAGS_RE = compile('\\b(?:all|every)\\s+(?:the\\s+)?(?:tags?|signals?|measures?)\\b|\\btou(?:s|tes)\\s+les\\s+(?:tags|signaux|mesures)\\b', IGNORECASE)
module-attribute
¶
module-attribute
¶
_ALIGN_RE = compile('\\b(superimpos\\w*|super-impos\\w*|superpos\\w*|overlay\\w*|align\\w*|same\\s+start\\w*|(?:each|every|per)\\s+(?:stress\\s+)?(?:episode|period)|chaque\\s+(?:[ée]pisode|p[ée]riode)|m[êe]me\\s+(?:date\\s+de\\s+)?d[ée]part|day\\s*0|jour\\s*0|relative\\s+(?:time|dates?))\\b', IGNORECASE)
module-attribute
¶
_SEPARATE_RE = compile('\\b(separately|one\\s+(?:panel|chart|plot|figure)\\s+(?:per|for\\s+each)|side\\s+by\\s+side|s[ée]par[ée]ment|un\\s+graphique\\s+par)\\b', IGNORECASE)
module-attribute
¶
_RAW_RE = compile('\\b(raw|absolute\\s+values?|actual\\s+prices?|valeurs?\\s+brutes?|prix\\s+r[ée]els?)\\b', IGNORECASE)
module-attribute
¶
_REF_WORD_RE = compile('\\b(median|m[ée]diane|mean|average|moyenne|first\\s+value|premi[eè]re\\s+valeur)\\b', IGNORECASE)
module-attribute
¶
_REF_MOD_RE = compile('\\b(?:on|to|at|sur|à|a|by|par|with|using|relative\\s+to|par\\s+rapport\\s+[àa])\\s+(?:the\\s+|its\\s+|their\\s+|each\\s+|la\\s+|sa\\s+|leur\\s+|le\\s+)?(?:median|m[ée]diane|mean|average|moyenne)\\b|\\b(?:median|m[ée]diane|mean|average|moyenne)\\s+(?:instead|à\\s+la\\s+place|as\\s+(?:the\\s+)?reference)', IGNORECASE)
module-attribute
¶
_REBASE_RE = compile('\\b(superimpos\\w*|super-impos\\w*|overlay\\w*|superpos\\w*|rebas\\w*|base\\s*100|same\\s+(?:origin|starting\\s+point|start)|m[êe]me\\s+(?:origine|point\\s+de\\s+d[ée]part)|align\\w*\\s+(?:on|sur)|normali[sz]\\w*|indexed)\\b', IGNORECASE)
module-attribute
¶
_NUM_RANGE_RE = compile('\\b(?:between|from|entre|de|du)\\s+(-?\\d+(?:[.,]\\d+)?)\\s+(?:and|to|et|à|a|au)\\s+(-?\\d+(?:[.,]\\d+)?)\\b', IGNORECASE)
module-attribute
¶
module-attribute
¶
module-attribute
¶
_FR_HINT_RE = compile('\\b(graphique|courbe|histogramme|trace[rz]?|dessine[rz]?|fais|montre[rz]?|[ée]volution|cat[ée]gories?|quotidien\\w*|montants?|clients?|matrice|corr[ée]lation|zoome[rz]?|entre|pour|les|des|du|de|la|le|par|au|aux)\\b', IGNORECASE)
module-attribute
¶
_TOP_RE = compile('\\b(?:top|first|largest|biggest|highest|best|bottom|smallest|lowest|least|worst)\\s+(\\d{1,3})\\b|\\b(\\d{1,3})\\s+(?:largest|biggest|highest|best|smallest|lowest|least|worst|top|bottom)\\b|\\b(?:les\\s+)?(\\d{1,3})\\s+(?:plus\\s+(?:grand|gros|[ée]lev|fort|petit|faible|bas)\\w*|premi[eè]r\\w*|derni[eè]r\\w*)', IGNORECASE)
module-attribute
¶
_SMALLEST_RE = compile('\\b(bottom|smallest|lowest|least|worst|petit\\w*|faible\\w*|bas|derni[eè]r\\w*)\\b', IGNORECASE)
module-attribute
¶
module-attribute
¶
_TAG_WORDS_RE = compile('\\b(sensors?|capteurs?|tags?|signals?|signaux|variables?|parameters?|param[eè]tres?|measurement\\s+(?:names?|types?|ids?)|point\\s+names?|metrics?\\s+names?)\\b', IGNORECASE)
module-attribute
¶
_VALUE_WORDS_RE = compile('\\b(values?|valeurs?|measured|mesur[ée]e?s?|readings?|lectures?|payload|measures?|mesures?|amounts?)\\b', IGNORECASE)
module-attribute
¶
_TIME_WORDS_RE = compile('\\b(time|timestamp|date|datetime|horodatage|instant|heure|moment)\\b', IGNORECASE)
module-attribute
¶
_ID_WORDS_RE = compile('(?:\\b|_)(id|ids|nr|no|num|number|numbers|identifier|identifiant|code|key|num[ée]ro)(?:\\b|_)', IGNORECASE)
module-attribute
¶
_HISTORIAN_WORD_RE = compile('\\b(historians?|historiens?|historis[ée]e?s?|deep\\s+tables?|tables?\\s+profondes?)\\b', IGNORECASE)
module-attribute
¶
_TAG_NOISE = {'sensor', 'sensors', 'capteur', 'capteurs', 'tag', 'tags', 'signal', 'signals', 'signaux', 'measure', 'measures', 'mesure', 'mesures', 'the', 'of', 'du', 'de', 'la', 'le', 'les', 'des', 'value', 'values', 'valeur', 'valeurs'}
module-attribute
¶
_HOW_WORDS = {'plot', 'chart', 'show', 'draw', 'graph', 'trace', 'curve', 'curves', 'during', 'around', 'period', 'periods', 'episode', 'episodes', 'superimposed', 'superimpose', 'same', 'origin', 'over', 'time', 'with', 'from', 'between', 'every', 'each', 'their', 'them', 'these', 'those', 'please', 'want', 'would', 'like', 'could', 'values', 'value', 'data'}
module-attribute
¶
_STOP = {'the', 'its', 'their', 'his', 'her', 'with', 'during', 'plot', 'show', 'chart', 'and', 'for', 'les', 'des', 'ses', 'son', 'sa', 'pendant', 'avec', 'trace', 'the', 'last', 'dernier', 'dernière', 'derniere'}
module-attribute
¶
_SPEC_SCHEMA = {'type': 'object', 'properties': {'target': {'type': 'string', 'enum': ['dataset', 'features', 'object', 'none']}, 'object': {'type': 'string'}, 'tags': {'type': 'array', 'items': {'type': 'string'}}, 'dataset': {'type': 'string'}, 'features': {'type': 'array', 'items': {'type': 'string'}}, 'y': {'type': 'array', 'items': {'type': 'string'}}, 'x': {'type': 'string'}, 'kind': {'type': 'string', 'enum': ['line', 'scatter', 'bar', 'hist', 'history', 'distribution', 'correlation']}, 'agg': {'type': 'string', 'enum': ['avg', 'sum', 'min', 'max', 'count']}}, 'required': ['target']}
'first', 'mean' or 'median': the last reference word of the
question (a follow-up is appended to the previous request, so it wins).
True for "plot X", "show a histogram of X", "trace l'évolution de X"...
True for a short reply that only changes the previous chart: "on the median", "the mean instead", "raw values", "separately", "around it", a zoom window. Used only when the conversation has a previous chart.
Position of name as a whole word in text (case-insensitive), else None.
The names mentioned in text, in order of appearance, longest first on ties.
A window end for people: 2023-10-02 for midnight UTC, else as given.
(n, order) from "top 5" / "the 3 smallest" / "les 5 plus grands", else (None, None).
Words and quoted phrases of the question that could be entity values.
They are only candidates: the store says which ones are real values (one query per entity column). Names already used (features, datasets, columns) and the number of a "top N" are left out.
A span back from now written in the question ("the latest week", "last 7
days", "les 3 derniers mois"): (n, unit), else None. Pure (no database).
The database's current time as a datetime (UTC offset kept), or None.
The window of "the latest week" ends at the database's now, not the client's: the rows were stamped by the database.
(now - n unit, None) as text for :func:tdfs4ds.plotting.window_predicate.
Months and years go back on the calendar (31 March minus one month is 28 or
29 February), like ADD_MONTHS; the window has no upper end (up to now).
A zoom window (low, high) written in the question, else None.
Two dates (ISO, YYYY-MM, or in words) give both ends, in order; one date
after "from / since / depuis" or "until / to / jusqu'au" gives one end;
"between 100 and 500" / "from 10 to 20" / "entre 10 et 20" give a numeric
window; "the latest week" / "last 7 days" give (now - span, None) with
the database's now (resolve_relative False: not read, no window). Dates are
returned as text for :func:tdfs4ds.plotting.window_predicate.
skill_plot(question: str, object_name: Optional[str] = None, entity_name: Optional[str] = None, data_domain: Optional[str] = None, as_of: Optional[str] = None, last_plot: Optional[dict] = None) -> dict
Draw what the question asks for (see the module docstring).
last_plot is the previous chart request of the conversation: a question that names nothing but a window ("zoom on 2023-03 to 2023-06") redraws it on that window.
The skill result of a drawn (or failed) chart.
{DATASET_NAME: 'db.name'} from the dataset catalog of the turn.
The dataset holding the most of features (at least two), for a correlation.
_plot_dataset_from_question(plotting, dataset: str, question: str, top_n=None, order=None, xlim=None, corr: bool = False, features=()) -> dict
_plot_features_from_question(plotting, features, question, entity_name, data_domain, as_of, top_n=None, order=None, xlim=None)
A feature-store object (storage tables, catalogs, hidden manager tables): never a source object to plot.
{NAME_UPPER: {db, name, kind, tags, columns, layout, documented}}, read once per turn.
The plot targets beyond datasets and features: tables and views described
in the business dictionary (objects, or owners of documented columns) and
the tables and views of the lineage graph. A table with a documented
historian layout is kind='historian' and lists its tags (documented
names that are not its layout's columns). Feature-store internals, process
views and datasets (they have their own path) are left out.
A lineage-only name is a real table or view (the graph also stores SQL tokens such as POSITION as objects).
A column's name (_ as space, camel case split) and its description.
The column documentation of o speaks of a sensor (tag) and of a value: worth reading its column types to see whether it is a historian table.
The historian layout a table's columns describe: (layout, problem).
columns: [{"name", "type", "description"}] (type = a HELP COLUMN
code). The timestamp is the date/timestamp column (the one described as a
time when there are several); the payload is the numeric column described as
a value / measure / reading; the tag is the text column described as a
sensor / tag / signal; the id columns are the other documented columns (the
drawing tower, the machine). Each role must have exactly one column:
otherwise (None, "<what is ambiguous or missing>"). Pure.
o as a historian table whose layout comes from its column documentation (one HELP COLUMN read), or None. A copy: the turn's object list is unchanged.
The historian table a question means without naming it: the only historian when the question says "historian", else the one whose documentation (description, column descriptions, tags) shares the most words with the question, when it is the only one with that score.
The words of a tag or of a question, lower case, camel case and _ split,
units in brackets and measure words left out.
The tags question names, tolerant of spelling: "drawing length sensor"
names DRAWING_LENGTH, DrawingLength and Drawing length (m).
A tag counts when all its words are words of the question, or when its
compact form (drawinglength) is; the tags with the most words win (so
DRAWING_LENGTH beats LENGTH for "drawing length"). Several tags with
the same best score are all returned: the caller asks. Pure.
The id columns "per / by / for each MACHINE_NR, described as the drawing tower).
The source object a question names, or whose historian tag it names.
{"object": <entry>, "tags": [...]}. A tag named for two historian
tables, with neither table named, is ambiguous: None (asked later
through the general path). A named table described column by column as a
historian (sensor, value, timestamp) is drawn as one (:func:_as_historian).
The documented object a question describes without naming it ("all the stocks" -> the table described as "daily prices of five US stocks").
Words of the question (4+ letters, not drawing words) are matched against
each object's documented description, plurals included; the object with the
most matches wins, only when it is the only one with that score. {"object",
"tags": [], "matched": [...]} or None.
Features holding a date, a timestamp or a period: candidates for a window.
The date/period feature a question uses as its window, or None.
A time feature named in the question; else, when the question speaks of an episode / period, the time feature whose name shares the most words with the question (ties: the first by name, said in the answer).
The periods a question could mean ("the stress period"), for the agent to ask.
Pairs of start / end time features (and PERIOD features) sharing a word with
the question, each with the window it stores today: [{"feature", "label",
"span"}]. Never raises.
What is missing to draw the chart, asked in one message. The request is
kept (pending): the reply completes it.
Draw a documented source object; (result, understood).
For a historian table, what is missing is asked (a plot_needs_detail
result carrying pending): the measure when no tag is named, and the
period when the question speaks of one that no feature clearly gives.
_plot_historian_object(plotting, ca, hit: dict, fq: str, question: str, top_n, order, as_of, xlim, fr)
The historian branch of :func:_plot_object (its layout already known).
The model's object target, checked against the documented objects.
Datasets with their columns, and features with entity and domain, from the turn snapshot.
The instruct model maps a request that names nothing exactly to a dataset or
features of the store. None without a model or when it finds nothing.
_plot_from_spec(plotting, spec, datasets, question, entity_name, data_domain, as_of, top_n, order, fr, xlim=None)
Run what the model understood, after checking every name against the store.
The name designates several features (entities / data domains): ask which.
Nothing to draw was recognised: say what can be named. The request is
kept (pending) so the reply ("it is in the historian", a tag, a table)
completes it instead of starting another question.