tdfs4ds.feature_store.type_changes¶
Type changes of registered features (0.3.5.1).
A feature (data domain, entity, name) has ONE catalog entry, so one value type and one
storage table, whatever the number of processes computing it. A pre-existing feature whose
source type changed used to be ignored silently: its values kept going into the old
FEATURE_VALUE type, cast or truncated.
The source type is compared to the stored one by whether the values FIT, never by equality:
fits: same type, or one whose values the stored type holds (a shorter VARCHAR of the same character set or LATIN into UNICODE, an integer into a wider integer, FLOAT or a large enough DECIMAL, a smaller DECIMAL into a larger one, a lower TIMESTAMP precision). Nothing to say.wider: a type exists that holds the stored type, this process's type and every other producer's type (:func:common_type). Migrating the feature to it is safe for every producer, and the processes that already fit keep fitting: there is no ping-pong.incompatible: no common type (text and number, number and date, ...). One feature cannot hold both kinds of values.
upload_features refuses a re-registration with a feature that does not fit, and the
message explains the alternatives with code (:func:conflict_message); run() logs the
same text as a warning and writes with the stored type. The migration itself is
relocation.plan_type_change.
module-attribute
¶
module-attribute
¶
_TEXT_RE = compile('^(VARCHAR|CHAR|CHARACTER\\s+VARYING|CHARACTER|VARGRAPHIC|LONG\\s+VARCHAR)\\s*(?:\\(\\s*(\\d+)\\s*\\))?(.*)$')
module-attribute
¶
_DEC_RE = compile('^(?:DECIMAL|NUMERIC|DEC|NUMBER)\\s*(?:\\(\\s*(\\d+)\\s*(?:,\\s*(\\d+)\\s*)?\\))?')
module-attribute
¶
module-attribute
¶
sql_type with its VARCHAR length replaced by n, every other character kept (its character-set spelling is part of the storage table name: rewriting it would change the name).
{"kind": int|float|decimal|text|date|timestamp|other, ...} for a SQL type text. Pure.
Whether every value of type source is stored without loss in type stored. Pure.
The narrowest type holding every value of types, or None when there is none. Pure.
One of types when it holds the others; otherwise a computed one (integers -> the widest
integer, integers and decimals -> a DECIMAL large enough (FLOAT beyond 38 digits), any
FLOAT -> FLOAT, texts -> VARCHAR(max) in UNICODE when any is UNICODE, the length
rounded up to varchar_multiple). Mixed kinds (text and number, ...) -> None.
One VARCHAR type per character set for a whole result set. Pure.
types: {column: sql type} of the feature columns of a process. Every VARCHAR of a
character set gets VARCHAR(ceil(longest / multiple) * multiple) of that set, the
longest being taken over the result set; LATIN and UNICODE stay apart. multiple=None
keeps the longest length unrounded. Other types are returned unchanged.
new_feature_types(source_types: Dict[str, str], new_names, stored: Dict[str, dict], multiple=1024) -> Dict[str, str]
The types of the features being registered, one text type per process and character set. Pure.
source_types: {column: source type} of every feature of the result set; new_names:
the features not registered yet; stored: {FEATURE upper: {"type": ...}} of the
registered ones. A new VARCHAR takes the stored type of a registered VARCHAR of the same
character set that holds it (the widest such), so it joins the process's existing text
table; otherwise the new VARCHARs of a character set share
VARCHAR(ceil(longest / multiple) * multiple), the longest taken over the NEW features.
Registered features never change type here. Non-text types are returned unchanged.
The conflict of one feature, or None when source fits stored. Pure.
others: [{"process_id", "view", "type"}] of the OTHER processes computing the
feature (type None when their view could not be read).
The select-list item that produces column in a view text, when it can be found
(<expression> AS <column> on one line, or the bare column). Pure, best effort.
conflict_message(conflicts: List[dict], process_view: str, process_id: Optional[str] = None, view_ddl: Optional[str] = None, other_ddls: Optional[Dict[str, str]] = None, at_run: bool = False, entity=None, data_domain=None) -> str
The refusal (registration) or warning (at_run) text with the alternatives and code. Pure.
{FEATURE_NAME upper: {"type", "table", "database", "id"}} of the registered features
of this entity and domain (current catalog rows).
other_producers(feature_names, entity_columns, data_domain=None, exclude_view=None, exclude_pid=None) -> Dict[str, List[dict]]
{FEATURE upper: [{"process_id", "view", "type"}]}: the other registered processes of
this domain and entity that compute each feature, with their source type (None when the
view could not be read).
find_conflicts(source_types: Dict[str, str], entity_columns, data_domain=None, process_view=None, process_id=None, varchar_multiple=None, stored=None) -> List[dict]
The conflicts of the registered features of a result set (I/O: the catalog, and the
other producers only when a conflict exists). source_types: {feature: type} after
the VARCHAR unification.