tdfs4ds.lineage.flow¶
Zoomable left-to-right lineage view ("Flow") for the admin app (0.3.1.240).
The Sankey diagram does not scale: on a real workflow the links cross and the labels overlap, and it cannot be zoomed. This view keeps its reading order (sources on the left, datasets on the right) and adds interaction:
- zoom and pan (wheel, drag, navigation buttons) and a Fit button;
- a click on an object highlights its whole upstream and downstream path and dims the rest (click the background to clear), and posts the object's name to the page embedding it, so the admin app can show its documentation;
- a search box zooms to matching objects;
- filters: hide source tables, hide managers, or show processes and datasets only (intermediate views collapsed into the edges they connect).
:func:flow_model is pure (tested offline); :func:plot_lineage_flow returns one
self-contained HTML page with vis-network inlined from the pyvis package, so it
works offline and makes no external request.
module-attribute
¶
KIND_STYLE = {'table': {'color': '#31a354', 'shape': 'box', 'label': 'source table'}, 'view': {'color': '#6baed6', 'shape': 'box', 'label': 'intermediate view'}, 'process': {'color': '#e6550d', 'shape': 'box', 'label': 'process view'}, 'dataset': {'color': '#756bb1', 'shape': 'box', 'label': 'dataset'}, 'time manager': {'color': '#f2b705', 'shape': 'diamond', 'label': 'TimeManager'}, 'filter manager': {'color': '#c9a227', 'shape': 'diamond', 'label': 'FilterManager'}, 'manager table': {'color': '#d9c27a', 'shape': 'box', 'label': 'manager table'}, 'unknown': {'color': '#bdbdbd', 'shape': 'box', 'label': 'other'}}
module-attribute
¶
_CSS = '\nbody{margin:0;font-family:Inter,Arial,sans-serif;background:#fff;color:#14273A}\n#bar{display:flex;flex-wrap:wrap;gap:10px;align-items:center;padding:8px 10px;border-bottom:1px solid #E2E8F0;\n background:#F7F8FA;font-size:13px}\n#bar input[type=search]{padding:4px 8px;border:1px solid #CBD5E1;border-radius:6px;min-width:200px}\n#bar button{padding:4px 10px;border:1px solid #00233C;background:#00233C;color:#fff;border-radius:6px;cursor:pointer}\n#bar label{display:flex;gap:4px;align-items:center;cursor:pointer}\n#net{position:absolute;top:44px;bottom:28px;left:0;right:0}\n#legend{position:absolute;bottom:0;left:0;right:0;height:28px;display:flex;gap:14px;align-items:center;\n padding:0 10px;font-size:12px;border-top:1px solid #E2E8F0;background:#fff;overflow:hidden}\n.sw{display:inline-block;width:11px;height:11px;border-radius:2px;margin-right:4px;vertical-align:-1px}\n#info{margin-left:auto;color:#5A6B7B}\n'
module-attribute
¶
_JS = "\n(function(){\nvar M = __MODEL__, STYLE = __STYLE__;\nvar nodes = new vis.DataSet(), edges = new vis.DataSet();\nvar byId = {}; M.nodes.forEach(function(n){ byId[n.id] = n; });\nvar up = {}, down = {}; // up[x] = sources x reads, down[x] = consumers of x\nM.edges.forEach(function(e){ (down[e.from] = down[e.from] || []).push(e.to); (up[e.to] = up[e.to] || []).push(e.from); });\nvar state = {hideTables:false, hideManagers:false, compact:false, focus:null};\n\nfunction visible(n){\n if (state.compact) return n.kind === 'process' || n.kind === 'dataset';\n if (state.hideTables && n.kind === 'table') return false;\n if (state.hideManagers && (n.kind === 'time manager' || n.kind === 'filter manager' || n.kind === 'manager table')) return false;\n return true;\n}\nfunction closure(start){\n var keep = {}; keep[start] = 1;\n [up, down].forEach(function(adj){\n var seen = {}, st = [start];\n while (st.length){\n var x = st.pop();\n (adj[x] || []).forEach(function(y){ if (!seen[y]){ seen[y] = 1; keep[y] = 1; st.push(y); } });\n }\n });\n return keep;\n}\nfunction draw(){\n var keep = state.focus ? closure(state.focus) : null;\n var ns = [], es = [];\n M.nodes.forEach(function(n){\n if (!visible(n)) return;\n var s = STYLE[n.kind] || STYLE.unknown, dim = keep && !keep[n.id];\n ns.push({id:n.id, label:n.label, level:n.level, shape:s.shape,\n title: n.database + '.' + n.label + '\\n' + s.label,\n color:{background: dim ? '#EEF1F5' : s.color, border: n.id === state.focus ? '#00233C' : (dim ? '#E2E8F0' : s.color)},\n font:{color: dim ? '#B8C2CC' : (n.kind === 'dataset' || n.kind === 'process' ? '#fff' : '#14273A'), size:13},\n borderWidth: n.id === state.focus ? 3 : 1});\n });\n var list = state.compact ? M.compact_edges : M.edges, k = 0;\n list.forEach(function(e){\n if (!byId[e.from] || !byId[e.to] || !visible(byId[e.from]) || !visible(byId[e.to])) return;\n var dim = keep && !(keep[e.from] && keep[e.to]);\n es.push({id:'e'+(k++), from:e.from, to:e.to, arrows:'to', dashes: !!e.driver,\n title: e.via ? ('through ' + e.via + ' intermediate object(s)') : (e.driver ? 'driver' : ''),\n color:{color: dim ? '#EEF1F5' : (e.driver ? '#c9a227' : '#94A3B8')}, width: dim ? 1 : 1.5});\n });\n nodes.clear(); edges.clear(); nodes.add(ns); edges.add(es);\n document.getElementById('info').textContent = ns.length + ' objects, ' + es.length + ' links' +\n (M.cycles.length ? ' - ' + M.cycles.length + ' cycle(s)' : '');\n}\nvar net = new vis.Network(document.getElementById('net'), {nodes:nodes, edges:edges}, {\n layout:{hierarchical:{enabled:true, direction:'LR', sortMethod:'directed', levelSeparation:230, nodeSpacing:60,\n treeSpacing:80, blockShifting:true, edgeMinimization:true, parentCentralization:true}},\n physics:false, interaction:{navigationButtons:true, keyboard:true, hover:true, zoomView:true, dragView:true, tooltipDelay:120},\n edges:{smooth:{type:'cubicBezier', forceDirection:'horizontal', roundness:0.4}},\n nodes:{margin:8, widthConstraint:{maximum:220}}\n});\nnet.on('click', function(p){\n if (p.nodes.length){\n state.focus = p.nodes[0];\n try { window.parent.postMessage({type:'tdfs4ds-lineage-node', node: state.focus}, '*'); } catch(e) {}\n } else { state.focus = null; }\n draw();\n});\ndocument.getElementById('fit').onclick = function(){ net.fit({animation:true}); };\ndocument.getElementById('q').addEventListener('input', function(){\n var q = this.value.trim().toUpperCase(); if (!q) { net.unselectAll(); return; }\n var hits = M.nodes.filter(function(n){ return visible(n) && (n.database + '.' + n.label).toUpperCase().indexOf(q) >= 0; })\n .map(function(n){ return n.id; });\n net.selectNodes(hits); if (hits.length) net.fit({nodes:hits, animation:true});\n});\n[['t-tables','hideTables'],['t-managers','hideManagers'],['t-compact','compact']].forEach(function(p){\n document.getElementById(p[0]).addEventListener('change', function(){ state[p[1]] = this.checked; draw(); net.fit(); });\n});\ndraw(); net.fit();\n})();\n"
Nodes with a column (depth) and a kind, data-flow edges, compact edges.
graph: {'nodes', 'edges'} as :func:lineage_store.get_persisted_graph
returns it (edge = parent -> child = consumer -> source). process_views:
node ids ('DB.OBJECT') of registered process views.
Columns: a source (reads nothing) is in column 0, every other object one
column right of its deepest source, so data flows left to right as in the
Sankey. A cycle (reported in cycles) is placed as one block.
Edges between processes and datasets only: an intermediate object between
them is folded into the edge (via = how many objects were folded).
plot_lineage_flow(graph: Dict[str, Any], *, process_views=None, output_path: Optional[str] = None, title: str = 'Lineage') -> str
The Flow view as one self-contained HTML page (returned, and written to output_path when given).