Log Discovery

What Cluster every log line into a small set of templates with wildcards <*> for variable parts. "User u-123 logged in" and "User u-456 logged in" collapse to one template.

Algorithm Drain3 tree-based online clustering, refined by a small EM merge pass for near-duplicates.

Why it matters Templates are the atomic unit downstream. Over-merge and distinct events vanish; over-split and review is exhausting.

In Studio Each row in log_templates becomes a candidate :Event_N class with a sample log line attached.

What Classify each <*> position by the values it takes (IRI, UUID, enum, numeric, free text) and flag possible PII. Build an undirected graph over slot values that co-occur within a short time window.

Algorithm Regex ladder + Shannon entropy for typing. Pointwise mutual information (PMI) + temporal-lead ratio for the graph.

Why it matters Entity proposals start here. "slot 2 of template #7 is a UUID with 4 130 distinct values" tells you it's an identifier property worth keeping.

In Studio log_template_slots + log_entity_edges rows; visible as Entity and Relationship proposals.

What Group log lines by service into time-ordered sequences of template IDs. Fit a per-service Hidden Markov Model. Flag sequences in the bottom 5 % of per-step log-likelihood.

Algorithm CategoricalHMM (hmmlearn). State count by BIC. Combined with cluster-bag novelty for trajectories that miss the service's mode templates.

Why it matters Individual log lines look fine; the sequence tells the story. The HMM learns the normal rhythm of each service so deviations surface.

In Studio LOG_EVENT proposals at the top of the queue, each linked back to the anomalous trajectory.

What For each directed entity edge, test whether the history of src actually informs the future of dst beyond dst's own history.

Algorithm Granger F-test on time-binned rate series; transfer entropy z-score as fallback. Triangulation: PMI + lead + statistical causality must all agree.

Why it matters Temporal-lead is necessary but not sufficient for causality. Without this gate, every "occurs before" pair looks like a cause.

In Studio Directed edges that pass become LOG_CAUSAL_EDGE; those that fail become LOG_RELATIONSHIP instead.

What Load every proposal into the wizard's Step 5 queue, sorted by confidence × consequence. Engineer approves, rejects, edits, or merges.

Algorithm Decision-theoretic ordering by expected information value.

Why it matters Your time is the bottleneck. The highest-leverage decisions sit at the top so you can stop reading after the most important N.

In Studio This panel.

What A logistic-regression model trained on your past approve/ reject decisions reorders the queue so the things you tend to approve rise to the top.

Algorithm scikit-learn LogisticRegression on per-proposal features (kind, confidence, evidence volume, slot-PII flag, age, regime tag, …) with median imputation + class balancing.

Why it matters Without it, engineers re-reject the same noise template every session. With it, the queue learns from each decision.

In Studio "↻ Re-rank with feedback" button on Step 5; a small badge under the controls shows "ranker · 47 decisions · AUC 0.83".

What Replace v1's single per-service HMM with a mixture of HMMs. A per-trajectory regime indicator selects which HMM applies — separating business hours from weekend maintenance from incident response.

Algorithm VB-EM with Dirichlet prior on regime mixing weights; K selected by ELBO over 3 random restarts; K ≤ 5.

Why it matters v1's single HMM averages across regimes and flags normal-but-rare patterns (e.g. weekend maintenance) as anomalies. The largest single source of false positives.

In Studio Each LOG_EVENT proposal carries an amber chip — "Regime 2 of 3" — so you know which operating mode produced it.

What Replace v1's heuristic max(c, 0.6) floor with a real posterior probability. Confidence becomes "P(per-step logL < anomaly threshold)" under a Variational Bayesian HMM.

Algorithm VB-HMM with Dirichlet priors on π, A, B; mean-field VB-EM via log-space forward-backward; Bayesian model averaging across K. Confidence via Monte-Carlo posterior predictive.

Why it matters An audit-ready confidence. Comparable across proposal kinds, defensible to a reviewer. ECE < 0.05 on synthetic data.

In Studio Confidence scores stop clustering at 0.6 / 0.7 / 0.8 (the v1 floors) — they spread smoothly across [0, 1].

What Replace pairwise Granger with a structural causal graph. The PC algorithm tests conditional independence under expanding conditioning sets, so shared upstream causes eliminate spurious "B causes C" edges.

Algorithm Partial Pearson correlation + Fisher z-transform CI test. Skeleton phase with conditioning sets ≤ 3. V-structure orientation. Meek rules R1–R4.

Why it matters Pairwise tests can't tell "A causes both B and C" from "B causes C". PC can.

In Studio Causal-edge titles gain a suffix like "— shared upstream cause: template #41" when the PDAG identifies a confounder.

What A 2-D PCA scatter of all active templates, with merge candidates highlighted in amber when their distance in the full-dim embedding falls below a threshold.

Algorithm Bag-of-tokens vectoriser + probabilistic PCA; L2-normalised chord distance for the merge-candidate test.

Why it matters Drain's sim_th=0.4 is a hard knob. Some near-duplicates always slip through. The scatter lets you spot them visually and merge with one click.

In Studio The "Template similarity" card on Step 5; close pairs are connected by faint amber lines.

What Per template, fit a Gaussian Process over hour-of-day rate and flag observations outside the 99 % posterior interval — catching the burst-of-normal-templates pattern v1's HMM misses entirely.

Algorithm Periodic (24 h) + RBF + white-noise kernel; per-hour-median aggregation for robustness; MAD-based residual z-score against the smooth baseline.

Why it matters When a deploy goes bad and a normally-quiet endpoint floods the queue with success logs, v1's HMM sees nothing wrong — the templates are fine, there are just too many of them. The GP sees the rate spike.

In Studio Rows tagged with a green "GP rate" chip and an inline sparkline. "↻ GP rate scan" button on the toolbar.