The Third Eye — AI Health & ModelOps Control Plane for Databricks

Fleet Health Overview

12 models monitored across 4 business domains

Healthy Models

—

✅

Under Watch

—

👀

At Risk

—

⚠️

Critical

—

🚨

Avg Health Score

—

out of 100

💯

Open Incidents

—

requiring attention

📋

Tier Changes (24h)

—

models degraded/improved

📈

Total Models

—

in asset inventory

🗂️

Health Distribution

Healthy —

Watch —

At Risk —

Critical —

🚨 Models Needing Attention (worst first)

Model	Tier	Score	Health	Incidents

⚙️ How Third Eye Works — Data Flow

┌─────────────── DATABRICKS NATIVE (read-only) ───────────────────┐
│ Lakehouse Monitoring · system.access.table_lineage              │
│ Unity Gateway (Guardrails, Usage, Trace) · MLflow Registry       │
└────────────────────────┬────────────────────────────────────────┘
                         │  read-only adapters
                         ▼
         ┌─────────────────────────────────┐
         │   DISCOVERY & SYNC JOB          │
         │  MLflow/UC/Gateway diff →        │
         │  model_registry_map             │
         └──────────────┬──────────────────┘
                         ▼
         ┌─────────────────────────────────┐
         │   SIGNAL NORMALIZATION LAYER    │
         │  drift · accuracy · cost ·      │
         │  guardrail · latency            │
         └──────────────┬──────────────────┘
                         ▼
         ┌─────────────────────────────────┐
         │  CORRELATION & SCORING ENGINE   │  ← the actual new engineering
         │  time-window correlation        │
         │  composite health + confidence  │
         │  criticality weighting          │
         └──────────────┬──────────────────┘
                         ▼
               governance.model_health.*
                         │
         ┌───────────────┼───────────────────┐
         ▼               ▼                   ▼
   AI/BI DASHBOARD   GENIE SPACE     COPILOT STUDIO
   (this demo)      (ask anything)  (Teams/Slack/GChat)

Model Leaderboard

Click any row to inspect the model detail. Sorted worst-first by default.

Model	Type	Tier	Health Score ↑	Status	Trend	Incidents	Domain	Last Scored

Active Incidents

Correlated evidence bundles — opened only when multiple signal types co-occur (anti-alert-fatigue).

🚨 3 Open

Model Passports

Auto-generated per-model health summaries from the governed schema — no raw data access needed.

Pipeline Execution Log

Last run: 8m ago · Total duration: 4m 23s · Status: All tasks succeeded

Task Execution Flow

Run Summary

${[ ['Models discovered', '12'], ['New lineage events', '6'], ['Signals emitted', '42'], ['Models scored', '12'], ['Incidents opened', '3'], ['Narratives generated', '3'], ].map(([k,v]) => `

${k} ${v}

`).join('')}

Workflow Schedule

Main Pipeline — Every hour at :00

Passport Generator — Daily at 06:00 UTC

[V2] Auto-Retrain — Paused (threshold-triggered)

[V2] Blast Radius — Paused (event-triggered)

🧿 Genie Investigation Space

Ask anything about your AI fleet in plain English.

🧿

Third Eye Genie is ready.

I can answer questions about your 12 monitored models, 3 open incidents, cost anomalies, drift signals, and lineage changes — all powered by governance.model_health.*.

Try: "Which models are at risk?" or "Why is credit_risk at risk?"

V2 Capabilities

Built — schema designed from day one to support governed autonomy without replacing V1.

${[ { icon:'🔄', title:'Auto-Retraining', desc:'Sustained critical health triggers champion/challenger retraining. Never auto-promotes Tier 1 models.', status:'ready', color:'brand' }, { icon:'💥', title:'Blast Radius Analysis', desc:'Walk multi-hop lineage to find which models/dashboards are downstream of an upstream change.', status:'ready', color:'blue' }, { icon:'🔍', title:'Silent Failure Detection', desc:'Aggregate health looks fine but a population slice has collapsed. Powered by Lakehouse Monitoring fairness metrics.', status:'ready', color:'watch' }, { icon:'🌐', title:'Multi-Workspace Federation', desc:'Aggregate risk scores across workspaces with workspace_id tagging and UC row-level security.', status:'ready', color:'healthy' }, { icon:'📊', title:'Cost-of-Drift Estimation', desc:'Translate health_score degradation into estimated dollar exposure for leadership framing.', status:'planned', color:'brand' }, { icon:'📦', title:'Databricks App Packaging', desc:'Installable app with setup wizard, usage-based pricing mirroring Databricks consumption model.', status:'planned', color:'blue' }, ].map(cap => `

${cap.icon}

${cap.title}

${cap.desc}

${cap.status === 'ready' ? '✅ Built & ready' : '📅 Planned'}

`).join('')}

V2 Schema Design (V1-compatible)

governance.model_health (V1 + V2 tables)
├── model_registry_map       ← V1 + V2 fields (autonomy_level, retrain_trigger_threshold)
├── signal_index             ← V1 (unchanged)
├── lineage_events           ← V1 (unchanged)
├── risk_scores              ← V1 (unchanged)
├── incidents                ← V1 + V2 fields (auto_action_taken, approval_required)
├── remediation_suggestions  ← V1 + V2 fields (blast_radius_summary, cost_impact_estimate)
├── model_passport           ← V1 (unchanged)
├── blast_radius_analysis    ← V2 NEW
└── retrain_candidates       ← V2 NEW