Metadata-Version: 2.4
Name: shkit
Version: 1.2.12
Summary: Deterministic + AI hybrid engine that converts Databricks pipeline failures into prioritized, explainable incident intelligence
Author-email: Sathya <yogasathyandrun.rajesh@zeb.co>
License: MIT
Requires-Python: >=3.11
Requires-Dist: pyyaml
Requires-Dist: requests
Provides-Extra: agent
Requires-Dist: psycopg2-binary>=2.9.0; extra == 'agent'
Provides-Extra: build
Requires-Dist: build>=1.0.0; extra == 'build'
Provides-Extra: dev
Requires-Dist: hypothesis>=6.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest>=7.4.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: sqlglot>=20.0.0; extra == 'dev'
Provides-Extra: full
Requires-Dist: click>=8.1.0; extra == 'full'
Requires-Dist: databricks-sdk>=0.20.0; extra == 'full'
Requires-Dist: databricks-sql-connector>=3.1.0; extra == 'full'
Requires-Dist: dbt-databricks>=1.8.0; extra == 'full'
Requires-Dist: pydantic>=2.0.0; extra == 'full'
Requires-Dist: sqlglot>=20.0.0; extra == 'full'
Provides-Extra: publish
Requires-Dist: databricks-sdk>=0.20.0; extra == 'publish'
Description-Content-Type: text/markdown

# Incident Intelligence Core (IIC)

**A deterministic + AI hybrid engine that converts raw Databricks pipeline failures into structured, prioritized, explainable incident intelligence.**

> **Product model (current):** install from **PyPI** (`shkit==X.Y.Z`) and configure via a
> Databricks **secret scope** (`iic`) in your own workspace — see **[docs/INSTALL.md](docs/INSTALL.md)**.
> The old operator path (`publish-tenant`, `pull/push-antibodies`, the `DBX_TOKENS`
> secret, per-tenant configs in git) is **DEPRECATED** and kept for one release only —
> see **[docs/MIGRATION.md](docs/MIGRATION.md)**.

> Not "AI that fixes pipelines." Not a DevOps automation tool.
> **IIC is a structured intelligence engine** that explains *why* a pipeline failed,
> *what it affects*, and *how urgent it is* — in business and technical terms — and
> only spends an LLM call when the structure says it's worth it.

---

## Design principles

1. **Deterministic first, AI second.** Stages 1–8 never call an LLM. The model runs only at stage 9, and only when the router decides it's warranted.
2. **Every failure becomes a structured object.** The whole system is organised around a handful of dataclasses that flow stage to stage.
3. **No external workflow dependencies.** Databricks + an optional logs webhook. No Jira, ServiceNow, or Airflow lock-in.
4. **No "chatty AI."** The diagnosis engine returns structured output only — it interprets the pre-built `IncidentDNA`, it does not classify or converse.
5. **Every decision is traceable from evidence.** Each `IncidentDNA` records the exact signals it fired on; each `ImpactScore` exposes its term-by-term breakdown.

---

## The 11-stage pipeline

```
 1  Event ingestion        Databricks Jobs/Runs API · logs webhook
 2  Normalize              → NormalizedFailureEvent   (source-agnostic)
 3  Context builder        logs + notebook + metadata + schema snapshot
 4  Dependency analyzer    upstream/downstream blast radius
 5  Change detector        diff failed run vs last successful run
 6  Incident DNA builder   structured failure fingerprint   ← the heart (rules)
 7  Impact engine          deterministic severity + business risk   ← NO LLM
 8  Model router           none / lightweight / powerful, by impact
 9  Diagnosis engine       root cause + reasoning + fix      ← LLM only if routed
10  Report generator       one IncidentReport (JSON + markdown)
11  Notifier               severity-ranked Teams card (optional)
```

Stages 3–8 are pure/deterministic and fully unit-tested. The LLM is invoked at
most once per incident, and never for cache hits or derived dependency failures.

---

## The core data models (the moat)

Everything flows through these, in order (`src/iic/models/`):

| Model | Stage | What it captures |
|-------|-------|------------------|
| `NormalizedFailureEvent` | 2 | source-agnostic failure (pipeline, task, error, run) |
| `IncidentContextBundle` | 3 | logs, notebook source, cluster/job metadata, schema, lineage |
| `ChangeDiffObject` | 5 | schema / config / code / runtime / deployment changes since last success |
| **`IncidentDNA`** | 6 | `failure_type`, `affected_layer`, `root_signal`, `pattern_id`, `confidence_signature`, `signals` |
| `ImpactScore` | 7 | `severity`, `business_risk`, blast radius, transparent `breakdown` |
| `RoutingDecision` | 8 | `tier` (none/lightweight/powerful) + reason |
| `DiagnosisResult` | 9 | `root_cause`, `confidence`, `suggested_fix`, `evidence`, `produced_by` |
| `IncidentReport` | 10 | the final product — `to_dict()` and `to_markdown()` |

### IncidentDNA — deterministic classification

The heart of the system. A rule engine maps the raw error + context + change diff
onto a canonical taxonomy (`SCHEMA_DRIFT`, `DATA_QUALITY`, `MISSING_DATA`,
`PERMISSION`, `DEPENDENCY`, `RESOURCE`, `TIMEOUT`, `CONFIG`, `CODE_ERROR`,
`UNKNOWN`) and records exactly which signals fired. Rules — not an LLM — because
the classification drives the cost decision (routing), so it must be free,
instant, predictable, and traceable.

### Impact engine — deterministic scoring (NO LLM)

```
score = downstream_jobs * 2
      + affected_tables * 1.5
      + dashboard_impact * 3        # a broken exec dashboard outranks an internal job
      + recurrence_score * 2
```

Bands → `LOW / MEDIUM / HIGH / CRITICAL`. A business-facing layer (`gold` / `mart`)
bumps severity one band. The full term contribution is exposed in `breakdown`, so
every score is explainable.

---

## Example output

```
# Incident INC-555-1 — MEDIUM
Pipeline: customer_ingestion · Task: validate_schema · 2026-06-10T10:02:00+00:00

Summary: MEDIUM schema drift in customer_ingestion.validate_schema (silver layer)
         — 2 downstream job(s), 0 dashboard(s) at risk; business risk MODERATE.

Root cause (85%): The failing task hit a schema mismatch — a column it expects is
                  missing or changed type.

Failure DNA:  type=SCHEMA_DRIFT · layer=silver · pattern=schema_drift__silver_v1
Suggested fix: Reconcile the ingestion mapping with the new source schema, then re-run.
Diagnosed by:  rules   (deterministic — no tokens spent)
```

---

## Project structure

```
src/iic/                      ← the engine (packaged as a wheel, fully tested)
  models/                       6 core dataclasses + enums
  ingestion/                    Databricks source + logs-webhook normalizer
  context/                      ContextBuilder
  dependency/                   DependencyAnalyzer (blast radius)
  change/                       ChangeDetector (pure diff core + I/O wrapper)
  dna/                          IncidentDNABuilder  ← the heart
  impact/                       ImpactEngine  ← deterministic, no LLM
  routing/                      IncidentModelRouter
  diagnosis/                    DiagnosisEngine (structured LLM, post-DNA)
  report/                       ReportGenerator
  notify/                       TeamsNotifier (optional)
  runtime/                      IncidentEngine orchestrator + notebook entrypoint
src/healing_kit/              ← shared utility library reused by IIC
                                (Databricks/Teams clients, sql_safety, dependency
                                 graph, hmac, error hashing, auth, query guard)
notebooks/incident_engine.py  ← thin Databricks driver (imports the wheel)
onboarding/                   ← provisioner / preflight / rollback / CLI
tests/unit/test_iic_*.py      ← 66 unit tests for the deterministic core
resources/incident_workflows.yml  ← single consolidated DAB job
```

> **What was removed in the rebuild:** the old multi-notebook self-healing fan-out
> (`event_aggregator`, `context_agent`, `diagnosis_engine`, `resolution_executor`,
> `token_budget_enforcer`, `healing_agent_v2`) and the old orchestrator — all
> redundant duplicate logic. The pipeline is now one consolidated, tested engine.

---

## Quick start

```bash
git clone https://github.com/yogasathyandrun/Self-healing-kit.git
cd Self-healing-kit
pip install -r requirements.txt
python setup.py
```

`setup.py` runs preflight checks, provisions the schema + the single
`[IIC] Incident Intelligence Engine` job, uploads the thin driver, and wires a
failure-triggered task onto the pipeline you want to observe.

After setup, run your pipeline normally. When a task fails you get one
severity-ranked incident report in Teams.

---

## Development

```bash
pip install -e ".[dev]"
pytest tests/unit/ -q          # 66 IIC tests + existing healing_kit tests
ruff check src/ onboarding/
```

The deterministic core has no Spark/Databricks dependency, so the whole engine is
testable with plain fakes — see `tests/unit/test_iic_engine.py` for the full
11-stage flow exercised end-to-end.

---

## What still needs a live workspace

The deterministic core (stages 3–8, 10) is fully tested here. These touch live
Databricks and are exercised against a real workspace, not in CI unit tests:

- **Unity Catalog lineage** for table-level blast radius (falls back to task-level DAG when unavailable).
- **Model Serving** for the LLM diagnosis path (the engine is fully functional without it — it falls back to deterministic templates).
- **Provisioner / DAB deployment** of the consolidated job and trigger wiring.

---

## v4 — Databricks-native Cluster Agent SDK (recommended)

The intelligence core ships as a **Databricks-native SDK** (a Python wheel) that
**auto-activates on import**. Install it as a cluster library and restart — that's
the whole setup. **No external infrastructure, no notebook changes:**

```
merge to main → CI auto-publishes the rolling "latest" release + copyable URL
             → install on cluster from URL (init script / %pip) → restart cluster
             → import iic → auto_bootstrap() → excepthook + reconciler active
```

Every merge to `main` refreshes a rolling **`latest`** release and prints the
install URL in the run summary (no tagging needed):
`…/releases/download/latest/iic-1.0.0-py3-none-any.whl`. Pushing a version tag
(`v1.2.0`) additionally cuts an immutable, versioned release.

Uncaught Python errors in any job on that cluster are then captured, analyzed by
the engine, and reported. A Jobs API **reconciler** backstops the rest, and an
optional one-line `with iic_monitor():` gives guaranteed immediate capture for
critical sections. Importing `iic` stays fully inert off-cluster (tests/local).
See [docs/V4_ARCHITECTURE.md](docs/V4_ARCHITECTURE.md).

## Author

**Sathya** | Data Engineer
Rebuilt as the Incident Intelligence Core, June 2026.
GitHub: https://github.com/yogasathyandrun/Self-healing-kit
