Metadata-Version: 2.4
Name: clusop
Version: 0.0.17
Summary: Photon Fallback Analyzer — zero-touch Databricks cost forensics (.pth self-arm + bundled JVM listener)
Author: yogasathyandrun
License: Proprietary
Keywords: cost,databricks,finops,photon,spark
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: service
Requires-Dist: databricks-sdk>=0.30; extra == 'service'
Requires-Dist: requests>=2.31; extra == 'service'
Description-Content-Type: text/markdown

# clusop — Photon Fallback Analyzer

`pip install clusop`

clusop finds the money Databricks Photon quietly burns. When Photon hits an operator it
can't run (a Python UDF, a struct-IN filter, an unsupported Delta feature), it silently
**falls back** to the JVM — you keep paying the 2–2.9× Photon DBU premium while getting
JVM speed. clusop detects those fallbacks from the executed plan, estimates the wasted
spend, and proposes a fix. It never touches your job.

## One install, two halves, zero config

A single `pip install clusop` ships both halves and arms itself:

- a **Python `.pth`** (`clusop_autoload.pth`) that Python's `site` machinery runs at
  interpreter startup → imports `clusop.runtime.bootstrap` → arms automatically. You do
  not `import clusop` anywhere in your job.
- a bundled **Scala JAR** (`clusop/jars/photon_listener.jar`) — the actual
  `QueryExecutionListener`. The bootstrap `addJar`s it and registers it over Py4J.

Install it as a **cluster/job library** (production path) so every interpreter that
starts already has it. `%pip install clusop` + `dbutils.library.restartPython()` works for
dev.

Everything is **auto-detected** at runtime — cloud (from the instance type), DBR (from
`DATABRICKS_RUNTIME_VERSION`), cluster shape, and whether the JVM is reachable. No
per-user setup; a new customer pip-installs and it works.

## What it costs you: nothing if it can't run safely

- **Fail-open everywhere.** Any error in arming, listening, parsing, or dispatch is
  swallowed. clusop never breaks, slows, or blocks a customer query.
- **Needs a SINGLE_USER / dedicated cluster** for the JVM listener. On USER_ISOLATION
  (Shared) clusters the JVM is sealed behind Spark Connect — clusop detects this and stays
  dormant rather than failing.
- **Propose-never-apply.** clusop emits a signal and a Teams card. A human decides.

## How a signal is born

1. The JVM listener sees a finished query and reads its **executed plan**.
2. The **structural parser** ([src/clusop/analysis/parser.py](src/clusop/analysis/parser.py),
   mirrored in Scala) decides if this is a *real* fallback. The hard part: a clean Photon
   query always ends in one **terminal** `ColumnarToRow` (the normal result boundary) —
   that is **not** a fallback. Real fallback is **mid-plan** `ColumnarToRow`,
   `RowToColumnar` round-trips, or `BatchEvalPython`/`ArrowEvalPython`. Counting raw
   occurrences false-positives; clusop doesn't.
3. The **waste model** ([waste.py](src/clusop/analysis/waste.py)) sizes the loss:
   `runtime × Σ(node DBU/hr) × $rate × (photon_premium−1) × fallback_weight`.
4. **Confidence** ([confidence.py](src/clusop/analysis/confidence.py)) is decomposed into
   four legs — parse / diagnosis / cost / recommendation. A *fix* (rewrite the UDF) needs
   only parse+diagnosis; a *disable-Photon* recommendation is a dollar decision and is
   capped by the cost leg.
5. The signal flows to a **driver batcher** (dedup by signature) → **central aggregator**
   (idempotent upsert, suppression, prioritization) → Teams card.

## Cost: modeled now, billed if granted

clusop always works with a **modeled** cost (public price table → MEDIUM cost confidence).
If the workspace can read `system.billing.usage`, the
[CostTierResolver](src/clusop/service/resolver.py) reconciles the estimate to **billed**
dollars (HIGH confidence). Detection never depends on system tables — they only upgrade
the cost layer. Prices are reference data, not something clusop invents per-row.

## Certifying it's real

[harness/certify.py](harness/certify.py) runs on a dedicated cluster, captures real
executed plans for known clean / known-fallback fixtures, and reports catch-rate plus a
**DBR-stamped Delta feature-support matrix** — derived, never hand-maintained.

## Layout

```
src/clusop/
  runtime/      bootstrap (arming), detect (cloud/DBR/shape/jvm)
  analysis/     parser · waste · confidence · signal
  pricing/      price_table.json · provider
  service/      aggregator · resolver · suppression · teams · onboard
  jars/         photon_listener.jar  (baked in by the release workflow)
scala/          the QueryExecutionListener (sbt; mirror of the Python parser)
harness/        certify.py  (catch-rate + Delta matrix, run on a dedicated cluster)
tests/          parser / waste / confidence
```

See [SPEC.md](SPEC.md) for the full design and invariants.

## Release

Push-button: the [Publish workflow](.github/workflows/release-pypi.yml) auto-versions
(`max(PyPI, tags)+1`), builds the JAR with sbt, bakes it into the wheel, lints+tests,
gates on *(.pth at wheel root AND JAR bundled)*, then publishes via PyPI Trusted
Publishing (OIDC — no stored token) and tags the bump.
