Metadata-Version: 2.4
Name: assaylab
Version: 0.2.0
Summary: Validation intelligence for CI — failure-signature RCA, flaky-vs-real classification, and attested test-selection with a verifiable confidence bound. Contract-compatible with agentsensory (pass/warn/fail + Handoff).
Project-URL: Homepage, https://github.com/amitpatole/assaylab
Project-URL: Repository, https://github.com/amitpatole/assaylab
Author-email: Amit Patole <amit.patole@gmail.com>
License: MIT
License-File: LICENSE
Keywords: agentsensory,attestation,ci,flaky-tests,root-cause-analysis,test-selection,testing,validation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: agentsensory>=0.1
Requires-Dist: defusedxml>=0.7
Requires-Dist: platformdirs>=4.0
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: pydantic>=2.6
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: anthropic>=0.40; extra == 'all'
Requires-Dist: fastapi>=0.110; extra == 'all'
Requires-Dist: httpx>=0.27; extra == 'all'
Requires-Dist: mcp>=1.2; extra == 'all'
Requires-Dist: python-multipart>=0.0.9; extra == 'all'
Requires-Dist: uvicorn[standard]>=0.29; extra == 'all'
Provides-Extra: datasets
Requires-Dist: httpx>=0.27; extra == 'datasets'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: llm
Requires-Dist: anthropic>=0.40; extra == 'llm'
Requires-Dist: httpx>=0.27; extra == 'llm'
Provides-Extra: mcp
Requires-Dist: mcp>=1.2; extra == 'mcp'
Provides-Extra: serve
Requires-Dist: fastapi>=0.110; extra == 'serve'
Requires-Dist: python-multipart>=0.0.9; extra == 'serve'
Requires-Dist: uvicorn[standard]>=0.29; extra == 'serve'
Description-Content-Type: text/markdown

# assaylab

**Validation intelligence for CI.** Attested test-selection with verifiable
confidence bounds, failure-signature root-cause analysis, and analytics that
support data-driven engineering decisions.

> **Status:** P1–P5 shipped — ingest + failure-signature clustering (P1),
> ML root-cause analysis + flaky-vs-real + risk (P2), attested test-selection
> with a verifiable confidence bound (P3), the Warm Paper dashboard (P4), and
> gated LLM-assisted test generation / self-healing (P5).

## Try it (no API key, no network)

```console
$ pipx install assaylab            # or: uv tool install assaylab
$ assaylab demo
assaylab demo — a synthetic suite graded, then its fix (no API key, no network)

1) broken suite:
verdict: FAIL  —  3 failing execution(s) across 2 signature(s); 1/4 passed.
  [error] failure_signature: NullPointerException across 2 tests (2 runs): ...
      tests: checkout.PaymentTest::test_charge, checkout.RefundTest::test_refund
      signature: 4926f0502c1b  files: checkout/pay.py
  [error] failure_signature: AssertionError across 1 test (1 run): ...
      tests: checkout.CartTest::test_total
      signature: 81ddd6026e88  files: checkout/cart.py

2) after the fix:
verdict: PASS  —  4/4 tests passed — no failure signatures.
```

The two `NullPointerException` failures cluster into **one** signature despite
differing memory addresses — that is the failure-signature engine at work.

Grade your own CI output (exits non-zero on FAIL, so it drops into a gate):

```console
$ assaylab check path/to/junit.xml           # JUnit/xUnit XML
$ assaylab check results.csv --baseline prev.csv   # flag NEW failures vs a baseline
$ assaylab signatures junit.xml              # list failure signatures, most frequent first
$ assaylab perceive junit.xml                # brain-facing Handoff (JSON)
```

### Root-cause analysis (P2)

Point `rca` at run history (many runs, ideally with commit + outcome) to get a
root-cause category, a flaky-vs-real verdict, and a risk score per signature.
An all-flaky failure set grades **WARN** so flakiness doesn't fail your gate:

```console
$ assaylab rca history.csv
verdict: FAIL  —  4 failing execution(s) across 2 signature(s) (1 real, 1 flaky); 4/8 passed.
  [error]   failure_signature  cause=null_deref (conf 1.0)  flaky=False  risk=0.7
  [warning] flaky_suspect      cause=timeout   (conf 0.85) flaky=True (p=0.95)  risk=0.53

$ assaylab risk history.csv --top 5          # rank tests by failure risk + forecast
$ assaylab train labeled.csv -o flaky.json   # fit the flaky model (JSON, never pickle)
$ assaylab rca history.csv --model flaky.json  # use the learned model
```

The flaky model is a pure-Python logistic regression persisted as JSON — no
heavy ML dependency, and loading a model can't execute code.

### Attested test-selection with a verifiable confidence bound (P3)

The distinctive idea: reduce the suite **and** emit a signed receipt that bounds
the confidence lost. Each test `t` gets a detection probability `q_t`; skipping
set `U` costs `ε = 1 − Π(1 − q_t)` — the chance a skipped test would have caught
a regression. Keep the highest value-density tests until `ε ≤ target`:

```console
$ export ASSAYLAB_SIGNING_KEY=$(python -c "import secrets;print(secrets.token_hex(32))")
$ assaylab select history.csv --target-epsilon 0.05 --receipt receipt.json
selected 3/43 tests  speedup 9.0x  confidence 1.0000  (epsilon 0.0000)
  run:  svc.Hot::t0, svc.Hot::t1, svc.Hot::t2
  wrote signed receipt -> receipt.json

$ assaylab verify receipt.json --against history.csv
receipt a1460d1de5ee: signature VALID  (epsilon 0.0000, confidence 1.0000, selected 3/43, speedup 9.0x)
  reproduction: OK — reproduced: selection and confidence bound are genuine
```

Tamper with the receipt's `epsilon` and verification fails closed — the
signature binds the actual bound, and `--against` re-derives it from the inputs.

**Honest limits** (the receipt's residual assumptions, not guarantees):
independence of test failures, `q_t` stationarity, and coverage only of
regression classes seen historically. HMAC is a symmetric trust domain
(verifier holds the key); asymmetric (ed25519) signatures are future work.

### Dashboard (P4)

One self-contained HTML report — verdict, failure signatures + root cause,
riskiest tests, and the confidence/speedup frontier as an inline SVG. No server,
no network; Warm Paper design system.

```console
$ assaylab report history.csv -o report.html --title "checkout service"
wrote dashboard -> report.html
```

### LLM-assisted test generation & self-healing (P5)

`assaylab` proposes a regression test (or a flaky mitigation) from a failure
signature — but it **never executes or applies the generated code**. The
proposal is a dry-run artifact; you run it in your own sandbox, and acceptance
is decided by grading that run through the verdict layer:

```console
$ assaylab generate fail.xml --provider template -o proposal.json   # DRY-RUN (key-free)
$ assaylab accept proposal.json fail.xml       # a real run that reproduces -> ACCEPTED (exit 0)
$ assaylab accept proposal.json pass.xml       # does not reproduce      -> REJECTED (exit 1)
```

Providers: `template` (deterministic, no key), `claude` (`pip install
assaylab[llm]`, `ANTHROPIC_API_KEY`), `ollama` (local/hosted). A generated test
is only trusted once it demonstrably reproduces the bug; a heal only once the
flaky signature stops failing.

## What it does (planned)

**Validation intelligence & analytics**
- Failure-signature clustering over historical CI/test results.
- Automated root-cause analysis and flaky-vs-real classification.
- Predictive failure forecasting and risk identification.
- A dashboard for data-driven engineering decisions.

**Test optimization**
- Risk-based test selection and prioritization from code-change diffs.
- Coverage-gap and redundancy detection.
- Runtime reduction that **preserves a stated confidence bound**.

**Automation**
- LLM-assisted test generation from requirements and code changes.
- Adaptive / self-healing execution, gated behind a verdict layer.

## The distinctive idea

Most test-optimization tools give you a speedup and ask you to trust it.
`assaylab` emits a **signed receipt that bounds the confidence lost** when it
reduces a suite: *ran subset S → probability of missing regression-class C ≤ ε*,
with an attested, checkable proof. Speedup **with** a formal confidence claim.

Verdicts follow the `agentsensory` contract (Report = verdict + grounded issues
+ Handoff), so results are portable and auditable.

MIT © 2026 Amit Patole

*— amitpatole*
