Metadata-Version: 2.3
Name: failurelab
Version: 0.1.0
Summary: Deterministic AI reliability analysis for RAG and agent traces
Author: FailureLab contributors
License: MIT
Keywords: ai,evaluation,rag,reliability,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: <3.15,>=3.11
Requires-Dist: pydantic<3,>=2.8.2
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: rich>=13.7.1
Requires-Dist: typer>=0.12.3
Provides-Extra: dev
Requires-Dist: bandit>=1.7.9; extra == 'dev'
Requires-Dist: build>=1.2.2; extra == 'dev'
Requires-Dist: check-wheel-contents>=0.6.0; extra == 'dev'
Requires-Dist: codespell>=2.3.0; extra == 'dev'
Requires-Dist: hypothesis>=6.112.1; extra == 'dev'
Requires-Dist: mypy>=1.11.1; extra == 'dev'
Requires-Dist: pip-audit>=2.7.3; extra == 'dev'
Requires-Dist: pre-commit>=3.8.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.3.2; extra == 'dev'
Requires-Dist: ruff>=0.6.2; extra == 'dev'
Requires-Dist: twine>=5.1.1; extra == 'dev'
Description-Content-Type: text/markdown

# FailureLab

[![CI](https://github.com/yellepeddibk/failurelab/actions/workflows/ci.yml/badge.svg)](https://github.com/yellepeddibk/failurelab/actions/workflows/ci.yml)
![Python](https://img.shields.io/badge/python-3.11%20to%203.14-blue)
![License: MIT](https://img.shields.io/badge/license-MIT-green)

FailureLab is a local-first AI reliability toolkit for deterministic analysis of RAG and agent traces. It finds concentrated failure slices, produces evidence-backed root-cause hypotheses, and drafts regression tests from production failures.

## Why FailureLab

Averages can hide failure concentration by model, prompt, retriever, or tool sequence. FailureLab keeps deterministic metric logic in typed services and allows agent layers to consume those deterministic skills.

## 60-second quickstart

```bash
python -m pip install -e ".[dev]"
failurelab --version
failurelab validate examples/rag_traces.jsonl
failurelab analyze examples/rag_traces.jsonl --output reports --overwrite
failurelab compare examples/baseline_traces.jsonl examples/candidate_traces.jsonl --output reports --overwrite
```

## Implemented capabilities

- Typed trace and agent-step contracts using Pydantic v2
- Incremental JSONL ingestion with strict and skip-invalid modes
- Deterministic metrics and categorical breakdowns
- Deterministic failure-slice discovery and root-cause heuristics
- Draft regression test generation in YAML
- Compare baseline/candidate runs with deterministic gate checks
- Typer CLI (`validate`, `analyze`, `compare`)
- JSON, Markdown, and YAML report outputs
- Deterministic skill interfaces and a routing investigation agent

## Example trace schema (minimal)

```json
{"schema_version":"0.1","trace_id":"rag-001","timestamp":"2026-07-10T10:00:00+00:00","success":true}
```

## Output files

- `metrics.json`
- `findings.json`
- `report.md`
- `regression_tests.yaml`
- `run_manifest.json`
- `invalid_traces.jsonl` (skip-invalid mode)
- `comparison.json`, `comparison.md`, `gate_result.json` (compare)

## Configuration precedence

FailureLab resolves configuration deterministically in this order:

1. built-in defaults
2. config file values (`--config`)
3. explicit CLI overrides (for example `--skip-invalid`, `--retrieval-k`)

`run_manifest.json` records the effective resolved configuration used for execution.

## Metrics notes

Each metric includes value, numerator, denominator, eligibility counts, direction, and unavailable reason when needed. Metrics that require observations are reported as unavailable (`null`) when no eligible observations exist; they are not coerced to `0`.

## Analysis semantics

- Failure slices include only elevated failure segments (positive uplift over global failure rate).
- Root-cause hypotheses are generated only for explicitly failed traces.
- Draft regression cases are generated only from failed traces with concrete replayable input.
- Comparison outputs include full-dataset and matched-ID scopes, plus metric deltas.
- Gate status is tri-state: `not_configured`, `passed`, or `failed`.

## Architecture

```mermaid
flowchart TD
  A[JSONL traces] --> B[Ingestion + validation]
  B --> C[Deterministic metrics]
  B --> D[Slice discovery]
  B --> E[Root-cause rules]
  D --> F[Regression draft generation]
  C --> G[Reports]
  D --> G
  E --> G
  F --> G
  C --> H[Typed skills]
  D --> H
  E --> H
```

## Privacy and limitations

- Local-first, no network calls in core analysis.
- Raw query/context content is excluded from reports by default.
- No significance or causal certainty claims.

## Development

```bash
ruff check .
ruff format --check .
mypy src
pytest
python -m build
```

## Roadmap (not implemented)

Statistical inference, embedding clustering, calibrated judge models, active learning, external orchestrators, and optional API/storage/dashboard layers.

## Contributing and license

See [CONTRIBUTING.md](CONTRIBUTING.md) and [LICENSE](LICENSE).
