Metadata-Version: 2.4
Name: evalvitals
Version: 0.1.1
Summary: Failure case analysis in the era of LLM.
License: Creative Commons Legal Code
        
        CC0 1.0 Universal
        
            CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
            LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
            ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
            INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
            REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
            PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
            THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
            HEREUNDER.
        
        Statement of Purpose
        
        The laws of most jurisdictions throughout the world automatically confer
        exclusive Copyright and Related Rights (defined below) upon the creator
        and subsequent owner(s) (each and all, an "owner") of an original work of
        authorship and/or a database (each, a "Work").
        
        Certain owners wish to permanently relinquish those rights to a Work for
        the purpose of contributing to a commons of creative, cultural and
        scientific works ("Commons") that the public can reliably and without fear
        of later claims of infringement build upon, modify, incorporate in other
        works, reuse and redistribute as freely as possible in any form whatsoever
        and for any purposes, including without limitation commercial purposes.
        These owners may contribute to the Commons to promote the ideal of a free
        culture and the further production of creative, cultural and scientific
        works, or to gain reputation or greater distribution for their Work in
        part through the use and efforts of others.
        
        For these and/or other purposes and motivations, and without any
        expectation of additional consideration or compensation, the person
        associating CC0 with a Work (the "Affirmer"), to the extent that he or she
        is an owner of Copyright and Related Rights in the Work, voluntarily
        elects to apply CC0 to the Work and publicly distribute the Work under its
        terms, with knowledge of his or her Copyright and Related Rights in the
        Work and the meaning and intended legal effect of CC0 on those rights.
        
        1. Copyright and Related Rights. A Work made available under CC0 may be
        protected by copyright and related or neighboring rights ("Copyright and
        Related Rights"). Copyright and Related Rights include, but are not
        limited to, the following:
        
          i. the right to reproduce, adapt, distribute, perform, display,
             communicate, and translate a Work;
         ii. moral rights retained by the original author(s) and/or performer(s);
        iii. publicity and privacy rights pertaining to a person's image or
             likeness depicted in a Work;
         iv. rights protecting against unfair competition in regards to a Work,
             subject to the limitations in paragraph 4(a), below;
          v. rights protecting the extraction, dissemination, use and reuse of data
             in a Work;
         vi. database rights (such as those arising under Directive 96/9/EC of the
             European Parliament and of the Council of 11 March 1996 on the legal
             protection of databases, and under any national implementation
             thereof, including any amended or successor version of such
             directive); and
        vii. other similar, equivalent or corresponding rights throughout the
             world based on applicable law or treaty, and any national
             implementations thereof.
        
        2. Waiver. To the greatest extent permitted by, but not in contravention
        of, applicable law, Affirmer hereby overtly, fully, permanently,
        irrevocably and unconditionally waives, abandons, and surrenders all of
        Affirmer's Copyright and Related Rights and associated claims and causes
        of action, whether now known or unknown (including existing as well as
        future claims and causes of action), in the Work (i) in all territories
        worldwide, (ii) for the maximum duration provided by applicable law or
        treaty (including future time extensions), (iii) in any current or future
        medium and for any number of copies, and (iv) for any purpose whatsoever,
        including without limitation commercial, advertising or promotional
        purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
        member of the public at large and to the detriment of Affirmer's heirs and
        successors, fully intending that such Waiver shall not be subject to
        revocation, rescission, cancellation, termination, or any other legal or
        equitable action to disrupt the quiet enjoyment of the Work by the public
        as contemplated by Affirmer's express Statement of Purpose.
        
        3. Public License Fallback. Should any part of the Waiver for any reason
        be judged legally invalid or ineffective under applicable law, then the
        Waiver shall be preserved to the maximum extent permitted taking into
        account Affirmer's express Statement of Purpose. In addition, to the
        extent the Waiver is so judged Affirmer hereby grants to each affected
        person a royalty-free, non transferable, non sublicensable, non exclusive,
        irrevocable and unconditional license to exercise Affirmer's Copyright and
        Related Rights in the Work (i) in all territories worldwide, (ii) for the
        maximum duration provided by applicable law or treaty (including future
        time extensions), (iii) in any current or future medium and for any number
        of copies, and (iv) for any purpose whatsoever, including without
        limitation commercial, advertising or promotional purposes (the
        "License"). The License shall be deemed effective as of the date CC0 was
        applied by Affirmer to the Work. Should any part of the License for any
        reason be judged legally invalid or ineffective under applicable law, such
        partial invalidity or ineffectiveness shall not invalidate the remainder
        of the License, and in such case Affirmer hereby affirms that he or she
        will not (i) exercise any of his or her remaining Copyright and Related
        Rights in the Work or (ii) assert any associated claims and causes of
        action with respect to the Work, in either case contrary to Affirmer's
        express Statement of Purpose.
        
        4. Limitations and Disclaimers.
        
         a. No trademark or patent rights held by Affirmer are waived, abandoned,
            surrendered, licensed or otherwise affected by this document.
         b. Affirmer offers the Work as-is and makes no representations or
            warranties of any kind concerning the Work, express, implied,
            statutory or otherwise, including without limitation warranties of
            title, merchantability, fitness for a particular purpose, non
            infringement, or the absence of latent or other defects, accuracy, or
            the present or absence of errors, whether or not discoverable, all to
            the greatest extent permissible under applicable law.
         c. Affirmer disclaims responsibility for clearing rights of other persons
            that may apply to the Work or any use thereof, including without
            limitation any person's Copyright and Related Rights in the Work.
            Further, Affirmer disclaims responsibility for obtaining any necessary
            consents, permissions or other rights required for any use of the
            Work.
         d. Affirmer understands and acknowledges that Creative Commons is not a
            party to this document and has no duty or obligation with respect to
            this CC0 or use of the Work.
        
Project-URL: Homepage, https://github.com/evalvitals/evalvitals
Project-URL: Documentation, https://evalvitals.github.io/evalvitals/
Project-URL: Repository, https://github.com/evalvitals/evalvitals
Project-URL: Issues, https://github.com/evalvitals/evalvitals/issues
Project-URL: Changelog, https://github.com/evalvitals/evalvitals/blob/main/CHANGELOG.md
Keywords: llm,vlm,interpretability,failure-analysis,explainability
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Requires-Dist: numpy>=1.24
Provides-Extra: local
Requires-Dist: torch>=2.0.0; extra == "local"
Requires-Dist: transformers<5.0.0,>=4.51.0; extra == "local"
Requires-Dist: accelerate>=0.26.0; extra == "local"
Provides-Extra: api
Requires-Dist: openai>=1.0; extra == "api"
Provides-Extra: interp
Requires-Dist: captum>=0.7; extra == "interp"
Requires-Dist: inseq>=0.6; extra == "interp"
Requires-Dist: nnsight>=0.3; extra == "interp"
Provides-Extra: data
Requires-Dist: datasets>=2.0; extra == "data"
Requires-Dist: pillow>=9.0; extra == "data"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.7; extra == "viz"
Provides-Extra: stats
Requires-Dist: statsmodels>=0.14; extra == "stats"
Requires-Dist: scikit-learn>=1.3; extra == "stats"
Provides-Extra: cluster
Requires-Dist: scikit-learn>=1.3; extra == "cluster"
Requires-Dist: hdbscan>=0.8.33; extra == "cluster"
Provides-Extra: dashboard
Requires-Dist: streamlit>=1.30; extra == "dashboard"
Requires-Dist: pandas>=2.0; extra == "dashboard"
Requires-Dist: pillow>=9.0; extra == "dashboard"
Requires-Dist: pypdf>=4.0; extra == "dashboard"
Requires-Dist: pyarrow>=14.0; extra == "dashboard"
Requires-Dist: openpyxl>=3.1; extra == "dashboard"
Provides-Extra: gemini
Requires-Dist: google-genai>=1.0; extra == "gemini"
Provides-Extra: all
Requires-Dist: torch>=2.0.0; extra == "all"
Requires-Dist: transformers<5.0.0,>=4.51.0; extra == "all"
Requires-Dist: accelerate>=0.26.0; extra == "all"
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: captum>=0.7; extra == "all"
Requires-Dist: inseq>=0.6; extra == "all"
Requires-Dist: nnsight>=0.3; extra == "all"
Requires-Dist: matplotlib>=3.7; extra == "all"
Requires-Dist: statsmodels>=0.14; extra == "all"
Requires-Dist: scikit-learn>=1.3; extra == "all"
Requires-Dist: hdbscan>=0.8.33; extra == "all"
Requires-Dist: streamlit>=1.30; extra == "all"
Requires-Dist: pandas>=2.0; extra == "all"
Requires-Dist: pillow>=9.0; extra == "all"
Requires-Dist: pypdf>=4.0; extra == "all"
Requires-Dist: pyarrow>=14.0; extra == "all"
Requires-Dist: openpyxl>=3.1; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: jsonschema>=4.0; extra == "dev"
Dynamic: license-file

<div align="center">

# EvalVitals

### Your eval tells you *what* failed. EvalVitals investigates *why*—and tests what fixes it.

[![PyPI version](https://img.shields.io/pypi/v/evalvitals?color=0A7BBC)](https://pypi.org/project/evalvitals/)
[![Python versions](https://img.shields.io/pypi/pyversions/evalvitals)](https://pypi.org/project/evalvitals/)
[![CI](https://github.com/evalvitals/evalvitals/actions/workflows/ci.yml/badge.svg)](https://github.com/evalvitals/evalvitals/actions/workflows/ci.yml)
[![Docs](https://img.shields.io/badge/docs-live-6C63FF)](https://evalvitals.github.io/evalvitals/)
[![License: CC0-1.0](https://img.shields.io/badge/license-CC0--1.0-green)](LICENSE)

[Get started](#quickstart-analyze-your-eval-logs) · [Documentation](https://evalvitals.github.io/evalvitals/) · [Examples](examples/README.md) · [PyPI](https://pypi.org/project/evalvitals/)

</div>

Most evaluation tools end with a score and a table of failures. EvalVitals
starts there. It can explore raw eval logs, find recurring patterns, propose
falsifiable explanations, test them on held-out cases, and validate candidate
interventions against the unchanged baseline.

Use the whole investigation loop—or take only the layer you need:

- **Explore existing results:** point a coding agent at arbitrary JSON/JSONL
  logs and get observations, charts, tables, and testable hypotheses.
- **Investigate a target model:** adaptively probe failures, diagnose systematic
  modes, verify hypotheses, and test interventions.
- **Run one focused analyzer:** use the same sklearn-like interface for
  black-box APIs and white-box local models.

```mermaid
flowchart LR
    A[Eval logs or target model] --> B[Probe]
    B --> C[Explore patterns]
    C --> D[Diagnose]
    D --> E{Held-out verification}
    E -->|supported| F[Test interventions]
    E -->|refuted| B
    F --> G[Validated fix or honest inconclusive result]
```

## Quickstart: Analyze Your Eval Logs

Install EvalVitals with the report dashboard:

```bash
pip install "evalvitals[dashboard]"
```

Then point it at a file or directory of JSON/JSONL results:

```bash
evalvitals explore ./results \
  --backend codex \
  -q "What distinguishes failed cases from successful ones?" \
  --dashboard
```

`codex` can be replaced with `claude_code`, `opencode`, `gemini_cli`,
`kimi_cli`, or `antigravity`. The selected coding-agent CLI must be installed
and authenticated separately.

Prefer a browser? `evalvitals web` serves a local data-analysis workbench:
drop a **.zip** containing JSON/JSONL/CSV/TSV/Parquet/Excel data and/or images,
PDFs, audio, or video. Each upload becomes a persistent data thread: the page
shows ingestion and M2/M3 progress, renders M2 as soon as it finishes, and
accepts artifact-grounded follow-up questions without uploading again.

EvalVitals writes an auditable analysis bundle instead of returning only prose:

```text
evalvitals_explore_output/
├── exploratory_report.json   # observations, candidate signals, hypotheses
├── records.json              # normalized records used by the analysis
├── figures/                  # rendered charts
├── tables/                   # analysis-ready tables
└── analysis.py               # the generated code that was actually run
```

**A real bundled run:** on the synthetic-yield example, Explore identified
temperature as the strongest observed correlate (`r = 0.86`), separated that
finding from weak pressure evidence (`r = -0.21`), and proposed mechanism-level
hypotheses for a later confirmatory experiment. [See the reproducible example →](examples/m2_statistics/synthetic_yield_explore/)

Already have your own analysis code? Use the analyzer toolkit directly, or
feed the resulting cases into the full diagnosis loop. EvalVitals does not
require you to replace your existing eval or observability stack.

## What Makes It Different

| Typical eval workflow | EvalVitals |
|---|---|
| Aggregate a metric | Investigate the cases behind the metric |
| Browse failures manually | Search for recurring, structured failure modes |
| Accept an LLM explanation | Turn explanations into falsifiable hypotheses |
| Test on the same cases used for discovery | Separate exploration from held-out confirmation |
| Report a promising prompt rewrite | Compare interventions with the unchanged baseline |
| Choose either API-level or internal analysis | Negotiate black-box and white-box capabilities through one interface |

Statistical gates use paired tests and e-values, including multiplicity control
when several hypotheses or fixes are tried. A run may end **inconclusive**;
EvalVitals does not turn weak evidence into a success verdict.

## Three Ways to Use EvalVitals

### 1. Explore — raw results to testable hypotheses

`evalvitals explore` recursively samples arbitrary JSON/JSONL shapes. The
coding agent performs exploratory data analysis; the host records generated
code, adjudicates host-checkable statistics, renders figures, and proposes
1–3 falsifiable hypotheses.

[Explore guide →](docs/m2_analysis.md)

### 2. Investigate — failures to verified interventions

`VLDiagnoseLoop` chains the full workflow:

```text
M1 targeted probes
 → M2 exploratory and statistical analysis
 → M3 diagnosis hypotheses
 → M5 held-out hypothesis verification
 → M4 surgery and tiered fixes
```

Interventions can range from prompt changes and scaffolds to read/write access
to model internals. Each candidate is evaluated against the unmodified
baseline; automatic escalation happens only when explicitly enabled.

[Full-loop quickstart →](docs/quickstart.md#vldiagnoseloop--automated-failure-attribution-current) ·
[Intervention guide →](docs/intervention.md)

### 3. Analyze — one model, one question

Every registered analyzer follows the same call shape:

```python
from evalvitals import Capability, compose
from evalvitals.analyzers.attention.summary import AttentionAnalyzer

model = compose(
    "qwen2.5-7b-instruct",
    "hf_local",
    want={Capability.ATTENTION},
)

result = AttentionAnalyzer(layer=-1, top_k=5).run(
    model, "The Eiffel Tower is in"
)

print(result.summary())
```

The analyzer zoo includes attention, uncertainty, hallucination, attribution,
logit-lens, representation-geometry, and agent-trajectory analysis.

[Browse the Analyzer Zoo →](docs/analyzers.md)

## Installation

The core install stays lightweight—no Torch required:

```bash
pip install evalvitals
```

Add only the capabilities you need:

```bash
pip install "evalvitals[api]"        # OpenAI-compatible API models
pip install "evalvitals[local]"      # local Hugging Face models + Torch
pip install "evalvitals[interp]"     # interpretability toolchains
pip install "evalvitals[viz]"        # plots
pip install "evalvitals[dashboard]"  # Streamlit reports
pip install "evalvitals[stats]"      # inferential statistics
```

For development:

```bash
git clone https://github.com/evalvitals/evalvitals.git
cd evalvitals
pip install -e ".[dev]"
pytest -m "not gpu"
```

## Architecture in One Minute

Model identity is separate from runtime, and analyzers declare the
capabilities they need. The same model spec can run through a black-box API or
a white-box local backend; only the available capability set changes.

| Contract | Role |
|---|---|
| `ModelSpec` | Model identity: family, repository, architecture traits, modalities. |
| `Backend` | Runtime: local internals, black-box API, or offline batch engine. |
| `Model` | Runnable model with generation and optional internal capture. |
| `Analyzer` | `Analyzer(**params).run(model, data) -> Result`. |
| `Capability` | Matches analyzers to compatible model runtimes before execution. |
| `FailureCase` | Prompts, labels, provenance, metadata, and agent trajectories. |
| `Result` | Human-readable summary plus structured, serializable findings. |

[Read the architecture guide →](docs/architecture.md)

## Reproducible Examples

| Example | What it demonstrates |
|---|---|
| [`synthetic_yield_explore`](examples/m2_statistics/synthetic_yield_explore/) | Standalone Explore on structured tabular outcomes. |
| [`deco_hallu_explore`](examples/m2_statistics/deco_hallu_explore/) | Explore → held-out hypothesis tests → tiered repair. |
| [`deco_hallu`](examples/diagnosis_loops/deco_hallu/) | Decoupled multimodal hallucination diagnosis and intervention. |
| [`qwen_attention`](examples/analyzer_demos/qwen_attention/) | White-box attention analysis on a local model. |

[See all examples →](examples/README.md)

## Documentation

- [Quickstart](docs/quickstart.md)
- [Exploratory Analysis](docs/m2_analysis.md)
- [Intervention & Verification](docs/intervention.md)
- [Analyzer Zoo](docs/analyzers.md)
- [Architecture](docs/architecture.md)
- [Extending EvalVitals](docs/extending.md)
- [Roadmap](docs/roadmap.md)

## Project Status

EvalVitals is an early-stage research toolkit. Interfaces may evolve, and some
full-loop examples require model weights, a GPU, or an external coding-agent
CLI. Bug reports, reproducible failure cases, analyzer contributions, and
evaluation integrations are welcome.

If EvalVitals helps you understand a model failure, consider starring the repo
and sharing the smallest reproducible case—it makes the toolkit better for the
next investigation.
