Metadata-Version: 2.4
Name: vibeval
Version: 0.1.5
Summary: vibeval (Vibe Coding Eval) — AI application testing framework
Project-URL: Homepage, https://github.com/SandyKidYao/vibeval
Project-URL: Repository, https://github.com/SandyKidYao/vibeval
Project-URL: Issues, https://github.com/SandyKidYao/vibeval/issues
Author-email: SandyKid <yupengyao912@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# vibeval — Vibe Coding Eval

A fast evaluation framework for AI applications. Install Claude Code and the vibeval CLI to get an end-to-end workflow from code analysis to test generation to evaluation.

## What Problem Does It Solve

Traditional software testing frameworks cannot assess the quality of AI outputs; traditional AI evaluation platforms rely on dataset construction and cannot keep up with the pace of feature iteration. vibeval strikes a balance between the two:

- Analyze your code via VibeCoding to quickly generate synthetic data and test cases
- Deterministic rules + LLM semantic judgment for dual-layer evaluation
- Cross-version comparison to track quality changes over time
- Language-agnostic: generated test code adapts to your project's framework without depending on the vibeval package

## Prerequisites

- [Claude Code](https://claude.ai/code)
- Python 3.10+

## Installation

```bash
# Install the vibeval CLI
pip install vibeval

# Install the Claude Code plugin (run this inside Claude Code)
/install-plugin https://github.com/SandyKidYao/vibeval
```

## Usage

Before first use, verify that the LLM provider is set up correctly:

```bash
vibeval check
```

Then run everything inside Claude Code without ever leaving the conversation:

```
/vibeval-analyze  →  /vibeval-design  →  /vibeval-generate  →  /vibeval-run
                                                      ↑
                      Code changes → /vibeval-update ──────┘
```

### Analyze Code

```
/vibeval-analyze meeting_summary src/services/
```

Analyze source code to identify AI call sites, data flows, and mock points, and provide suggestions for improving testability.

### Design Tests

```
/vibeval-design meeting_summary
```

Design synthetic data specifications, judging criteria, and test structure. Each step produces editable intermediate files that you can review and modify.

### Generate Code and Data

```
/vibeval-generate meeting_summary
```

Generate a complete test suite (synthetic data + test code) using your project's test framework (pytest/vitest/jest/...).

### Run + Evaluate

```
/vibeval-run meeting_summary
```

All in one step: run tests → evaluate → diagnostic analysis → generate report.

### Update After Code Changes

```
/vibeval-update meeting_summary
```

Detect code changes and incrementally update the test suite and data.

### Cross-Version Comparison

```bash
# Statistical comparison
vibeval diff meeting_summary run_a run_b

# LLM deep comparison
vibeval compare meeting_summary run_a run_b
```

### Visual Report

```bash
vibeval report meeting_summary latest --open
```

Generate a self-contained HTML report covering test design, data, process traces, and judgment results.

### Other Commands

```bash
# Show evaluation summary
vibeval summary meeting_summary latest

# List features and runs
vibeval features
vibeval runs meeting_summary

# See all commands
vibeval --help
```

## License

MIT
