Metadata-Version: 2.4
Name: vowel
Version: 0.3.2
Summary: A modular evaluation framework for testing functions with YAML-based specifications
Project-URL: Homepage, https://github.com/fswair/vowel
Project-URL: Documentation, https://github.com/fswair/vowel#readme
Project-URL: Repository, https://github.com/fswair/vowel.git
Project-URL: Issues, https://github.com/fswair/vowel/issues
Author-email: Mert <contact@tomris.dev>
License: Apache-2.0
License-File: LICENSE
Keywords: evaluation,pydantic-evals,testing,yaml
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: click>=8.0.0
Requires-Dist: pydantic-ai>=1.0.0
Requires-Dist: pydantic-evals>=1.0.0
Requires-Dist: pydantic-monty>=0.0.7
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: rich
Requires-Dist: watchdog
Provides-Extra: all
Requires-Dist: basedpyright>=1.0.0; extra == 'all'
Requires-Dist: black>=23.0.0; extra == 'all'
Requires-Dist: logfire; extra == 'all'
Requires-Dist: mcp; extra == 'all'
Requires-Dist: mypy>=1.0.0; extra == 'all'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'all'
Requires-Dist: pytest-cov>=4.0.0; extra == 'all'
Requires-Dist: pytest>=7.0.0; extra == 'all'
Requires-Dist: ruff>=0.1.0; extra == 'all'
Requires-Dist: types-pyyaml; extra == 'all'
Requires-Dist: vowel-optimization; extra == 'all'
Provides-Extra: dev
Requires-Dist: basedpyright>=1.0.0; extra == 'dev'
Requires-Dist: black>=23.0.0; extra == 'dev'
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: logfire
Requires-Dist: logfire; extra == 'logfire'
Provides-Extra: mcp
Requires-Dist: mcp; extra == 'mcp'
Provides-Extra: optimize
Requires-Dist: vowel-optimization; extra == 'optimize'
Description-Content-Type: text/markdown

# VOWEL

**YAML-based evaluation framework for testing Python functions with AI-powered test generation and function healing.**

vowel makes it easy to define test cases in YAML and run them against your Python functions. It also provides AI-powered generators that can automatically create test specs, generate implementations, and fix buggy functions.

[![PyPI Downloads](https://static.pepy.tech/personalized-badge/vowel?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/vowel) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/fswair/vowel) 

## Installation

```bash
pip install vowel

# Or with uv
uv add vowel
```

### Development

```bash
git clone https://github.com/fswair/vowel.git
cd vowel
pip install -e .
```

---

## Quick Start

### 1. Create a YAML spec

```yaml
# evals.yml
add:
  dataset:
    - case:
        inputs: { x: 1, y: 2 }
        expected: 3
    - case:
        inputs: { x: -5, y: 5 }
        expected: 0

divide:
  evals:
    Type:
      type: "float"
  dataset:
    - case:
        inputs: { a: 10, b: 2 }
        expected: 5.0
    - case:
        inputs: { a: 1, b: 0 }
        raises: ZeroDivisionError
```

### 2. Run from CLI

```bash
vowel evals.yml
```

### 3. Or programmatically

```python
from vowel import run_evals

def add(x: int, y: int) -> int:
    return x + y

def divide(a: float, b: float) -> float:
    return a / b

summary = run_evals("evals.yml", functions={"add": add, "divide": divide})
print(f"All passed: {summary.all_passed}")
print(f"Coverage: {summary.coverage * 100:.1f}%")
```

### 4. Or use the fluent API

```python
from vowel import RunEvals

summary = (
    RunEvals.from_file("evals.yml")
    .with_functions({"add": add, "divide": divide})
    .filter(["add"])
    .debug()
    .run()
)

summary.print()
```

---

## Features

### Evaluators

8 built-in evaluators for flexible testing:

| Evaluator | Purpose |
|-----------|---------|
| **Expected** | Exact value matching |
| **Type** | Return type checking (strict/lenient) |
| **Assertion** | Custom Python expressions (`output > 0`, `output == input * 2`) |
| **Duration** | Performance constraints (function-level & case-level) |
| **Pattern** | Regex validation on output |
| **ContainsInput** | Verify output contains the input |
| **Raises** | Exception class + optional message matching |
| **LLMJudge** | AI-powered rubric evaluation |

```yaml
factorial:
  evals:
    Assertion:
      assertion: "output > 0"
    Type:
      type: "int"
    Duration:
      duration: 1.0
  dataset:
    - case: { input: 0, expected: 1 }
    - case: { input: 5, expected: 120 }
```

> **Full reference:** [docs/EVALUATORS.md](https://github.com/fswair/vowel/blob/main/docs/EVALUATORS.md)

### Fixtures (Dependency Injection)

Inject databases, temp files, caches into functions under test. Three patterns: generator (yield), tuple (setup/teardown), simple (setup only).

```yaml
fixtures:
  db:
    setup: myapp.setup_db
    teardown: myapp.close_db
    scope: module

query_user:
  fixture: [db]
  dataset:
    - case:
        inputs: { user_id: 1 }
        expected: { name: "Alice" }
```

```python
def query_user(user_id: int, *, db: dict) -> dict | None:
    return db["users"].get(user_id)
```

> **Full reference:** [docs/FIXTURES.md](https://github.com/fswair/vowel/blob/main/docs/FIXTURES.md)

### Input Serializers

Transform YAML inputs into Pydantic models, dates, or custom types:

```python
summary = (
    RunEvals.from_file("evals.yml")
    .with_functions({"get_user": get_user})
    .with_serializer({"get_user": User})      # Schema mode
    .run()
)
```

> **Full reference:** [docs/SERIALIZERS.md](https://github.com/fswair/vowel/blob/main/docs/SERIALIZERS.md)

### AI-Powered Generation

#### EvalGenerator — test existing functions

```python
from vowel import EvalGenerator, Function

generator = EvalGenerator(model="openai:gpt-4o", load_env=True)
func = Function.from_callable(my_function)

result = generator.generate_and_run(func, auto_retry=True, heal_function=True)
print(f"Coverage: {result.summary.coverage * 100:.1f}%")
```

#### TDDGenerator — generate everything from a description

```python
from vowel.tdd import TDDGenerator

generator = TDDGenerator(model="gemini-3-flash-preview", load_env=True)

result = generator.generate_all(
    description="Binary search for target in sorted list. Returns index or -1.",
    name="binary_search"
)

result.print()  # Shows: signature → tests → code → results
```

Step-by-step control:

```python
signature = generator.generate_signature(description="...", name="factorial")
runner, yaml_spec = generator.generate_evals_from_signature(signature)
func = generator.generate_implementation(signature, yaml_spec)
summary = runner.with_functions({"factorial": func.impl}).run()
```

> **Full reference:** [docs/AI_GENERATION.md](https://github.com/fswair/vowel/blob/main/docs/AI_GENERATION.md)

### MCP Server

Expose vowel's capabilities to AI assistants like Claude Desktop via Model Context Protocol.

> **Setup guide:** [docs/MCP.md](https://github.com/fswair/vowel/blob/main/docs/MCP.md)

---

## CLI

```bash
vowel evals.yml                          # Run single file
vowel -d ./tests                         # Run directory
vowel evals.yml -f add,divide            # Filter functions
vowel evals.yml --ci --coverage 90       # CI mode
vowel evals.yml --watch                  # Watch mode
vowel evals.yml --dry-run                # Show plan without running
vowel evals.yml --export-json out.json   # Export results
```

> **Full reference:** [docs/CLI.md](https://github.com/fswair/vowel/blob/main/docs/CLI.md)

---

## EvalSummary

```python
summary = run_evals("evals.yml", functions={...})

summary.all_passed       # bool
summary.success_count    # int
summary.failed_count     # int
summary.total_count      # int
summary.coverage         # float (0.0-1.0)
summary.failed_results   # list[EvalResult]

summary.meets_coverage(0.9)    # Check threshold
summary.print()                # Rich formatted output
summary.to_json()              # Export as dict
summary.xml()                  # Export as XML
```

---

## Documentation

| Document | Description |
|----------|-------------|
| [YAML Spec](https://github.com/fswair/vowel/blob/main/docs/YAML_SPEC.md) | Complete YAML format reference |
| [Evaluators](https://github.com/fswair/vowel/blob/main/docs/EVALUATORS.md) | All 8 evaluator types |
| [Fixtures](https://github.com/fswair/vowel/blob/main/docs/FIXTURES.md) | Dependency injection guide |
| [Serializers](https://github.com/fswair/vowel/blob/main/docs/SERIALIZERS.md) | Input serializer patterns |
| [AI Generation](https://github.com/fswair/vowel/blob/main/docs/AI_GENERATION.md) | EvalGenerator & TDDGenerator |
| [CLI](https://github.com/fswair/vowel/blob/main/docs/CLI.md) | Command-line reference |
| [MCP Server](https://github.com/fswair/vowel/blob/main/docs/MCP.md) | AI assistant integration |
| [Troubleshooting](https://github.com/fswair/vowel/blob/main/docs/TROUBLESHOOTING.md) | Common errors & solutions |

---

## License

Apache License 2.0
