Metadata-Version: 2.4
Name: evalform
Version: 0.1.0a2
Summary: Declarative orchestration for AI evaluation libraries
Author: Vasundhra Sharma
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Vasundhra02/evalform
Project-URL: Repository, https://github.com/Vasundhra02/evalform
Project-URL: Issues, https://github.com/Vasundhra02/evalform/issues
Keywords: llm,evaluation,ragas,promptfoo,ai,testing,orchestration
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Requires-Dist: cel-python>=0.1.5
Requires-Dist: jsonschema>=4.0
Provides-Extra: ragas
Requires-Dist: ragas<0.3,>=0.2; extra == "ragas"
Requires-Dist: langchain-openai>=0.2; extra == "ragas"
Provides-Extra: deepeval
Requires-Dist: deepeval<5,>=4.1; extra == "deepeval"
Dynamic: license-file

# EvalForm

[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![PyPI](https://img.shields.io/badge/PyPI-0.1.0a1-blue.svg)](https://pypi.org/project/evalform/0.1.0a1/)

**Declarative AI evaluation testing from a single YAML file.**

EvalForm lets you test an AI or RAG application from one YAML file. It sends
your test questions to your application, runs an evaluator, and fails a CI job
when a quality rule is not met.

## ⚠️ Alpha Release

**EvalForm is in active development.** While the core architecture is stable:
- Provider packs have been validated against live APIs
- Configuration schema may evolve based on user feedback  
- Docker images are published to GHCR
- Documentation is being actively expanded

**We welcome early adopters!** Please report issues and share feedback.

## Included providers

- RAGAS for RAG metrics such as faithfulness
- DeepEval for LLM evaluation metrics
- Promptfoo for red-team probes
- Mock for deterministic tests without an API key

EvalForm runs providers in Docker containers. You install EvalForm itself, but
you do not need to install RAGAS, DeepEval, Node.js, or Promptfoo locally.

## Prerequisites

- **Python 3.9-3.13** (Python 3.14+ has asyncio compatibility issues with RAGAS in local mode)
- **Docker Desktop or Docker Engine** (required for Docker mode with provider containers)
  - Optional: Use `--local` flag to run without Docker (faster, but requires compatible Python version)

## Install

### From PyPI
```bash
pip install evalform
# OR
python -m pip install evalform

# If using corporate pip config, install from public PyPI:
pip install evalform --index-url https://pypi.org/simple/

# Verify installation
evalform --version
#or 
python -m pip show evalform
evalform providers
```

**Windows Users**: If `evalform` command is not found:

**Option 1 - Add to PATH (Recommended):**
1. Search for "Environment Variables" in Windows
2. Edit your user PATH variable  
3. Add: `C:\Users\<YourUsername>\AppData\Roaming\Python\Python3XX\Scripts`
4. Restart your terminal
5. Now you can use `evalform` directly

**Option 2 - Use full path each time:**
```powershell
& "C:\Users\<YourUsername>\AppData\Roaming\Python\Python3XX\Scripts\evalform.exe" providers
```

**Note**: If you see `cel-python` errors, install it:
```bash
pip install cel-python
```

### Docker Images

**Important**: Docker images are required for Docker mode execution.

#### Pull RAGAS Image (Required for RAG evaluation)

**For Apple Silicon (M1/M2/M3) Mac users:**
```bash
# Simple pull works natively
docker pull ghcr.io/vasundhra02/evalform-ragas:0.0.1

# Verify the image
docker images | grep evalform-ragas
```

**For Windows and Intel Mac users:**
```bash
# Pull with specific digest for platform compatibility
# Current digest: a6914c29eaf62f8395245323bdc7b3d3d7d270a0b0695b43f12bed3c430a708c
docker pull ghcr.io/vasundhra02/evalform-ragas:0.0.1@sha256:<digest>

# Tag it for evalform to find
docker tag ghcr.io/vasundhra02/evalform-ragas@sha256:<digest> ghcr.io/vasundhra02/evalform-ragas:0.0.1

# Verify the image
docker images | grep evalform-ragas
```

**Example with current digest:**
```bash
docker pull ghcr.io/vasundhra02/evalform-ragas:0.0.1@sha256:a6914c29eaf62f8395245323bdc7b3d3d7d270a0b0695b43f12bed3c430a708c
docker tag ghcr.io/vasundhra02/evalform-ragas@sha256:a6914c29eaf62f8395245323bdc7b3d3d7d270a0b0695b43f12bed3c430a708c ghcr.io/vasundhra02/evalform-ragas:0.0.1
```

#### Optional: Pull Other Provider Images

**For Apple Silicon (M1/M2/M3) Mac users:**
```bash
# DeepEval
docker pull ghcr.io/vasundhra02/evalform-deepeval:0.0.1

# Promptfoo
docker pull ghcr.io/vasundhra02/evalform-promptfoo:0.0.1
```

**For Windows and Intel Mac users:**
```bash
# DeepEval - check GitHub packages for latest digest
docker pull ghcr.io/vasundhra02/evalform-deepeval:0.0.1@sha256:<digest>
docker tag ghcr.io/vasundhra02/evalform-deepeval@sha256:<digest> ghcr.io/vasundhra02/evalform-deepeval:0.0.1

# Promptfoo - check GitHub packages for latest digest
docker pull ghcr.io/vasundhra02/evalform-promptfoo:0.0.1@sha256:<digest>
docker tag ghcr.io/vasundhra02/evalform-promptfoo@sha256:<digest> ghcr.io/vasundhra02/evalform-promptfoo:0.0.1
```

**Note**: Find the latest digests at https://github.com/Vasundhra02?tab=packages

**Platform Note**: 
- Images were built on **Apple Silicon (M1/M2/M3)** and are `linux/arm64` architecture
- **Apple Silicon users**: Images will run natively (no emulation needed)
- **Windows and Intel Mac users**: Docker will run them via emulation (x86_64 → arm64)
- This is why you need to use the specific digest - Docker can't auto-detect the right platform
- Performance may be slightly slower on Windows/Intel due to emulation, but functionality is identical
- **Future**: Multi-platform images (arm64 + amd64) will be published to eliminate this step

Create a starter configuration:

```bash
evalform init --kind mixed --name my-evaluation
```

This creates:
- `evalform.yaml` - Suite configuration
- `evalform-cases.jsonl` - Test cases

Now customize these files for your use case (see examples below).

## Add credentials

Create `.env` in the same directory as your suite:

### OpenAI
```env
OPENAI_API_KEY=sk-your-api-key
```

### Azure OpenAI
```env
LLM_API_KEY=your-azure-api-key
LLM_URL=https://your-resource.openai.azure.com
AZURE_OPENAI_API_VERSION=2024-02-01
```

**Important Notes:**
- EvalForm's `.env` loader does NOT support variable substitution (e.g., `${VAR_NAME}`)
- Use literal values only: `KEY=value` format
- Never commit `.env` files to version control
- For Docker mode, ensure API keys are passed via `execution.env_passthrough` in your suite YAML

**Azure OpenAI Configuration:**
The RAGAS provider checks for API keys in this order:
1. `config.azure_api_key` in the suite YAML
2. `LLM_API_KEY` environment variable
3. `AZURE_OPENAI_API_KEY` environment variable

You can either:
- Set the key directly in YAML: `config.azure_api_key: your-key` (not recommended for production)
- Use environment variables and pass them via `env_passthrough: [LLM_API_KEY]`

## Customize the Suite Configuration

After running `evalform init`, you need to customize `evalform.yaml` for your specific use case.

### Key Sections to Update:

#### 1. Target Configuration
Update the `target` section to point to your API:

```yaml
target:
  system: my-rag-app          # Your system name
  environment: local          # or staging, production
  runner:
    kind: http
    url: http://localhost:8000/chat    # Your API endpoint
    method: POST
    request_map:
      question: question      # Maps JSONL field to API request
    response_map:
      answer: answer          # Maps API response to evalform variable
      contexts: contexts      # For RAG systems
```

**Important**: 
- `request_map`: Maps test case fields → API request body
- `response_map`: Maps API response fields → variables for metrics
- When using `response_map`, test cases only need input fields (e.g., `question`)

#### 2. Metrics Configuration
Update the `metrics` section for your LLM provider:

**For Azure OpenAI:**
```yaml
metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini              # Your Azure deployment name
      azure_endpoint: https://your-resource.openai.azure.com
      azure_api_version: "2024-02-01"
      azure_api_key: your-key-here          # Or use env var
    map:
      question: question
      contexts: contexts
      answer: answer
```

**For OpenAI:**
```yaml
metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini
    map:
      question: question
      contexts: contexts
      answer: answer
```

#### 3. Test Cases Format
Update `evalform-cases.jsonl` based on your `response_map`:

**With response_map (recommended for live APIs):**
```jsonl
{"id": "case-1", "question": "What is your return policy?"}
{"id": "case-2", "question": "Do you ship internationally?"}
```
The `answer` and `contexts` will come from your API response.

**Without response_map (static test data):**
```jsonl
{"id": "case-1", "question": "What is your return policy?", "contexts": ["Returns accepted within 30 days."], "answer": "You can return items within 30 days."}
```

#### 4. Execution Mode
Choose between Docker or local mode:

```yaml
execution:
  mode: docker                    # Recommended: reproducible, Python 3.14 compatible
  env_passthrough: [LLM_API_KEY]  # Pass environment variables to container
```

OR

```yaml
execution:
  mode: local                     # Faster, but requires Python 3.9-3.13
```

## Suite Examples

### Example 1: RAG System with Live API (OpenAI)

```yaml
version: 1
suite: support-bot-quality

target:
  system: support-bot
  environment: staging
  runner:
    kind: http
    url: https://staging.example.com/chat
    method: POST
    request_map:
      question: question
    response_map:
      answer: answer
      contexts: contexts

test_data:
  source: file
  path: ./evalform-cases.jsonl

metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini
    map:
      question: question
      contexts: contexts
      answer: answer

policy:
  - name: faithfulness-floor
    when: "metric.id == 'faithfulness'"
    assert: "normalized.value >= 0.8"

execution:
  mode: docker
  env_passthrough: [OPENAI_API_KEY]
```

### Example 2: RAG System with Live API (Azure OpenAI)

```yaml
version: 1
suite: rag-chatbot-quality

target:
  system: rag-chatbot
  environment: local
  runner:
    kind: http
    url: http://localhost:8000/chat
    method: POST
    request_map:
      question: question
    response_map:
      answer: answer
      contexts: contexts

test_data:
  source: file
  path: ./evalform-cases.jsonl

metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini              # Azure deployment name
      azure_endpoint: https://your-resource.openai.azure.com
      azure_api_version: "2024-02-01"
      azure_api_key: your-key-here          # Or use LLM_API_KEY env var
    map:
      question: question
      contexts: contexts
      answer: answer

policy:
  - name: faithfulness-floor
    when: "metric.id == 'faithfulness'"
    assert: "normalized.value >= 0.7"

execution:
  mode: docker
  env_passthrough: [LLM_API_KEY]
```

**Test cases (evalform-cases.jsonl):**
```jsonl
{"id": "case-1", "question": "What is your return policy?"}
{"id": "case-2", "question": "Do you ship internationally?"}
```

### Test Cases

Each JSONL line is one test question:

```jsonl
{"id":"case-1","question":"What is your return policy?"}
{"id":"case-2","question":"Do you ship internationally?"}
```

**Note**: When using `response_map` to get `answer` and `contexts` from your API, you don't need to include them in the test cases. The test cases only need the input fields (e.g., `question`).

Change `provider` and `metric` to use DeepEval. For Promptfoo, use `mode: probe`
and set `target_endpoint`, `plugins`, and `num_probes` in the metric config.
The example suites in `examples/` show each provider.

## Run an evaluation

```bash
evalform plan --suite evalform.yaml   # Validate configuration (dry run)
evalform apply --suite evalform.yaml  # Run actual evaluation
```

`plan` checks the configuration without evaluator calls. `apply` runs your
target and evaluator, prints scores, applies policies, and saves history in
`.evalform/`.

**Note**: `plan` may show warnings about missing fields (like `contexts`) when using `response_map`. This is expected - those fields will be populated from the API response during `apply`.

For a smoke test without credentials:

```bash
evalform apply --suite examples/suite-mock.yaml --local --no-save
```

Exit codes are `0` for pass, `1` for a failed quality policy, and `2` for a
configuration, provider, credential, or execution error.

### Quick Troubleshooting

**"Azure OpenAI requires an API key" error:**
- Ensure `LLM_API_KEY` is set in `.env` file (no variable substitution like `${VAR}`)
- Add `LLM_API_KEY` to `execution.env_passthrough` list in your suite YAML
- OR set `config.azure_api_key` directly in the metric config (not recommended for production)

**"no matching manifest for linux/amd64" error:**
- Pull the image with the specific digest (see installation section above)
- Images are built for ARM64 but will run via emulation on x86_64

**Python 3.14 asyncio errors with local mode:**
- Use Docker mode instead: `execution.mode: docker`
- OR downgrade to Python 3.13 or earlier for local mode

**"evalform: command not found":**
- Add Python Scripts directory to PATH (see installation section)
- OR use full path to evalform.exe
- OR use `python -m evalform` instead

## CI

Store the API key as a CI secret:

```yaml
- run: pip install evalform
- run: evalform apply --suite evalform.yaml
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

Persist `.evalform/` between runs if you want baseline comparisons.

## Add a provider

Providers are plug-ins described by YAML manifests, not hard-coded into EvalForm. 

**Interactive mode** (recommended):
```bash
evalform provider create
```

**Non-interactive mode**:
```bash
evalform provider init trulens \
  --package trulens_eval \
  --image ghcr.io/your-org/evalform-trulens:0.1.0
```

This creates a provider manifest, dependency file, and fixtures. Edit the
manifest to declare the provider version, Docker image, required environment
variables, config fields, input mapping, and output-to-score mapping.

Validate and build it without changing EvalForm code:

```bash
evalform provider validate providers/trulens
evalform provider test providers/trulens      # Test with fixtures (no API key)
evalform provider doctor providers/trulens    # Pre-publish checks
evalform provider build providers/trulens --tag evalform/trulens:0.1.0
```

See **[docs/providers.md](docs/providers.md)** for the complete guide.

## Known Limitations

### Current Limitations

- **Test data sources**: Only `file` (JSONL) is supported. Live trace ingestion from observability platforms is planned.
- **Target runners**: Only HTTP POST is implemented. GraphQL, gRPC, and custom runners are planned.
- **Storage backends**: Only SQLite. Postgres support is planned for shared team baselines.
- **Provider coverage**: RAGAS, DeepEval, Promptfoo, and Mock are included. Community contributions for TruLens, Garak, LangSmith adapters are welcome.
- **Windows support**: Tested on Windows 11 with Docker Desktop. WSL2 backend recommended.
- **Docker-in-Docker**: If running EvalForm inside a container, bind mounts must be on a shared volume accessible to the host Docker daemon.

### Workarounds

**No Docker available?**
```bash
evalform apply --suite evalform.yaml --local
```
Note: Results are stamped with `execution_mode: local` and depend on your installed libraries.

**Baseline too stale?**
```yaml
baseline:
  strategy: rolling_window  # Average last N passing runs
  window: 5
```

**Provider not available?**
Create a custom provider pack (see [docs/providers.md](docs/providers.md)) or open an issue requesting it.

## Documentation

- **[Troubleshooting Guide](docs/troubleshooting.md)** - Common errors and solutions
- **[Provider Creation](docs/providers.md)** - Add new evaluation tools
- **[Examples](examples/)** - Sample suite configurations
- **[CHANGELOG](CHANGELOG.md)** - Version history
- **[CONTRIBUTING](CONTRIBUTING.md)** - Development guidelines

## Roadmap

**Beta (Q3 2024)**:
- Stabilize configuration schema
- Add more provider packs (TruLens, Garak)
- Postgres storage backend
- Web UI for result visualization

**v1.0**:
- Live trace ingestion
- GraphQL/gRPC target runners
- Hosted service option
- Performance optimizations for large test suites

See [GitHub Issues](https://github.com/Vasundhra02/evalform/issues) for detailed planning.

## Contributing

We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for:
- How to add provider packs (no Python required!)
- Development setup
- Testing guidelines
- Code of conduct

**Quick wins for contributors**:
- Add provider packs for your favorite eval tools
- Improve documentation
- Report bugs with minimal reproducers
- Share your suite configurations as examples

## Support

- **Issues**: https://github.com/Vasundhra02/evalform/issues
- **Discussions**: https://github.com/Vasundhra02/evalform/discussions
- **Email**: [Your support email if available]

## License

Apache-2.0 - See [LICENSE](LICENSE) for details.

## Acknowledgments

Built with:
- [CEL-Python](https://github.com/cloud-custodian/cel-python) for policy expressions
- [RAGAS](https://github.com/explodinggradients/ragas) for RAG evaluation
- [Promptfoo](https://github.com/promptfoo/promptfoo) for red-teaming
- [DeepEval](https://github.com/confident-ai/deepeval) for LLM evaluation
