Metadata-Version: 2.4
Name: evalform
Version: 0.1.0a1
Summary: Declarative orchestration for AI evaluation libraries
Author: Vasundhra Sharma
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Vasundhra02/evalform
Project-URL: Repository, https://github.com/Vasundhra02/evalform
Project-URL: Issues, https://github.com/Vasundhra02/evalform/issues
Keywords: llm,evaluation,ragas,promptfoo,ai,testing,orchestration
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Requires-Dist: cel-python>=0.1.5
Requires-Dist: jsonschema>=4.0
Provides-Extra: ragas
Requires-Dist: ragas<0.3,>=0.2; extra == "ragas"
Requires-Dist: langchain-openai>=0.2; extra == "ragas"
Provides-Extra: deepeval
Requires-Dist: deepeval<5,>=4.1; extra == "deepeval"
Dynamic: license-file

# EvalForm

[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![PyPI version](https://img.shields.io/pypi/v/evalform.svg)](https://pypi.org/project/evalform/)

**Declarative AI evaluation testing from a single YAML file.**

EvalForm lets you test an AI or RAG application from one YAML file. It sends
your test questions to your application, runs an evaluator, and fails a CI job
when a quality rule is not met.

## ⚠️ Alpha Release

**EvalForm is in active development.** While the core architecture is stable:
- Provider packs have been validated against live APIs
- Configuration schema may evolve based on user feedback  
- Docker images are published to GHCR
- Documentation is being actively expanded

**We welcome early adopters!** Please report issues and share feedback.

## Included providers

- RAGAS for RAG metrics such as faithfulness
- DeepEval for LLM evaluation metrics
- Promptfoo for red-team probes
- Mock for deterministic tests without an API key

EvalForm runs providers in Docker containers. You install EvalForm itself, but
you do not need to install RAGAS, DeepEval, Node.js, or Promptfoo locally.

## Prerequisites

- **Python 3.9 or later**
- **Docker Desktop or Docker Engine** (for reproducible execution)
  - Optional: Use `--local` flag to run without Docker (faster, but results depend on local installs)

## Install

```bash
pip install evalform
evalform --version
evalform providers
```

**Note**: If you see `cel-python` errors, install it:
```bash
pip install cel-python
```

Create a starter configuration:

```bash
evalform init --kind mixed --name my-evaluation
```

This creates a suite YAML file and a JSONL test-case file. Edit the generated
files with your target URL, response fields, metrics, and quality thresholds.

## Add credentials

Create `.env` in the same directory as your suite:

```env
OPENAI_API_KEY=your-api-key
```

EvalForm passes only variables declared by the provider to its container. Never
commit `.env` or share your key. Azure OpenAI users can configure the Azure
endpoint, deployment, API version, and key in the metric config and environment.

## Create a suite

Example RAGAS suite:

```yaml
version: 1
suite: support-bot-quality

target:
  system: support-bot
  environment: staging
  runner:
    kind: http
    url: https://staging.example.com/chat
    method: POST
    request_map:
      question: question
    response_map:
      answer: answer
      contexts: contexts

test_data:
  source: file
  path: ./evalform-cases.jsonl

metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini
    map:
      question: question
      contexts: contexts
      answer: answer

policy:
  - name: faithfulness-floor
    when: "metric.id == 'faithfulness'"
    assert: "normalized.value >= 0.8"

execution:
  mode: docker
  docker:
    env_passthrough: [OPENAI_API_KEY]
```

Each JSONL line is one test question:

```jsonl
{"id":"case-1","question":"What is your return policy?"}
{"id":"case-2","question":"Do you ship internationally?"}
```

Change `provider` and `metric` to use DeepEval. For Promptfoo, use `mode: probe`
and set `target_endpoint`, `plugins`, and `num_probes` in the metric config.
The example suites in `examples/` show each provider.

## Run an evaluation

```bash
evalform plan --suite evalform.yaml
evalform apply --suite evalform.yaml
```

`plan` checks the configuration without evaluator calls. `apply` runs your
target and evaluator, prints scores, applies policies, and saves history in
`.evalform/`.

For a smoke test without credentials:

```bash
evalform apply --suite examples/suite-mock.yaml --local --no-save
```

Exit codes are `0` for pass, `1` for a failed quality policy, and `2` for a
configuration, provider, credential, or execution error.

## CI

Store the API key as a CI secret:

```yaml
- run: pip install evalform
- run: evalform apply --suite evalform.yaml
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

Persist `.evalform/` between runs if you want baseline comparisons.

## Add a provider

Providers are plug-ins described by YAML manifests, not hard-coded into EvalForm. 

**Interactive mode** (recommended):
```bash
evalform provider create
```

**Non-interactive mode**:
```bash
evalform provider init trulens \
  --package trulens_eval \
  --image ghcr.io/your-org/evalform-trulens:0.1.0
```

This creates a provider manifest, dependency file, and fixtures. Edit the
manifest to declare the provider version, Docker image, required environment
variables, config fields, input mapping, and output-to-score mapping.

Validate and build it without changing EvalForm code:

```bash
evalform provider validate providers/trulens
evalform provider test providers/trulens      # Test with fixtures (no API key)
evalform provider doctor providers/trulens    # Pre-publish checks
evalform provider build providers/trulens --tag evalform/trulens:0.1.0
```

See **[docs/providers.md](docs/providers.md)** for the complete guide.

## Known Limitations

### Current Limitations

- **Test data sources**: Only `file` (JSONL) is supported. Live trace ingestion from observability platforms is planned.
- **Target runners**: Only HTTP POST is implemented. GraphQL, gRPC, and custom runners are planned.
- **Storage backends**: Only SQLite. Postgres support is planned for shared team baselines.
- **Provider coverage**: RAGAS, DeepEval, Promptfoo, and Mock are included. Community contributions for TruLens, Garak, LangSmith adapters are welcome.
- **Windows support**: Tested on Windows 11 with Docker Desktop. WSL2 backend recommended.
- **Docker-in-Docker**: If running EvalForm inside a container, bind mounts must be on a shared volume accessible to the host Docker daemon.

### Workarounds

**No Docker available?**
```bash
evalform apply --suite evalform.yaml --local
```
Note: Results are stamped with `execution_mode: local` and depend on your installed libraries.

**Baseline too stale?**
```yaml
baseline:
  strategy: rolling_window  # Average last N passing runs
  window: 5
```

**Provider not available?**
Create a custom provider pack (see [docs/providers.md](docs/providers.md)) or open an issue requesting it.

## Documentation

- **[Troubleshooting Guide](docs/troubleshooting.md)** - Common errors and solutions
- **[Provider Creation](docs/providers.md)** - Add new evaluation tools
- **[Examples](examples/)** - Sample suite configurations
- **[CHANGELOG](CHANGELOG.md)** - Version history
- **[CONTRIBUTING](CONTRIBUTING.md)** - Development guidelines

## Roadmap

**Beta (Q3 2024)**:
- Stabilize configuration schema
- Add more provider packs (TruLens, Garak)
- Postgres storage backend
- Web UI for result visualization

**v1.0**:
- Live trace ingestion
- GraphQL/gRPC target runners
- Hosted service option
- Performance optimizations for large test suites

See [GitHub Issues](https://github.com/Vasundhra02/evalform/issues) for detailed planning.

## Contributing

We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for:
- How to add provider packs (no Python required!)
- Development setup
- Testing guidelines
- Code of conduct

**Quick wins for contributors**:
- Add provider packs for your favorite eval tools
- Improve documentation
- Report bugs with minimal reproducers
- Share your suite configurations as examples

## Support

- **Issues**: https://github.com/Vasundhra02/evalform/issues
- **Discussions**: https://github.com/Vasundhra02/evalform/discussions
- **Email**: [Your support email if available]

## License

Apache-2.0 - See [LICENSE](LICENSE) for details.

## Acknowledgments

Built with:
- [CEL-Python](https://github.com/cloud-custodian/cel-python) for policy expressions
- [RAGAS](https://github.com/explodinggradients/ragas) for RAG evaluation
- [Promptfoo](https://github.com/promptfoo/promptfoo) for red-teaming
- [DeepEval](https://github.com/confident-ai/deepeval) for LLM evaluation
