Metadata-Version: 2.4
Name: openvals
Version: 0.0.5
Summary: AI evaluation, validation, benchmarking, and security framework
Author: Vishwanath Akuthota
License: MIT
Project-URL: Homepage, https://github.com/vishwanathakuthota/openvals
Project-URL: Documentation, https://docs.openvalidations.com
Project-URL: Source, https://github.com/vishwanathakuthota/openvals
Keywords: ai,llm,evaluation,benchmarking,validation,ml,security
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.21
Requires-Dist: pandas>=1.3
Requires-Dist: scikit-learn>=1.0

# OpenVals

OpenVals is an open evaluation and benchmarking framework for LLMs, SLMs, and AI systems, designed to help organizations measure, compare, and trust AI models before deployment.

> Evaluate. Benchmark. Trust. Deploy with Confidence.

---

## 🚀 Why OpenVals?

Most AI evaluation tools stop at metrics.

OpenVals goes further:

- ✅ Aligns evaluation with business objectives
- ✅ Enables multi-model benchmarking
- ✅ Quantifies trust, risk, and performance
- ✅ Supports deployment decision-making

---

## 🔷 Core Capabilities

### 1. Model Evaluation
Evaluate model outputs against structured datasets using:

- Accuracy
- Semantic similarity
- Latency

---

### 2. Multi-Model Benchmarking
Compare multiple models under the same conditions:

- Side-by-side evaluation
- Normalized scoring
- Model ranking
- Performance insights

---

### 3. Scoring Engine

Weighted scoring aligned to business priorities:

[
\text{Trust Score} = \sum_{i=1}^{n} w_i \cdot m_i
]

- Customize weights per use case
- Balance accuracy, cost, and latency

---

### 4. Extensible Architecture

- Plug-and-play model adapters
- Custom metrics support
- Scalable evaluation pipelines

---

## 📦 Installation

bash pip install -e . 

---

## ⚡ Quick Start

### 1. Run Evaluation

bash openvals run --dataset examples/sample_eval.json 

---

### 2. Run Multi-Model Benchmark

bash openvals benchmark --dataset examples/sample_eval.json 

---

## 🧪 Example Dataset

json [   {     "id": "1",     "input": "hello",     "expected_output": "olleh"   } ] 

---

## 🧠 Python API Usage

python from openvals.core.evaluator import Evaluator from openvals.models.dummy_model import DummyModel from openvals.datasets.loader import load_dataset  dataset = load_dataset("examples/sample_eval.json") model = DummyModel()  evaluator = Evaluator(model, dataset) result = evaluator.run()  print(result) 

---

## 🔷 Multi-Model Benchmarking Example

python from openvals.benchmarking.benchmark import BenchmarkRunner from openvals.benchmarking.normalization import normalize_scores from openvals.benchmarking.ranking import rank_models  models = {     "model_a": DummyModel(),     "model_b": DummyModel() }  runner = BenchmarkRunner(models, dataset) results = runner.run()  normalized = normalize_scores(results)  ranking = rank_models(normalized, {     "accuracy": 0.5,     "semantic": 0.3,     "latency": 0.2 })  print(ranking) 

---

## 🏗️ Project Structure

openvals/ │ ├── core/              # Evaluation engine ├── models/            # Model adapters ├── datasets/          # Dataset loading & schema ├── metrics/           # Evaluation metrics ├── benchmarking/      # Multi-model benchmarking layer ├── scoring/           # Scoring logic ├── safety/            # Risk & safety checks (WIP) ├── reporting/         # Output & reports (WIP) ├── cli.py             # Command-line interface

---

## 🔐 Roadmap

### v0.3
- [ ] OpenAI / Ollama / HuggingFace adapters
- [ ] Config-driven benchmarking (config.yaml)
- [ ] Parallel execution

### v0.4
- [ ] Safety layer (prompt injection, hallucination)
- [ ] Cost tracking
- [ ] Advanced reporting

### v1.0
- [ ] Trust Score engine
- [ ] Industry-specific benchmarks
- [ ] Certification layer

---

## 🌐 Vision

OpenVals is evolving into:

> A Trust Layer for AI Systems

Where organizations can answer:

- Which model should be deployed?
- Is it safe?
- Is it aligned with business goals?
- Can it be trusted in production?

---

## 🤝 Contributing

Contributions are welcome.

- Fork the repo
- Create a feature branch
- Submit a pull request

---

## 📄 License

MIT License

---

## 🧠 Backed by

Developed as part of DrPinnacle’s AI Trust & Validation Initiative, focused on building secure, scalable, and trustworthy AI systems.

---

## ⚡ Final Thought

AI models are easy to build.

Trusting them is the hard part.

OpenVals exists to solve that.
