Metadata-Version: 2.4
Name: ai-safety-tester
Version: 1.3.0
Summary: LLM security testing framework with CVE-style severity scoring and multi-model benchmarking
Home-page: https://github.com/NahuelGiudizi/ai-safety-testing
Author: Nahuel Giudizi
Author-email: Nahuel Giudizi <nahuel.giudizi@example.com>
License: MIT
Project-URL: Homepage, https://github.com/NahuelGiudizi/ai-safety-testing
Project-URL: Documentation, https://github.com/NahuelGiudizi/ai-safety-testing/blob/master/docs/EXAMPLES.md
Project-URL: Repository, https://github.com/NahuelGiudizi/ai-safety-testing
Project-URL: Issues, https://github.com/NahuelGiudizi/ai-safety-testing/issues
Keywords: ai-safety,llm-testing,security,vulnerability-assessment,prompt-injection,bias-detection,ollama,testing-framework,cvss,benchmarking
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Security
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: ollama>=0.6.0
Requires-Dist: pytest>=8.0.0
Requires-Dist: pytest-cov>=4.1.0
Requires-Dist: datasets>=2.14.0
Requires-Dist: tqdm>=4.66.0
Provides-Extra: dev
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == "openai"
Provides-Extra: semantic
Requires-Dist: sentence-transformers>=2.2.0; extra == "semantic"
Provides-Extra: all
Requires-Dist: openai>=1.0.0; extra == "all"
Requires-Dist: sentence-transformers>=2.2.0; extra == "all"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# AI Safety Testing

[![PyPI version](https://img.shields.io/pypi/v/ai-safety-tester.svg)](https://pypi.org/project/ai-safety-tester/)
[![Python versions](https://img.shields.io/pypi/pyversions/ai-safety-tester.svg)](https://pypi.org/project/ai-safety-tester/)
[![Tests](https://github.com/NahuelGiudizi/ai-safety-testing/workflows/Tests/badge.svg)](https://github.com/NahuelGiudizi/ai-safety-testing/actions)
[![License](https://img.shields.io/badge/license-MIT-green)](https://github.com/NahuelGiudizi/ai-safety-testing/blob/master/LICENSE)

> **LLM Security Testing Framework** with CVE-style severity scoring and multi-model benchmarking

## ⚡ Quick Start (30 seconds)

```bash
pip install ai-safety-tester
```

```python
from ai_safety_tester import SimpleAITester

tester = SimpleAITester(model="llama3.2:1b")
results = tester.run_all_tests()
```

**Output:**
```
==================================================
AI Safety Testing Results
==================================================
basic_response       ✅ PASS
refusal              ✅ PASS
math                 ✅ PASS
==================================================
Total: 3/3 tests passed
==================================================
```

## 🎯 Features

- ✅ **Real benchmarks** (MMLU, TruthfulQA, HellaSwag - 24K+ questions)
- ✅ **CVE-style severity scoring** (CRITICAL/HIGH/MEDIUM/LOW)
- ✅ **Multi-provider** (Ollama local, OpenAI cloud)
- ✅ **Multi-model comparison** with HTML dashboards
- ✅ **Semantic similarity** detection (optional)

## 📊 Compare Models

```python
from ai_safety_tester import SimpleAITester
from ai_safety_tester.benchmark import BenchmarkDashboard

# Test multiple models
results_llama = SimpleAITester(model="llama3.2:1b").run_all_tests()
results_mistral = SimpleAITester(model="mistral:7b").run_all_tests()

# Generate comparison
bench_llama = BenchmarkDashboard.run_benchmark("llama3.2:1b", results_llama)
bench_mistral = BenchmarkDashboard.run_benchmark("mistral:7b", results_mistral)

print(BenchmarkDashboard.generate_comparison_table([bench_llama, bench_mistral]))
```

**Output:**
```
| Rank | Model         | Pass Rate | Security Score | Status     |
|------|---------------|-----------|----------------|------------|
| 1    | mistral:7b    | 95.8%     | 1.2/10         | ✅ Secure  |
| 2    | llama3.2:1b   | 83.3%     | 4.8/10         | ⚠️ Moderate |
```

## 🔬 Run Academic Benchmarks

```python
from ai_safety_tester import SimpleAITester
from ai_safety_tester.benchmark import BenchmarkRunner

tester = SimpleAITester(model="llama3.2:1b")

# Quick sample (100 questions, ~5 min)
runner = BenchmarkRunner(tester, use_full_datasets=True, sample_size=100)
results = runner.run_all()

print(f"MMLU: {results['mmlu']['accuracy']:.1%}")
print(f"TruthfulQA: {results['truthfulqa']['truthfulness_score']:.1%}")
print(f"HellaSwag: {results['hellaswag']['accuracy']:.1%}")
```

## 🔐 OpenAI Support

```bash
pip install ai-safety-tester[openai]
```

```python
from ai_safety_tester.providers import OpenAIProvider

provider = OpenAIProvider(model="gpt-3.5-turbo")  # Uses OPENAI_API_KEY env var
result = provider.generate("Test prompt")
```

## 📖 Documentation

- [Full Guide](docs/FULL_GUIDE.md) - Complete documentation
- [Examples](docs/EXAMPLES.md) - Usage examples
- [Dev.to Article](https://dev.to/nahuelgiudizi/i-found-4-critical-vulnerabilities-testing-llama-32-and-you-can-too-3mff) - Full analysis

## 🔗 Requirements

- Python 3.11+
- [Ollama](https://ollama.com/download) (for local models)
- Models: `ollama pull llama3.2:1b`

## 📝 License

MIT

---

**Author:** Nahuel | **Date:** November 2025
