Metadata-Version: 2.4
Name: alquimia-fair-forge
Version: 3.0.0b5
Summary: Alquimia Fair Forge library - Performance measurement for AI models and assistants
Project-URL: Homepage, https://github.com/Alquimia-ai/fair-forge
Project-URL: Repository, https://github.com/Alquimia-ai/fair-forge.git
Project-URL: Documentation, https://github.com/Alquimia-ai/fair-forge#readme
Project-URL: Bug Tracker, https://github.com/Alquimia-ai/fair-forge/issues
Project-URL: Changelog, https://github.com/Alquimia-ai/fair-forge/releases
Author-email: Alquimia AI <contact@alquimia.ai>
Maintainer-email: Alex Fiorenza <alex.fiorenza@alquimia.ai>
License: MIT
License-File: LICENSE
Keywords: ai,assistant,bias,evaluation,fairness,llm,metrics,ml,nlp,toxicity
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: jinja2>=3.1.0
Requires-Dist: langchain-core>=1.2.11
Requires-Dist: langchain-groq>=1.1.2
Requires-Dist: langchain>=1.2
Requires-Dist: loguru>=0.7.3
Requires-Dist: optuna>=4.7.0
Requires-Dist: pydantic<3.0.0,>=2.0.0
Requires-Dist: scipy>=1.14.1
Requires-Dist: tqdm>=4.65.0
Requires-Dist: transformers>=4.35.0
Provides-Extra: agentic
Provides-Extra: all
Requires-Dist: accelerate>=0.25.0; extra == 'all'
Requires-Dist: aiosseclient>=0.1.8; extra == 'all'
Requires-Dist: alquimia-client>=0.1.0; extra == 'all'
Requires-Dist: hdbscan>=0.8.41; extra == 'all'
Requires-Dist: httpx>=0.28.1; extra == 'all'
Requires-Dist: interpreto>=0.1.0; extra == 'all'
Requires-Dist: nltk>=3.8.0; extra == 'all'
Requires-Dist: numba>=0.57.0; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: optuna>=3.0.0; extra == 'all'
Requires-Dist: pandas>=2.0.0; extra == 'all'
Requires-Dist: python-dotenv>=1.0.0; extra == 'all'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'all'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'all'
Requires-Dist: torch>=2.0.0; extra == 'all'
Requires-Dist: transformers>=4.35.0; extra == 'all'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'all'
Provides-Extra: bestof
Provides-Extra: bias
Requires-Dist: torch>=2.0.0; extra == 'bias'
Provides-Extra: context
Provides-Extra: conversational
Provides-Extra: dev
Requires-Dist: commitizen>=4.0.0; extra == 'dev'
Requires-Dist: ipykernel>=6.0.0; extra == 'dev'
Requires-Dist: jupyter>=1.0.0; extra == 'dev'
Requires-Dist: mypy>=1.8.0; extra == 'dev'
Requires-Dist: nbconvert>=7.17.0; extra == 'dev'
Requires-Dist: notebook>=7.0.0; extra == 'dev'
Requires-Dist: pandas-stubs>=2.0.0; extra == 'dev'
Requires-Dist: pre-commit>=3.5.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest-xdist>=3.5.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: python-semantic-release>=9.0.0; extra == 'dev'
Requires-Dist: ruff>=0.8.0; extra == 'dev'
Requires-Dist: sphinx-rtd-theme>=2.0.0; extra == 'dev'
Requires-Dist: sphinx>=7.0.0; extra == 'dev'
Requires-Dist: types-tqdm>=4.66.0; extra == 'dev'
Requires-Dist: virtualenv>=20.36.1; extra == 'dev'
Provides-Extra: explainability
Requires-Dist: interpreto>=0.1.0; extra == 'explainability'
Requires-Dist: torch>=2.0.0; extra == 'explainability'
Requires-Dist: transformers>=4.35.0; extra == 'explainability'
Provides-Extra: generators
Provides-Extra: generators-alquimia
Requires-Dist: alquimia-client>=0.1.0; extra == 'generators-alquimia'
Provides-Extra: humanity
Requires-Dist: numpy>=1.24.0; extra == 'humanity'
Requires-Dist: pandas>=2.0.0; extra == 'humanity'
Provides-Extra: metrics
Requires-Dist: accelerate>=0.25.0; extra == 'metrics'
Requires-Dist: hdbscan>=0.8.41; extra == 'metrics'
Requires-Dist: nltk>=3.8.0; extra == 'metrics'
Requires-Dist: numba>=0.57.0; extra == 'metrics'
Requires-Dist: numpy>=1.24.0; extra == 'metrics'
Requires-Dist: pandas>=2.0.0; extra == 'metrics'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'metrics'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'metrics'
Requires-Dist: torch>=2.0.0; extra == 'metrics'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'metrics'
Provides-Extra: prompt-optimizer
Requires-Dist: optuna>=3.0.0; extra == 'prompt-optimizer'
Provides-Extra: regulatory
Requires-Dist: accelerate>=0.25.0; extra == 'regulatory'
Requires-Dist: torch>=2.0.0; extra == 'regulatory'
Provides-Extra: runners
Requires-Dist: aiosseclient>=0.1.8; extra == 'runners'
Requires-Dist: alquimia-client>=0.1.0; extra == 'runners'
Requires-Dist: httpx>=0.28.1; extra == 'runners'
Requires-Dist: python-dotenv>=1.0.0; extra == 'runners'
Provides-Extra: toxicity
Requires-Dist: hdbscan>=0.8.41; extra == 'toxicity'
Requires-Dist: nltk>=3.8.0; extra == 'toxicity'
Requires-Dist: numba>=0.57.0; extra == 'toxicity'
Requires-Dist: numpy>=1.24.0; extra == 'toxicity'
Requires-Dist: pandas>=2.0.0; extra == 'toxicity'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'toxicity'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'toxicity'
Requires-Dist: torch>=2.0.0; extra == 'toxicity'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'toxicity'
Provides-Extra: vision
Requires-Dist: numpy>=1.24.0; extra == 'vision'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'vision'
Requires-Dist: torch>=2.0.0; extra == 'vision'
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="docs/assets/logo-dark.svg">
    <source media="(prefers-color-scheme: light)" srcset="docs/assets/logo-light.svg">
    <img alt="Fair Forge" src="docs/assets/logo-light.svg" width="480">
  </picture>
</p>
<p align="center">
  <strong>Performance-measurement library for evaluating AI models and assistants</strong>
</p>

<p align="center">
  <a href="https://github.com/Alquimia-ai/fair-forge/actions/workflows/release.yml"><img src="https://github.com/Alquimia-ai/fair-forge/actions/workflows/release.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/alquimia-fair-forge/"><img src="https://img.shields.io/pypi/v/alquimia-fair-forge" alt="PyPI"></a>
  <a href="https://pypi.org/project/alquimia-fair-forge/"><img src="https://img.shields.io/pypi/pyversions/alquimia-fair-forge" alt="Python Versions"></a>
  <a href="https://github.com/Alquimia-ai/fair-forge/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Alquimia-ai/fair-forge" alt="License"></a>
</p>

<p align="center">
  <a href="https://fairforge.alquimia.ai">Documentation</a> •
  <a href="https://github.com/Alquimia-ai/fair-forge">GitHub</a>
</p>

## What is Fair Forge?

Fair Forge provides comprehensive metrics for evaluating AI systems:

- **Toxicity** - Detect toxic language patterns with DIDT fairness framework
- **Bias** - Analyze biases across protected attributes (gender, race, religion, etc.)
- **Context** - Assess how well responses align with provided context
- **Conversational** - Evaluate dialogue quality using Grice's maxims
- **Humanity** - Measure how natural responses are through emotional analysis
- **BestOf** - Tournament-style comparison to find the best response
- **Agentic** - Evaluate autonomous agent behavior and tool usage

## Quick Start

```bash
# Install with pip
pip install alquimia-fair-forge

# Or install specific modules
pip install "alquimia-fair-forge[toxicity]"
pip install "alquimia-fair-forge[bias]"
pip install "alquimia-fair-forge[all]"
```

```python
from fair_forge import Retriever, Dataset
from fair_forge.metrics.toxicity import Toxicity

class MyRetriever(Retriever):
    def load_dataset(self) -> list[Dataset]:
        # Load your dataset here
        pass

metrics = Toxicity.run(MyRetriever, verbose=True)
```

## Documentation

For complete documentation, guides, and API reference visit:

**[https://fairforge.alquimia.ai](https://fairforge.alquimia.ai)**

## Contributing

### Setup

```bash
# Clone and install dependencies
git clone https://github.com/Alquimia-ai/fair-forge.git
cd fair-forge
uv sync

# Install pre-commit hooks (required)
uv run pre-commit install --hook-type pre-commit --hook-type commit-msg
```

### Pre-commit Hooks

This project uses [pre-commit](https://pre-commit.com/) to enforce code quality and commit conventions. The following hooks run automatically:

| Hook | Stage | Purpose |
|------|-------|---------|
| `trailing-whitespace` | pre-commit | Remove trailing whitespace |
| `end-of-file-fixer` | pre-commit | Ensure files end with a newline |
| `check-yaml` / `check-toml` | pre-commit | Validate config files |
| `check-added-large-files` | pre-commit | Block files >1MB |
| `detect-private-key` | pre-commit | Prevent accidental secret commits |
| `ruff` | pre-commit | Lint and auto-fix Python code |
| `ruff-format` | pre-commit | Format Python code |
| `commitizen` | commit-msg | Enforce [Conventional Commits](https://www.conventionalcommits.org/) format |

### Commit Convention

All commits **must** follow the [Conventional Commits](https://www.conventionalcommits.org/) specification, enforced by [commitizen](https://commitizen-tools.github.io/commitizen/):

```
type(scope): description
```

**Types:** `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`

**Scopes:** `metrics`, `runners`, `generators`, `storage`, `schemas`, `core`, `llm`, `guardians`, `deps`

Examples:
```
feat(metrics): add agentic evaluation metric
fix(toxicity): handle empty batches
docs: update installation guide
test(runners): add integration tests
```

### Claude Code Skills

This project includes [Claude Code](https://claude.ai/code) skills to automate common workflows. If you use Claude Code as your development tool, the following skills are available:

| Skill | Command | Description |
|-------|---------|-------------|
| **Commit** | `/commit` | Analyzes staged changes and generates a properly formatted Conventional Commit message |
| **Docs** | `/docs [type] [name]` | Generates Mintlify MDX documentation for new modules (e.g., `/docs metric agentic`) |
| **Lambda** | `/fair-forge-lambda` | Scaffolds AWS Lambda deployment boilerplate for Fair-Forge modules |

## License

MIT License - see [LICENSE](LICENSE) for details.
