Metadata-Version: 2.4
Name: alquimia-fair-forge
Version: 1.2.0b3
Summary: Alquimia Fair Forge library - Performance measurement for AI models and assistants
Project-URL: Homepage, https://github.com/Alquimia-ai/fair-forge
Project-URL: Repository, https://github.com/Alquimia-ai/fair-forge.git
Project-URL: Documentation, https://github.com/Alquimia-ai/fair-forge#readme
Project-URL: Bug Tracker, https://github.com/Alquimia-ai/fair-forge/issues
Project-URL: Changelog, https://github.com/Alquimia-ai/fair-forge/releases
Author-email: Alquimia AI <contact@alquimia.ai>
Maintainer-email: Alex Fiorenza <alex.fiorenza@alquimia.ai>
License: MIT
License-File: LICENSE
Keywords: ai,assistant,bias,evaluation,fairness,llm,metrics,ml,nlp,toxicity
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: jinja2>=3.1.0
Requires-Dist: langchain-core>=1.2.11
Requires-Dist: langchain>=1.2
Requires-Dist: loguru>=0.7.3
Requires-Dist: pydantic<3.0.0,>=2.0.0
Requires-Dist: scipy>=1.14.1
Requires-Dist: tqdm>=4.65.0
Requires-Dist: transformers>=4.35.0
Provides-Extra: agentic
Provides-Extra: all
Requires-Dist: aiosseclient>=0.1.8; extra == 'all'
Requires-Dist: alquimia-client>=0.1.0; extra == 'all'
Requires-Dist: boto3>=1.28.0; extra == 'all'
Requires-Dist: hdbscan>=0.8.41; extra == 'all'
Requires-Dist: httpx>=0.28.1; extra == 'all'
Requires-Dist: interpreto>=0.1.0; extra == 'all'
Requires-Dist: lakefs>=0.14.0; extra == 'all'
Requires-Dist: nltk>=3.8.0; extra == 'all'
Requires-Dist: numba>=0.57.0; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: pandas>=2.0.0; extra == 'all'
Requires-Dist: python-dotenv>=1.0.0; extra == 'all'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'all'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'all'
Requires-Dist: torch>=2.0.0; extra == 'all'
Requires-Dist: transformers>=4.35.0; extra == 'all'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'all'
Provides-Extra: bestof
Provides-Extra: bias
Requires-Dist: torch>=2.0.0; extra == 'bias'
Provides-Extra: cloud
Requires-Dist: boto3>=1.28.0; extra == 'cloud'
Requires-Dist: lakefs>=0.14.0; extra == 'cloud'
Provides-Extra: context
Provides-Extra: conversational
Provides-Extra: dev
Requires-Dist: commitizen>=4.0.0; extra == 'dev'
Requires-Dist: ipykernel>=6.0.0; extra == 'dev'
Requires-Dist: jupyter>=1.0.0; extra == 'dev'
Requires-Dist: mypy>=1.8.0; extra == 'dev'
Requires-Dist: nbconvert>=7.17.0; extra == 'dev'
Requires-Dist: notebook>=7.0.0; extra == 'dev'
Requires-Dist: pandas-stubs>=2.0.0; extra == 'dev'
Requires-Dist: pre-commit>=3.5.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest-xdist>=3.5.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: python-semantic-release>=9.0.0; extra == 'dev'
Requires-Dist: ruff>=0.8.0; extra == 'dev'
Requires-Dist: sphinx-rtd-theme>=2.0.0; extra == 'dev'
Requires-Dist: sphinx>=7.0.0; extra == 'dev'
Requires-Dist: types-tqdm>=4.66.0; extra == 'dev'
Requires-Dist: virtualenv>=20.36.1; extra == 'dev'
Provides-Extra: explainability
Requires-Dist: interpreto>=0.1.0; extra == 'explainability'
Requires-Dist: torch>=2.0.0; extra == 'explainability'
Requires-Dist: transformers>=4.35.0; extra == 'explainability'
Provides-Extra: generators
Provides-Extra: generators-alquimia
Requires-Dist: alquimia-client>=0.1.0; extra == 'generators-alquimia'
Provides-Extra: humanity
Requires-Dist: numpy>=1.24.0; extra == 'humanity'
Requires-Dist: pandas>=2.0.0; extra == 'humanity'
Provides-Extra: metrics
Requires-Dist: hdbscan>=0.8.41; extra == 'metrics'
Requires-Dist: nltk>=3.8.0; extra == 'metrics'
Requires-Dist: numba>=0.57.0; extra == 'metrics'
Requires-Dist: numpy>=1.24.0; extra == 'metrics'
Requires-Dist: pandas>=2.0.0; extra == 'metrics'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'metrics'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'metrics'
Requires-Dist: torch>=2.0.0; extra == 'metrics'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'metrics'
Provides-Extra: runners
Requires-Dist: aiosseclient>=0.1.8; extra == 'runners'
Requires-Dist: alquimia-client>=0.1.0; extra == 'runners'
Requires-Dist: httpx>=0.28.1; extra == 'runners'
Requires-Dist: python-dotenv>=1.0.0; extra == 'runners'
Provides-Extra: toxicity
Requires-Dist: hdbscan>=0.8.41; extra == 'toxicity'
Requires-Dist: nltk>=3.8.0; extra == 'toxicity'
Requires-Dist: numba>=0.57.0; extra == 'toxicity'
Requires-Dist: numpy>=1.24.0; extra == 'toxicity'
Requires-Dist: pandas>=2.0.0; extra == 'toxicity'
Requires-Dist: scikit-learn<1.8,>=1.3.0; extra == 'toxicity'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'toxicity'
Requires-Dist: torch>=2.0.0; extra == 'toxicity'
Requires-Dist: umap-learn<0.6.0,>=0.5.6; extra == 'toxicity'
Description-Content-Type: text/markdown

# Fair Forge

<p align="center">
  <img src="https://cdn.prod.website-files.com/68f2b0d5efa00a8133ee0b23/698275cf13a9d1263497724f_Alquimia%20Logo.svg" width="800" alt="Alquimia AI">
</p>

<p align="center">
  <strong>Performance-measurement library for evaluating AI models and assistants</strong>
</p>

<p align="center">
  <a href="https://github.com/Alquimia-ai/fair-forge/actions/workflows/release.yml"><img src="https://github.com/Alquimia-ai/fair-forge/actions/workflows/release.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/alquimia-fair-forge/"><img src="https://img.shields.io/pypi/v/alquimia-fair-forge" alt="PyPI"></a>
  <a href="https://pypi.org/project/alquimia-fair-forge/"><img src="https://img.shields.io/pypi/pyversions/alquimia-fair-forge" alt="Python Versions"></a>
  <a href="https://github.com/Alquimia-ai/fair-forge/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Alquimia-ai/fair-forge" alt="License"></a>
</p>

<p align="center">
  <a href="https://fairforge.alquimia.ai">Documentation</a> •
  <a href="https://github.com/Alquimia-ai/fair-forge">GitHub</a>
</p>

## What is Fair Forge?

Fair Forge provides comprehensive metrics for evaluating AI systems:

- **Toxicity** - Detect toxic language patterns with DIDT fairness framework
- **Bias** - Analyze biases across protected attributes (gender, race, religion, etc.)
- **Context** - Assess how well responses align with provided context
- **Conversational** - Evaluate dialogue quality using Grice's maxims
- **Humanity** - Measure how natural responses are through emotional analysis
- **BestOf** - Tournament-style comparison to find the best response
- **Explainability** - Compute token attributions to understand model decisions

## Quick Start

```bash
# Install with pip
pip install alquimia-fair-forge

# Or install specific modules
pip install "alquimia-fair-forge[toxicity]"
pip install "alquimia-fair-forge[bias]"
pip install "alquimia-fair-forge[explainability]"
pip install "alquimia-fair-forge[all]"
```

```python
from fair_forge import Retriever, Dataset
from fair_forge.metrics.toxicity import Toxicity

class MyRetriever(Retriever):
    def load_dataset(self) -> list[Dataset]:
        # Load your dataset here
        pass

metrics = Toxicity.run(MyRetriever, verbose=True)
```

### Explainability

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from fair_forge.explainability import AttributionExplainer, Lime

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

# Format prompt according to your model (user responsibility)
messages = [{"role": "user", "content": "What is gravity?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False)

explainer = AttributionExplainer(model, tokenizer)
result = explainer.explain(
    prompt=prompt,
    target="Gravity is the force of attraction between objects.",
    method=Lime,  # Pass the method class directly
)

# Get most important words
for attr in result.get_top_k(5):
    print(f"'{attr.text}': {attr.score:.4f}")
```

## Documentation

For complete documentation, guides, and API reference visit:

**[https://fairforge.alquimia.ai](https://fairforge.alquimia.ai)**

## License

MIT License - see [LICENSE](LICENSE) for details.
