Metadata-Version: 2.4
Name: pandora_llm
Version: 0.0.0
Summary: Red-teaming large language models for train data leakage
Author: Jeffrey Wang, Jason Wang, Marvin Li, Seth Neel
License: MIT License
        
        Copyright (c) 2025 Jeffrey Wang, Jason Wang, Marvin Li, Seth Neel
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://safr-ai.quarto.pub/pandora/
Project-URL: Documentation, https://pandora-llm.readthedocs.io/en/latest/
Project-URL: Repository, https://github.com/safr-ai-lab/pandora-llm
Keywords: red-teaming,privacy,large language model,membership inference attack,extraction
Classifier: Programming Language :: Python
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: jsonargparse>=4.35.0
Requires-Dist: wandb>=0.18.1
Requires-Dist: torch>=2.4.0
Requires-Dist: torchvision>=0.18.0
Requires-Dist: torchaudio>=2.3.0
Requires-Dist: transformers>=4.41.0
Requires-Dist: datasets>=3.2.0
Requires-Dist: huggingface_hub>=0.27.0
Requires-Dist: zstandard>=0.22.0
Requires-Dist: deepspeed>=0.14.2
Requires-Dist: accelerate>=1.0.1
Requires-Dist: scikit-learn>=1.5.0
Requires-Dist: matplotlib>=3.9.0
Requires-Dist: plotly>=5.22.0
Requires-Dist: kaleido>=0.1.0
Requires-Dist: sentencepiece>=0.2.0
Requires-Dist: setuptools>=70.0.0
Requires-Dist: einops>=0.7.0
Requires-Dist: jaxtyping>=0.2.36
Requires-Dist: traker>=0.3.2
Requires-Dist: gensim>=4.3.3
Dynamic: license-file

<a href="https://pandora-llm.readthedocs.io/en/latest/"><img alt="Documentation" src="https://img.shields.io/website?url=https%3A%2F%2Fpandora-llm.readthedocs.io%2Fen%2Flatest%2F&up_message=sphinx&label=docs&color=blue"></a>
<a href="https://pypi.org/project/pandora-llm/"><img alt="Python version" src="https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2Fsafr-ai-lab%2Fpandora_llm%2Fblob%2Fmain%2Fpyproject.toml&color=green"></a>
<a href="https://github.com/safr-ai-lab/pandora_llm/blob/main/LICENSE.txt"><img alt="Code license" src="https://img.shields.io/github/license/safr-ai-lab/pandora_llm?color=blue"></a>
<a href="https://github.com/safr-ai-lab/pandora_llm/releases"><img alt="GitHub release" src="https://img.shields.io/github/v/release/safr-ai-lab/pandora_llm?color=green"></a>

<p align="center">
   <img src="docs/source/assets/pandora_llm_title.png" alt="drawing" width="600"/>
</p>

## Overview

`pandora_llm` is a red-teaming library against Large Language Models (LLMs) that assesses their vulnerability to train data leakage.

It provides a unified [PyTorch](https://pytorch.org/) API for evaluating **membership inference attacks (MIAs)**.

Please refer to the [documentation](https://pandora-llm.readthedocs.io/en/latest/) for the API reference as well as tutorials on how to use this codebase.

`pandora_llm` abides by the following core principles:

- **Open Access** — Ensuring that these tools are open-source for all.
- **Reproducible** — Committing to providing all necessary code details to ensure replicability.
- **Self-Contained** — Designing attacks that are self-contained, making it transparent to understand the workings of the method without having to peer through the entire codebase or unnecessary levels of abstraction, and making it easy to contribute new code.
- **Model-Agnostic** — Supporting any [HuggingFace](https://huggingface.co/) model and dataset, making it easy to apply to any situation.
- **Usability** — Prioritizing easy-to-use starter scripts and comprehensive documentation so anyone can effectively use `pandora_llm` regardless of prior background.

We hope that our package serves to guide LLM providers to safety-check their models before release, and to empower the public to hold them accountable to their use of data.

## Installation

From pip:
```
pip install pandora-llm
```

From source:

```bash
git clone https://github.com/safr-ai-lab/pandora-llm.git
pip install -e .
```

## Quickstart
We maintain a collection of starter scripts in our codebase under ``experiments/``. If you are creating a new attack, we recommend making a copy of a starter script for a solid template.

```
python experiments/mia/run_loss.py --model_name EleutherAI/pythia-70m-deduped --model_revision step98000 --num_samples 2000 --pack --seed 229
```

```
bash scripts/run_mia_baselines_olmo.sh
bash scripts/run_mia_baselines_pile.sh
```

## Contributing
We welcome contributions! Please submit pull requests in our [GitHub](https://github.com/safr-ai-lab/pandora-llm).


## Authors

This library was created by Jeffrey G. Wang, Jason Wang, Marvin Li, and Seth Neel.
