Metadata-Version: 2.1
Name: reasoning-benchmarks
Version: 1.0.1
Summary: Generate argumentation reasoning benchmarks and prompts.
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: pandas
Requires-Dist: matplotlib
Requires-Dist: networkx

# reasoning-benchmarks

The `reasoning-benchmarks` library can be used to dynamically generate reasoning tasks based on formal argumentation to evaluate reasoning capabilities.


## Reasoning puzzles

The following types of reasoning puzzles can be generated:

```
The following is a reasoning puzzle. Witnesses should always be believed
unless there is believable testimony that they are lying. Now consider the
following facts:

Witness Livia says that the employee arrived on time
Witness Aniyah says that Livia is lying
Witness Tatum says that Aniyah is lying

Final question: can we believe that the employee arrived on time??
End your answer with: "Answer: yes or no".
```

In addition to the binary reasoning task, we also provide multi-label tasks, where the solution requires the identification of all believable arguments:
```
The following is a reasoning puzzle. Witnesses should always be believed
unless there is believable testimony that they are not credible. Now consider the
following facts:

Witness Grace says that the password was changed
Witness Amelia says that Grace is not credible
Witness Micah says that Amelia is not credible

Final question: which witnesses can be believed?
End your answer with: "Answer: list of names that can be believed".
E.g. Answer: ['Alice', 'Bob']
```

## Installation

```bash
pip install reasoning-benchmarks
```

## Quickstart

```python
from reasoning_benchmarks import generate_benchmark

# Quickly generate a benchmark
benchmark = generate_benchmark()

# Generate a benchmarks with optional parameters:
benchmark = generate_benchmark(
	chain_range=(1, 15), # Generates all unique chain graphs with 1 to 15 arguments 
	star_range=(3, 10), # Generates all unique 'star' graphs with 3 to 10 arguments
	shuffled=True, # Whether the arguments in the prompt should be shuffled or presented in order
	percentage_irrelevant=0.1, # The ratio of prompts that contain irrelevant argument chains
    irrelevant_len_range=(1,5), # A tuple (min_length, max_length) to determine the length of irrelevant chains, where (1, 5) would include irrelevant chains of length 1 to 5.
    irrelevant_num_range=(1,3), # A tuple (min_num, max_num) to determine the number of irrelevant chains, where (1, 3) would include 1 to 3 irrelevant chains.
	attack_words=["not credible", "unreliable", "lying"], # Which attack words are used in the prompt (randomly sampled from the provided list)
	num_variations_per_graph=(1, 1), # The number of variations per chain graph and star graph
	seed=95, # A fixed seed for reproducibility
    output_file='benchmark.csv', #save the benchmark DataFrame to this CSV file.
)

print(benchmark.head())
```

This allows for the dynamic generation of reasoning puzzles of scaling complexity, ranging from simple to complex. 

## Argumentation backend

The puzzles are based on argument attack graphs, that can be created using the library as follows:
```python
from reasoning_benchmarks import ArgumentGraph, Argument, plot_argument_graph

# Initialize the graph
graph = ArgumentGraph()

# Create two arguments
a = Argument(0, 'Alice says that the train is late.')
b = Argument(1, 'Bob says that Alice is lying.')

# Add the arguments to the graph
graph.add_argument(a)
graph.add_argument(b)

# Add the attack between argument b and a
graph.add_attack(b, a)

# Determine the values of the arguments 
graph.solve()

# Display the resulting argument graph visually
plot_argument_graph(graph)
```

Based on this graph, you can generate a prompt using the create_prompt function:

```python
from reasoning_benchmarks import create_prompt

prompt = create_prompt(graph)
```

Functions are included to quickly generate one, or multiple graphs:
```python
from reasoning_benchmarks import generate_chain, generate_star_graph, generate_all_chains, generate_all_star_graphs

# Generate a chain graph with 3 arguments
chain = generate_chain(3)

# Generate a star graph with one center arguments, with chains of 3, 1, and 4 arguments attacking it
star = generate_star_graph([3, 1, 4])

# Generate all unique chain graphs with between 1 and 25 total number of arguments
chains = generate_all_chains(min_n=1, max_n=25)

# Generate all unique star graphs between 3 and 6 total number of arguments
stars = generate_all_star_graphs(min_n=3, max_n=6)
```


## Development Build

```bash
python -m build
```

The package includes CSV resources under `reasoning_benchmarks/src` and accesses them via `importlib.resources`.
These contain lists of names and statements used to generate the prompts and can be customized. 
