Metadata-Version: 2.1
Name: parallel-iterator
Version: 1.0.0
Summary: A parallel iterator execution library for Python
Home-page: https://github.com/lh9171338/parallel-iterator
Author: lh9171338
Author-email: lh9171338 <lihao2015@whu.edu.cn>
License: MIT
Project-URL: Homepage, https://github.com/lh9171338/parallel-iterator
Project-URL: Bug Tracker, https://github.com/lh9171338/parallel-iterator/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tqdm
Requires-Dist: numpy
Requires-Dist: aiomultiprocess

# parallel-iterator

[English](README.md) | [中文](README_zh.md)

A parallel iterator execution library for Python, providing various executors for concurrent and parallel task processing.

## Installation

```bash
pip install parallel-iterator
```

Or install from source:

```bash
git clone https://github.com/lh9171338/parallel-iterator.git
cd parallel-iterator
pip install -e .
```

## Quick Start

```python
import time
from parallel_iterator import ProcessExecutor

# Define a task function
def process(a, b, delay=0.1):
    time.sleep(delay)
    return a + b

# Prepare data
a_list = [1, 2, 3, 4, 5]
b_list = [10, 20, 30, 40, 50]

# Execute in parallel using multiple processes
executor = ProcessExecutor(process, nprocs=4)
results = executor.run(a_list, b_list, delay=0.1)

print(results)  # [11, 22, 33, 44, 55]
```

## Executors

### Sequential Execution

| Executor | Description |
|----------|-------------|
| `SequentialExecutor` | Execute tasks sequentially in a single process |

### Process-based Executors

| Executor | Description |
|----------|-------------|
| `ProcessExecutor` | Execute tasks in parallel using multiple processes |
| `AutoProcessExecutor` | Process executor that automatically falls back to sequential execution when `nprocs=1` |
| `BoundedProcessExecutor` | Process executor with bounded resource slots (e.g., GPU ports) |
| `AutoBoundedProcessExecutor` | Auto-switching bounded process executor |

### Thread-based Executors

| Executor | Description |
|----------|-------------|
| `ThreadExecutor` | Execute tasks in parallel using a thread pool |
| `AutoThreadExecutor` | Automatically fallback to sequential execution when `nworkers=1` |

### Async Executors

| Executor | Description |
|----------|-------------|
| `AsyncExecutor` | Execute async tasks concurrently in a single process |
| `AsyncProcessExecutor` | Execute async tasks across multiple processes |
| `AutoAsyncProcessExecutor` | Auto-switching async process executor |

### Parallel Executors (Data Splitting)

| Executor | Description |
|----------|-------------|
| `TaskParallelExecutor` | Split data and process each item with a single-task function |
| `BatchParallelExecutor` | Split data and process batches with a batch function |
| `AutoTaskParallelExecutor` | Auto-switching task parallel executor |
| `AutoBatchParallelExecutor` | Auto-switching batch parallel executor |

## Usage Examples

### Using ProcessExecutor

```python
from parallel_iterator import ProcessExecutor

def compute(x, y, op="+"):
    if op == "+":
        return x + y
    return x - y

executor = ProcessExecutor(compute, nprocs=4)
results = executor.run([1, 2, 3], [4, 5, 6], op="+")
# results: [5, 7, 9]
```

### Using ThreadExecutor

```python
from parallel_iterator import ThreadExecutor

def io_task(url, timeout=10):
    # Simulate I/O operation
    return f"Downloaded: {url}"

executor = ThreadExecutor(io_task, nworkers=8)
results = executor.run(["url1", "url2", "url3"], timeout=5)
```

### Using AsyncExecutor

```python
import asyncio
from parallel_iterator import AsyncExecutor

async def async_task(item, delay=0.1):
    await asyncio.sleep(delay)
    return item * 2

executor = AsyncExecutor(async_task)
results = executor.run([1, 2, 3, 4, 5], delay=0.1)
# results: [2, 4, 6, 8, 10]
```

### Using BatchParallelExecutor

```python
from parallel_iterator import BatchParallelExecutor

def process_batch(items_a, items_b, factor=1):
    return [a + b * factor for a, b in zip(items_a, items_b)]

executor = BatchParallelExecutor(process_batch, nprocs=4)
results = executor.run([1, 2, 3, 4], [10, 20, 30, 40], factor=2)
# results: [21, 42, 63, 84]
```

## Executor Selection Guide

| Scenario | Recommended Executor |
|----------|---------------------|
| CPU-bound tasks | `ProcessExecutor` |
| I/O-bound tasks | `ThreadExecutor` or `AsyncExecutor` |
| Need GPU/port binding | `BoundedProcessExecutor` |
| Async functions + multiprocessing | `AsyncProcessExecutor` |
| Batch data processing | `BatchParallelExecutor` |
| Single-task function + data splitting | `TaskParallelExecutor` |
| Uncertain about process count | Use `Auto*` series executors |

## API Reference

- [API Reference (English)](docs/API.md)
- [API 参考文档 (中文)](docs/API_zh.md)

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
