Metadata-Version: 2.4
Name: pygolem
Version: 0.1.1
Summary: Spawn tiny little servants, running things in parallel.
Author-email: Phant0m1zed <myworkdesk2007@gmail.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/Phant0m1zed/pygolem
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/Phant0m1zed/pygolem/main/assets/pygolem_logo.png" width="200" alt="pygolem mascot">
</p>

# pygolem

**Spawn tiny little servants, running things in parallel.**

<p align="center">
  <a href="https://pypi.org/project/pygolem/"><img src="https://img.shields.io/pypi/v/pygolem.svg" alt="PyPI version"></a>
  <a href="https://pypi.org/project/pygolem/"><img src="https://img.shields.io/pypi/pyversions/pygolem.svg" alt="Python versions"></a>
  <a href="https://github.com/Phant0m1zed/pygolem/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License"></a>
  <a href="https://github.com/Phant0m1zed/pygolem"><img src="https://img.shields.io/badge/tests-125%20passing-brightgreen.svg" alt="Tests"></a>
  <a href="https://github.com/Phant0m1zed/pygolem/blob/main/CONTRIBUTING.md"><img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg" alt="PRs Welcome"></a>
</p>

## Before

```python
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_all(fn, items, workers=10, max_retries=3, rate_limit=5):
    values = [None] * len(items)
    errors = {}
    remaining = list(enumerate(items))
    timestamps = []
    lock = threading.Lock()

    def rate_limited(item):
        while True:
            with lock:
                now = time.monotonic()
                timestamps[:] = [t for t in timestamps if now - t < 1]
                if len(timestamps) < rate_limit:
                    timestamps.append(now)
                    break
            time.sleep(0.05)
        return fn(item)

    for attempt in range(max_retries + 1):
        with ThreadPoolExecutor(max_workers=workers) as executor:
            futures = {executor.submit(rate_limited, item): i for i, item in remaining}
            remaining = []
            for future in as_completed(futures):
                index = futures[future]
                try:
                    values[index] = future.result()
                except Exception as e:
                    remaining.append((index, items[index]))
                    errors[index] = e
        if not remaining or attempt == max_retries:
            break
        time.sleep(2 ** attempt)

    return values, errors

results, errors = fetch_all(fetch_url, urls)
```

Roughly 40 lines, and this version doesn't even validate its own inputs, guard
against the `bool`-is-`int` trap, or cap thread count safely at scale.

## After

```python
from pygolem import threads

results = threads.map_parallel(fetch_url, urls, workers=10, retries=3, rate_limit=("token", 5))

results.values   # successful outputs, in original input order
results.errors   # {index: {"type": ..., "message": ...}} for items that failed permanently
```

Four lines. Same results. Ordered output, isolated per-item failures, retries
with backoff, and a global rate limit.

---

## What it is

`pygolem` wraps `concurrent.futures.ThreadPoolExecutor` with the things a
real parallel workload eventually needs:

1. **Ordered results:** `.values` always matches your input order, even
   though threads finish in whatever order they finish in.
2. **Per-item error isolation:** one failing item never crashes the batch.
3. **Retries with backoff**, plus `retry_if` to skip retrying failures you
   already know are permanent.
4. **Global rate limiting:** cap total throughput across the *entire* pool,
   not per-worker, with a choice of token bucket, leaky bucket, or sliding
   window.

Dependency-free, just the standard library.

---

## Installation

```bash
pip install pygolem
```

---

## Quick example

```python
from pygolem import threads
import requests

def fetch(url):
    return requests.get(url, timeout=5).status_code

urls = [
    "https://example.com",
    "https://httpbin.org/status/500",   # will fail, then retry
    "https://github.com",
]

result = threads.map_parallel(
    fetch, urls,
    workers=3,
    retries=2,
    backoff=("exponential", 0.5),
    rate_limit=("token", 5),
)

print(result.values)   # [200, 500, 200]
print(result.errors)   # {1: {"type": "ConnectionError", "message": "..."}}
```

---

## API

```python
threads.map_parallel(fn, items, workers=None, retries=None, retry_if=None, backoff=("linear", 1), rate_limit=None)
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `fn` | callable | — | Function to call on each item. |
| `items` | list / tuple | — | Inputs to process. Must be non-empty. |
| `workers` | int | `min(32, len(items))` | Max concurrent threads. |
| `retries` | int | `0` | Retry attempts per item after a failure. |
| `retry_if` | callable | `None` | `callable(exception) -> bool` — return `False` to stop retrying that item. |
| `backoff` | `(mode, delay)` | `("linear", 1)` | `mode` is `"linear"` or `"exponential"`. |
| `rate_limit` | `(algorithm, value)` | `None` | `algorithm` is `"token"`, `"leaky"`, or `"sliding"`; `value` is calls/sec across all workers. |

**Returns** a `Data` object: `.values` (results in input order, `None` for
permanent failures) and `.errors` (`{index: {"type": str, "message": str}}`).

---

## Known limitations

- **A stuck `fn` hangs the whole call.** There's no timeout yet, and Python
  can't force-kill a thread, so one item that never returns means
  `map_parallel` never returns either.
- **Retries can repeat side effects.** If `fn` does something real (like a
  request) and then fails afterward, retrying does that thing again. Use
  `retry_if` to skip retrying failures you know already went through.
- **One slow retry delays everyone.** The backoff wait happens between
  rounds, so already-successful items still sit and wait for it.
- **More workers won't speed up CPU-heavy code.** Threads share one GIL,
  extra workers mainly help waiting on network/disk, not crunching numbers.

---

## Benchmarks

I/O-bound workload (`time.sleep(0.01)` per call), measured with
`time.perf_counter()`. Full breakdown in [tests/02_speed](tests/02_speed).

| n | sequential | pygolem | speedup |
|---:|---:|---:|---:|
| 100 | 1.014s | 0.064s | **15.8x** |
| 1000 | 10.260s | 0.525s | **19.5x** |

Overhead over a raw `ThreadPoolExecutor` stays under 1% at scale. The
retry/error/ordering machinery is close to free.

The `workers` default is capped at 32 for a reason: an earlier version that
defaulted to `workers=len(items)` crashed at 100,000 items
(`RuntimeError: can't start new thread`). The capped default handles the
same load in 33s without crashing.

---

## Tested

125 pytest tests, 0 failures — input validation, ordering, error isolation,
retries, all three rate-limit algorithms, and a dedicated suite that proves
out the known limitations. Full breakdown in
[tests/TEST_REPORTS.md](tests/TEST_REPORTS.md).

```bash
pytest tests/ -v --tb=short
```

---

## Roadmap

- [x] Token-bucket and leaky-bucket rate limiting (alongside sliding-window)
- [x] Conditional retries via `retry_if`
- [ ] `pygolem.aio:`  asyncio engine, with item-level backoff
- [ ] `pygolem.processes:`  multiprocessing engine, with real cancellation of hung workers
- [ ] Per-item timeout support
- [ ] Optional progress callback

---

## Contributing

Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for
setup instructions, PR guidelines, and how to report bugs.

---

## About the name

Inspired by Minecraft's copper golem. A small automaton that potters around
doing small repetitive tasks on its own, tirelessly, until it needs a little
maintenance. Not a bad description of a background worker.

---

## License

Apache License 2.0. see [LICENSE](LICENSE).
