Metadata-Version: 2.3
Name: scissiparity
Version: 2.0.0
Summary: Zero-input-serialization, GIL-free parallel map.
Keywords: parallelism,concurrency,futures,asyncio,decorator,copy-on-write
Author: Wannes Vantorre
Author-email: Wannes Vantorre <vantorrewannes@gmail.com>
License: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Dist: dill>=0.4.1
Requires-Dist: tblib>=3.0.0
Requires-Python: >=3.14
Project-URL: Homepage, https://codeberg.org/ProductionCode/scissiparity
Project-URL: Repository, https://codeberg.org/ProductionCode/scissiparity
Project-URL: Issues, https://codeberg.org/ProductionCode/scissiparity/issues
Description-Content-Type: text/markdown

# scissiparity 🦠

**Zero-input-serialization, GIL-free parallel map.**

`multiprocessing` forces you to serialize data _in both directions_, so large or
unpicklable inputs crash or blow up memory. `scissiparity` clones the interpreter
with `os.fork()`, reads inputs for free through Copy-On-Write, and serializes only
the results you yield.

```bash
uv add scissiparity
```

## Usage

Write ordinary sequential logic over a chunk. The `@fission` decorator splits the
iterable across your CPU cores, runs a forked process per chunk, and merges the
yielded results into one lazy stream.

```python
from collections.abc import Iterator

from scissiparity import fission

MASSIVE_UNPICKLABLE_STATE = load_everything_into_memory()


@fission
def process_users(user_ids: list[str]) -> Iterator[float]:
    for user_id in user_ids:
        state = MASSIVE_UNPICKLABLE_STATE[user_id]
        yield heavy_cpu_computation(state)


results = list(process_users(["user_1", "user_2", "user_3"]))
```

## Properties

- **See-through types.** `@fission` preserves the wrapped signature. Element and
  keyword-argument types survive the decoration, so type checkers infer
  `process_users(...) -> Iterator[float]` with no annotations lost.
- **Zero input serialization.** Children read the parent's memory natively via
  Copy-On-Write; only yielded outputs are piped back (via `dill`). Unpicklable
  inputs are fine.
- **Exception teleportation.** An exception in a child is serialized, piped back,
  and re-raised in the parent with its **original traceback and chained causes
  intact** (via `tblib`), so failures read exactly as they would in-process. One
  failure aborts the run.
- **Composes without fork-bombing.** A `@fission` function called from inside
  another runs sequentially, so simple cells combine into larger pipelines.
- **Zero configuration.** Parallelism is inferred from `os.sched_getaffinity(0)`
  (falling back to `os.cpu_count()`). There are no `workers=` knobs.

Results are streamed back in completion order, not input order.

## PyTorch & transformer models

`scissiparity` is an excellent fit for **CPU** inference: `os.fork()` lets every
worker share the loaded model weights through Copy-On-Write, so a multi-gigabyte
model is loaded into memory once — not re-serialized or re-loaded per worker.

- Load the model _before_ applying `@fission`, and don't run tensor ops in the
  parent first. `fork()` duplicates only the calling thread, so a live BLAS/OpenMP
  thread pool can deadlock the children.
- Set `torch.set_num_threads(1)` and `TOKENIZERS_PARALLELISM=false` so
  process-level and library-level parallelism don't oversubscribe your cores.

**CUDA / GPU is not supported.** A CUDA context cannot survive `os.fork()` (it
raises `Cannot re-initialize CUDA in forked subprocess`), and GPU memory is not
Copy-On-Write shareable, so the core advantage doesn't apply. Use spawn-based
tooling such as `torch.multiprocessing` for GPU workloads.
