Metadata-Version: 2.4
Name: pp
Version: 1.7.0
Summary: Parallel Python - parallelize Python workloads over local processes and/or remote servers
Author-email: Parallel Python contributors <support@parallelpython.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://www.parallelpython.com
Project-URL: Documentation, https://www.parallelpython.com
Keywords: cluster,clustering,computation,distributed,hpc,ipc,multicore,multiprocessing,parallel,smp
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Clustering
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Dynamic: license-file

# Parallel Python

**Parallel Python** is a Python module which provides a mechanism for parallel
execution of Python code on **SMP** (systems with multiple processors or
cores) and **clusters** (computers connected via network).

It is light, easy to install and integrate with other Python software.
Parallel Python is an open source and cross-platform module written in pure
Python — no third-party runtime dependencies.

Website: [www.parallelpython.com](https://www.parallelpython.com)

## Why processes instead of threads?

The most simple and common way to write parallel applications for SMP
computers is to use threads. However, if the application is
computation-bound, the `threading` module will not allow Python byte-code to
run in parallel: the interpreter uses the GIL (Global Interpreter Lock), so
only one byte-code instruction executes at a time even on an SMP machine.

Parallel Python overcomes this limitation with worker **processes** and IPC.
All the complexity of processes, pipes, sockets and scheduling is handled for
you — your application just submits jobs and retrieves their results. The
same jobs can run on local worker processes or on remote `ppserver` nodes,
with dynamic load-balancing across both.

## Quick start, SMP

```python
import pp

def square(x):
    return x * x

# 1) Start the pp execution server with one worker per processor
job_server = pp.Server()

# 2) Submit all the tasks for parallel execution
f1 = job_server.submit(square, (1,))
f2 = job_server.submit(square, (2,))

# 3) Retrieve the results as needed
r1 = f1()
r2 = f2()

job_server.destroy()
```

See `examples/` for complete programs: summing primes, reversing MD5 hashes,
dynamic worker counts, callbacks, auto-differentiation and a benchmark.

## Quick start, clusters

On the nodes, start a server on each remote computational node:

```bash
node-1> ppserver
node-2> ppserver
node-3> ppserver
```

On the client:

```python
import pp

ppservers = ("node-1", "node-2", "node-3")
job_server = pp.Server(ppservers=ppservers)

f1 = job_server.submit(func1, args1, depfuncs1, modules1)
f2 = job_server.submit(func2, args2, depfuncs2, modules2)

r1 = f1()
r2 = f2()
```

### Auto-discovery

Instead of listing nodes explicitly, run servers with auto-discovery
enabled and let the client find them over UDP broadcast:

```bash
node-1> ppserver -a
node-2> ppserver -a
```

```python
import pp

job_server = pp.Server(ppservers=("*",))
```

By default the discovery destination is derived from the wildcard pattern
(`"*"` uses `255.255.255.255`). On networks without broadcast delivery you
can point discovery at a specific address instead — on the client with the
`broadcast` argument, on `ppserver` with `-b BROADCAST`:

```python
job_server = pp.Server(ppservers=("*",), broadcast="192.168.1.255")
```

## Features

- Parallel execution of Python code on SMP machines and clusters
- Job-based parallelization that is easy to understand and convert from
  serial code
- Automatic detection of the optimal configuration (the number of worker
  processes defaults to the number of effective processors)
- Dynamic processor allocation — the worker count can be changed at runtime
  with `set_ncpus()`
- Low overhead for repeated jobs: identical function packages are shipped
  once and then referenced by content hash
- Dynamic load balancing — jobs are distributed between workers at runtime
- Fault tolerance — if a worker or node fails, tasks are rescheduled on
  others
- Auto-discovery of computational resources over UDP broadcast
- Dynamic allocation of computational resources (a consequence of
  auto-discovery and fault tolerance)
- SHA-based authentication for network connections
- Cross-platform portability (Windows, Linux, Unix, Mac OS X)
- Standard library only — no third-party runtime dependencies

## Requirements

- Python 3.10 or newer
- No third-party runtime dependencies (standard library only)

## Installation

```bash
pip install pp
```

From a source checkout:

```bash
pip install -e .
```

The install provides the `pp` module and the `ppserver` command-line tool.

## Running a network server

```bash
ppserver -i 0.0.0.0 -p 60001 -s mysecret -w 4
```

Then from any client on the network:

```python
job_server = pp.Server(ppservers=("192.168.1.10:60001",), secret="mysecret")
```

`ppserver` options:

| Option | Description |
| ------ | ----------- |
| `-d` | Set log level to debug |
| `-f FORMAT` | Log format |
| `-a` | Enable auto-discovery service |
| `-r` | Restart worker process after each task |
| `-n PROTO` | Pickle protocol number (default 4) |
| `-c PATH` | Read options from an INI config file (`[general]` / `[network]` sections) |
| `-i INTERFACE` | Network interface to listen on |
| `-b BROADCAST` | Broadcast address for auto-discovery |
| `-p PORT` | Port to listen on (default 60000) |
| `-w NWORKERS` | Number of workers to start |
| `-s SECRET` | Secret for authentication |
| `-t SECONDS` | Exit if no client connections exist for this long |
| `-k SECONDS` | Socket timeout (also the maximum remote job time) |
| `-P PID_FILE` | Write the server PID to this file |
| `-q` | Quiet mode: suppress startup banner and only print errors |

> **Security note:** always use a non-trivial secret key. A default secret is
> used when none is configured, which is not suitable for untrusted networks.

## API overview

```python
job_server = pp.Server(ncpus="autodetect", ppservers=(), secret=None,
                       restart=False, proto=4, socket_timeout=3600,
                       loglevel=None)
job = job_server.submit(func, args=(), depfuncs=(), modules=(),
                        callback=None, callbackargs=(), group="default",
                        globals=None)
result = job()                  # blocks until the job finishes
job.wait()                      # block until the job finishes
job_server.wait("group")        # wait for a group of jobs
job_server.set_ncpus(4)         # resize the local pool at runtime
job_server.get_active_nodes()   # {node: ncpus}
job_server.get_stats()          # job execution statistics
job_server.print_stats()        # print the statistics
job_server.destroy()            # kill workers and close files

template = pp.Template(job_server, func, depfuncs=(), modules=(),
                       callback=None, callbackargs=(), group="default")
job = template.submit(1, 2, 3)  # reuse the same job with new args
```

Notes:

- `submit` serializes the function by its source, so functions and classes
  defined in your script work out of the box. Functions that live in
  importable modules can instead be shipped by name via the `modules=`
  argument. Built-in callables (`math.sqrt`, `str.upper`, ...) are shipped
  as an import-and-bind.
- Lambdas must be assigned to a variable before submitting (`f = lambda x:
  x * x`), since workers resolve functions by name.
- A job that raises an exception returns `None` and prints the traceback;
  the worker keeps serving subsequent jobs.

## Examples

```
examples/sum_primes.py        # sum of primes across workers
examples/reverse_md5.py       # brute-force hash search
examples/dynamic_ncpus.py     # change the worker count at runtime
examples/callback.py          # result callbacks
examples/auto_diff.py         # automatic differentiation
examples/benchmark.py         # serial vs. parallel throughput
```

## Testing

```bash
pip install pytest pytest-timeout
pytest
```

All tests are self-contained (local subprocesses and loopback sockets only)
and each is held to a 60-second timeout by pytest-timeout.

## Project layout

```
src/pp/
    __init__.py    # public API
    _version.py    # version metadata
    _common.py     # shared utilities
    _transport.py  # length-framed pipe/socket transports + caching
    _server.py     # Server, Template, DestroyedServerError, scheduler
    _worker.py     # worker subprocess (python -m pp._worker)
    _auto.py       # UDP auto-discovery
    cli.py         # ppserver entry point (NetworkServer + CLI)
examples/          # runnable examples
tests/             # pytest test suite
```

## License

Apache-2.0 (see `LICENSE` and `NOTICE`). Questions and support: [support@parallelpython.com](mailto:support@parallelpython.com).
