Process backend

Use the process backend for CPU-heavy Python work on Python 3.11+ when you want work to run across CPU cores.

Good uses

• parsing large payloads;

• scoring algorithms;

• CPU-heavy transformations;

• compression or hashing;

• pure functions over serializable data.

Pickling rules

ProcessPoolExecutor sends functions, arguments, and return values between processes. They must be picklable.

Good:

def calculate_score(payload: dict[str, float]) -> float:
    return float(payload["value"]) * 1.5

Avoid:

lambda value: value * 2

Avoid sending:

• Redis clients;

• database connections;

• sockets;

• locks;

• async functions;

• objects with complex process-local state.

Basic example

examples/07_process_backend_cpu_work.py

"""
CPU-heavy work with ProcessPoolExecutor backend on Python 3.11.

Use the process backend for CPU-bound work that should use multiple CPU cores.

Important:
- submitted functions must be top-level importable functions
- arguments and return values must be picklable
- do not submit lambdas, nested functions, open sockets, Redis clients, or DB clients
"""

from __future__ import annotations

import asyncio

from leasepool import ExecutorBackend, LeasedExecutorManager


def count_primes(limit: int) -> int:
    count = 0

    for number in range(2, limit):
        for divisor in range(2, int(number**0.5) + 1):
            if number % divisor == 0:
                break
        else:
            count += 1

    return count


async def main() -> None:
    manager = LeasedExecutorManager(
        backend=ExecutorBackend.PROCESS,
        max_pools=1,
        min_pools=1,
        workers_per_pool=4,
    )

    await manager.start()

    try:
        async with await manager.acquire(owner="cpu-primes") as lease:
            results = await asyncio.gather(
                lease.run(count_primes, 20_000),
                lease.run(count_primes, 21_000),
                lease.run(count_primes, 22_000),
                lease.run(count_primes, 23_000),
            )

        print("Prime counts:", results)

    finally:
        await manager.stop()


if __name__ == "__main__":
    asyncio.run(main())

Keyword arguments

ExecutorLease.run() supports keyword arguments. With the process backend, keyword values must also be picklable.

async with await manager.acquire(owner="score") as lease:
    result = await lease.run(calculate_score, payload={"value": 21.0})

Executor kwargs

Extra keyword arguments passed to LeasedExecutorManager are forwarded to the underlying executor constructor.

For process pools this lets you pass options such as initializer, initargs, mp_context, and max_tasks_per_child:

manager = LeasedExecutorManager(
    backend="process",
    max_pools=1,
    workers_per_pool=4,
    initializer=configure_worker,
    initargs=("worker-config",),
)

When process log forwarding is enabled, leasepool composes its own initializer with your initializer so both run.

Operational notes

Run process-backend examples as files:

python examples/07_process_backend_cpu_work.py

Do not paste process-pool examples into an interactive REPL. Worker processes must be able to import top-level functions from a module.