Process log forwarding¶

Logs emitted inside ProcessPoolExecutor workers are created in child processes. They do not automatically flow through the parent application’s logger handlers.

leasepool provides optional process log forwarding for the process backend. It installs a queue handler in each worker and a queue listener in the parent process.

Explicit configuration¶

Use ProcessLoggingConfig when you want one object that documents the logging bridge configuration:

examples/13_process_log_forwarding.py¶

"""
Forward logs emitted by ProcessPoolExecutor workers.

Normal leasepool logs are emitted by the parent process. Logs created inside a
process worker need an explicit queue bridge. Use ProcessLoggingConfig when you
want the parent application logger to receive worker log records.

Run this file directly:

    python examples/13_process_log_forwarding.py
"""

from __future__ import annotations

import asyncio
import logging

from leasepool import LeasedExecutorManager, ProcessLoggingConfig


LOGGER_NAME = "leasepool.examples.process.worker"


def logged_cpu_work(value: int) -> int:
    logger = logging.getLogger(LOGGER_NAME)
    logger.info("worker received value=%s", value)

    total = 0
    for item in range(value):
        total += item * item

    logger.info("worker finished value=%s", value)
    return total


async def main() -> None:
    logging.basicConfig(
        level=logging.INFO,
        format="%(processName)s %(name)s %(levelname)s: %(message)s",
    )

    parent_logger = logging.getLogger("leasepool.examples.process.parent")

    manager = LeasedExecutorManager(
        backend="process",
        max_pools=1,
        min_pools=1,
        workers_per_pool=2,
        process_logging=ProcessLoggingConfig(
            enabled=True,
            level="INFO",
            target_logger=parent_logger,
        ),
    )

    await manager.start()

    try:
        async with await manager.acquire(owner="process-log-demo") as lease:
            first, second = await asyncio.gather(
                lease.run(logged_cpu_work, 10_000),
                lease.run(logged_cpu_work, 12_000),
            )

        print("Results:", first, second)
    finally:
        await manager.stop()


if __name__ == "__main__":
    asyncio.run(main())

Convenience configuration¶

For shorter setup, pass the convenience arguments directly to the manager:

import logging

from leasepool import LeasedExecutorManager


manager = LeasedExecutorManager(
    backend="process",
    max_pools=1,
    min_pools=1,
    workers_per_pool=2,
    forward_process_logs=True,
    process_log_level="INFO",
    process_log_target_logger=logging.getLogger("myapp.process-workers"),
)

Do not pass both process_logging and the convenience arguments in the same manager.

Configuration fields¶

enabled: Enable child-process log forwarding.
level: Minimum level configured on the child-process root logger. Accepts an integer logging level or a standard level name such as "INFO".
target_logger: Parent-process logger that receives worker log records. If omitted, the manager’s logger is used.
clear_child_handlers: Remove existing child-process root handlers before installing the queue handler. Defaults to True to avoid duplicate records after fork.

Composing worker initializers¶

If you pass initializer and initargs to the process executor, leasepool preserves them when log forwarding is enabled:

manager = LeasedExecutorManager(
    backend="process",
    max_pools=1,
    forward_process_logs=True,
    initializer=configure_worker,
    initargs=("worker-config",),
)

The leasepool logging initializer runs first, then your initializer runs.

When to enable it¶

Enable process log forwarding when worker logs are important for debugging, monitoring, or audit trails.

Leave it disabled when workers do not log or when you have your own process-aware logging setup.