Metadata-Version: 2.4
Name: concurrent-iterator
Version: 0.4.0
Summary: Classes to run producers (iterators) and consumers (coroutines) in a background thread/process.
Home-page: https://github.com/jruere/concurrent-iterator
Author: Javier Ruere
Author-email: javier@ruere.com.ar
License: LGPLv3
Keywords: concurrency,parallelism,iterator,iterable,pipeline
Platform: POSIX
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Operating System :: POSIX
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: author
Dynamic: author-email
Dynamic: home-page
Dynamic: license-file
Dynamic: platform

# concurrent-iterator

[![Supported Python versions](https://img.shields.io/pypi/pyversions/concurrent-iterator.svg)](https://pypi.python.org/pypi/concurrent-iterator/)
[![License](https://img.shields.io/pypi/l/concurrent-iterator.svg)](https://pypi.python.org/pypi/concurrent-iterator/)

## Intro

Classes to run producers (iterators) and consumers (coroutines) in a background thread/process.

There are many libraries to create pipelines with stages running in separate processes, a nice
one is [parallelpipe](https://pypi.python.org/pypi/parallelpipe), but this library does something
different. It will lift the entire pipeline up to the point of the Producer into a separate process 
or thread. It's a more coarse library but easier to integrate since things keep looking as normal generators.

## Implementations

There are currently 3 implementations:

* `dummy.Producer`: non-concurrent implementation
* `thread.Producer`: uses a background thread to run the generator
* `process.Producer`: uses a background process to run the generator

`dummy.Producer` is useless in practice.

`thread.Producer` is useful for IO bound generators.

`process.Producer` is useful for CPU or IO bound generators.
It has the complications of dealing with processes (different memory spaces,
logging, etc).
For logging, module [`multiprocessing-logging`](https://github.com/jruere/multiprocessing-logging) can be used.

> **Limitations:** `process.Producer`/`process.Consumer` with generators,
> coroutines and other unpicklable objects require the `fork` start method.
> Python 3.14 changes the default on Linux from `fork` to `forkserver`,
> and `spawn`/`forkserver` cannot pickle generators (`TypeError: cannot
> pickle 'generator' object`). In those cases `process.Producer` now raises
> `RuntimeError` with a clear message. Use `thread.Producer`/`thread.Consumer`,
> a picklable iterable (e.g. `iter([1,2,3])`), or force `fork` via
> `multiprocessing.set_start_method('fork', force=True)` or
> `multiprocessing.get_context('fork').Process`/`Queue` where `fork` is
> available (Linux).

## Usage

Basic example:

    from concurrent_iterator.thread import Producer
    
    ...
    
    items = Producer(slow_generator, maxsize=5)
    
    for item in items:
        [Do some time consuming task]

In the previous example, while doing some time consuming task, the
`slow_generator` will continue running in a background thread and will
pre-calculate up to 5 values.
