Module threadpool
[frames | no frames]

Module threadpool

Easy to use object-oriented thread pool framework.

A thread pool is an object that maintains a pool of worker threads to perform time consuming operations in parallel. It assigns jobs to the threads by putting them in a work request queue, where they are picked up by the next available thread. This then performs the requested operation in the background and puts the results in a another queue.

The thread pool object can then collect the results from all threads from this queue as soon as they become available or after all threads have finished their work. It's also possible, to define callbacks to handle each result as it comes in.

The basic concept and some code was taken from the book "Python in a Nutshell" by Alex Martelli, copyright 2003, ISBN 0-596-00188-6, from section 14.5 "Threaded Program Architecture". I wrapped the main program logic in the ThreadPool class, added the WorkRequest class and the callback system and tweaked the code here and there. Kudos also to Florent Aide for the exception handling mechanism.

Basic usage:

>>> pool = TreadPool(poolsize)
>>> requests = makeRequests(some_callable, list_of_args, callback)
>>> [pool.putRequest(req) for req in requests]
>>> pool.wait()

See the end of the module code for a brief, annotated usage example.

Website : http://chrisarndt.de/en/software/python/threadpool/


Classes
ThreadPool A thread pool, distributing work requests and collecting results.
WorkerThread Background thread connected to the requests/results queues.
WorkRequest A request to execute a callable for putting in the request queue later.

Exceptions
NoResultsPending All work requests have been processed.
NoWorkersAvailable No worker threads available to process remaining requests.

Function Summary
  makeRequests(callable, args_list, callback, exc_callback)
Create several work requests for same callable with different arguments.

Function Details

makeRequests(callable, args_list, callback=None, exc_callback=None)

Create several work requests for same callable with different arguments.

Convenience function for creating several work requests for the same callable where each invocation of the callable receives different values for its arguments.

args_list contains the parameters for each invocation of callable. Each item in 'args_list' should be either a 2-item tuple of the list of positional arguments and a dictionary of keyword arguments or a single, non-tuple argument.

See docstring for WorkRequest for info on callback and exc_callback.


Generated by Epydoc 2.1 on Fri Jun 23 13:37:22 2006 http://epydoc.sf.net