There are two kinds of manager: those that store data in a shared memory map and those that use a server process.
An instance of LocalManager can be created for sharing data in shared memory. For creating objects in shared memory it has the following methods:
- SharedValue(format, value)
Creates a shared value object. format must be a format string of the kind used by the struct module. The format must be for a struct type which corresponds to a tuple of length 1. The shared value is initialized with value.
A SharedValue object has a value property for getting or setting its contents.
For example to create and modify a shared string of length less than 256 one can do
>>> manager = LocalManager() >>> string = manager.SharedValue('256p', 'hello') >>> print string SharedValue('256p', 'hello') >>> string.value = 'goodbye' >>> print string SharedValue('256p', 'goodbye')- SharedStruct(format, value)
Creates a shared struct object. format must be a format string of the kind used by the struct module. The format must be for a struct type with length 1. The shared value is initialized with value which should be a tuple or list.
A SharedStruct object has a value property for getting or setting its value. Note that the value will always be a tuple even if it only has length 1.
For example to create and modify a struct of two short integers one can do
>>> manager = LocalManager() >>> struct = manager.SharedStruct('hh', (0, 0)) >>> print struct SharedStruct('hh', (0, 0)) >>> string.value = (1, 2) >>> print struct SharedStruct('hh', (1, 2))- SharedArray(format, value)
Creates a shared array object. format must be a format string of the kind used by the array module. The shared value is initialized with sequence which should be an iterable.
A SharedArray object can be used like an array.array object but it has fixed length and does not support negative indices.
For example to create and modify an array of ten integers one can do
>>> manager = LocalManager() >>> arr = manager.SharedArray('i', [0]*10) >>> print arr SharedArray('i', [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) >>> arr[:5] = [1, 2, 3, 4, 5] >>> print arr SharedArray('i', [1, 2, 3, 4, 5, 0, 0, 0, 0, 0]) >>> print arr[:] array('i', [1, 2, 3, 4, 5, 0, 0, 0, 0, 0]) >>> print arr.tolist() [1, 2, 3, 4, 5, 0, 0, 0, 0, 0] >>> arr.tostring() '\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04 ...
For compatibility with SyncManager (see below) LocalManager also has a shutdown() which does nothing and attributes Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue which are just aliases for functions in the processing namespace. However, unlike a SyncManager object a LocalManager object cannot be used to create list, dict or Namespace objects.
The space allocated to a SharedValue, SharedStruct or SharedArray object will be cleaned up when that object is garbage collected by the process that initially created it. Note, however, that if it was passed as an argument to the Process() constructor then it cannot be garbage collected until that process has completed.
A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.
Manager processes are daemonic so if they are still around when their parent process exits then it will attempt to kill them off.
The manager classes are defined in the processing.managers module.
BaseManager is the base class for all manager classes which use a server process. It does not possess any methods which create shared objects.
The public methods of BaseManager are the following:
- __init__(self, address=None, authkey=None)
Creates a manager object.
Once created one should call start(), serve_forever() or connect() to ensure that the manager object refers to a started manager process.
The arguments to the constructor are as follows:
- address
The address on which the manager process listens for new connections. If address is None then an arbitrary one is chosen.
See Listener objects.
- authkey
The authentication key which will be used to check the validity of incoming connections to the server process.
If authkey is None then currentProcess().getAuthKey(). Otherwise authkey is used and it must be a string.
See Authentication keys.
- start()
- Spawn or fork a subprocess to start the manager.
- serve_forever()
- Start the manager in the current process. See Using a remote manager.
- connect()
- Connect to an already created remote manager process. See Using a remote manager.
- shutdown()
Stop the process used by the manager. This is only available if start() has been used to start a process.
This can be called multiple times.
BaseManager instances also has one read-only property:
- address
- The address used by the manager.
BaseManager supports the context manager protocol which helps to ensure that shutdown() method is eventually called. shutdown() is also registered as a wekref callback, so it will be called when the manager is garbage collected.
The creation of managers which support arbitrary types is discussed below in Customized managers.
SyncManager is a subclass of BaseManager which can be used for the synchronization of processes. Objects of this type are returned by processing.Manager().
SyncManager support equivalents of all the methods of LocalManager, but each creates a shared object in the server process.
It also supports creation of shared lists and dictionaries. The instance methods defined by SyncManager are
- BoundedSemaphore(value=1)
- Creates a shared threading.BoundedSemaphore object and returns a proxy for it.
- Condition(lock=None)
Creates a shared threading.Condition object and returns a proxy for it.
If lock is supplied then it should be a proxy for a threading.Lock or threading.RLock object.
- Event()
- Creates a shared threading.Event object and returns a proxy for it.
- Lock()
- Creates a shared threading.Lock object and returns a proxy for it.
- Namespace()
Creates a shared Namespace object and returns a proxy for it.
See Namespace objects.
- Queue(maxsize=0)
- Creates a shared Queue.Queue object and returns a proxy for it.
- RLock()
- Creates a shared threading.RLock object and returns a proxy for it.
- Semaphore(value=1)
- Creates a shared threading.Semaphore object and returns a proxy for it.
- SharedArray(format, sequence)
- Create a shared SharedArray object and returns a proxy for it. (format is ignored.)
- SharedStruct(format, value)
- Create a shared SharedStruct object and returns a proxy for it. (format is ignored.)
- SharedValue(format, value)
- Create a shared SharedValue object and returns a proxy for it.
- dict(), dict(mapping), dict(sequence)
- Creates a shared dict object and returns a proxy for it.
- list(), list(sequence)
- Creates a shared list object and returns a proxy for it.
A namespace object has no public methods but does have writable attributes. Its representation shows the values of its attributes.
However, when using a proxy for a namespace object, an attribute beginning with '_' will be an attribute of the proxy and not an attribute of the referent:
>>> manager = processing.Manager() >>> Global = manager.Namespace() >>> Global.x = 10 >>> Global.y = 'hello' >>> Global._z = 12.3 # this is an attribute of the proxy >>> print Global Namespace(x=10, y='hello')
To create one's own manager one creates a subclass of BaseManager.
To create a method of the subclass which will create new shared objects one uses the following function:
- CreatorMethod(callable=None, proxytype=None, exposed=None, typeid=None)
Returns a function with signature func(self, *args, **kwds) which will create a shared object using the manager self and return a proxy for it.
The shared objects will be created by evaluating callable(*args, **kwds) in the manager process.
The arguments are:
- callable
- The callable used to create a shared object. If the manager will connect to a remote manager then this is ignored.
- proxytype
The type of proxy which will be used for object returned by callable.
If proxytype is None then each time an object is returned by callable either a new proxy type is created or a cached one is reused. The methods of the shared object which will be exposed via the proxy will then be determined by the exposed argument, see below.
- exposed
Given a shared object returned by callable, the exposed argument is the list of those method names which should be exposed via BaseProxy._callmethod(). [1] [2]
If exposed is None and callable.__exposed__ exists then callable.__exposed__ is used instead.
If exposed is None and callable.__exposed__ does not exist then all methods of the shared object which do not start with '_' will be exposed.
An attempt to use BaseProxy._callmethod() with a method name which is not exposed will raise an exception.
- typeid
- If typeid is a string then it is used as an identifier for the callable. Otherwise, typeid must be None and a string prefixed by callable.__name__ is used as the identifier.
[1] | A method here means any attribute which has a __call__ attribute. |
[2] | The method names __repr__, __str__, and __cmp__ of a shared object are always exposed by the manager. However, instead of invoking the __repr__(), __str__(), __cmp__() instance methods (none of which are guaranteed to exist) they invoke the builtin functions repr(), str() and cmp(). Note that one should generally avoid exposing rich comparison methods like __eq__(), __ne__(), __le__(). To make the proxy type support comparison by value one can just expose __cmp__() instead (even if the referent does not have such a method). |
from processing.managers import BaseManager, CreatorMethod class FooClass(object): def bar(self): print 'BAR' def baz(self): print 'BAZ' class NewManager(BaseManager): Foo = CreatorMethod(FooClass) if __name__ == '__main__': manager = NewManager() manager.start() foo = manager.Foo() foo.bar() # prints 'BAR' foo.baz() # prints 'BAZ' manager.shutdown()
See test_newtype.py for more examples.
It is possible to run a manager server on one machine and have clients use it from other machines (assuming that the firewalls involved allow it).
Running the following commands creates a server for a shared queue which remote clients can use:
>>> from processing.managers import BaseManager, CreatorMethod >>> import Queue >>> queue = Queue.Queue() >>> class QueueManager(BaseManager): ... get_proxy = CreatorMethod(callable=lambda:queue, typeid='get_proxy') ... >>> m = QueueManager(address=('foo.bar.org', 50000), authkey='none') >>> m.serve_forever()
One client can access the server as follows:
>>> from processing.managers import BaseManager, CreatorMethod >>> class QueueManager(BaseManager): ... get_proxy = CreatorMethod(typeid='get_proxy') ... >>> m = QueueManager(address=('foo.bar.org', 50000), authkey='none') >>> m.connect() >>> queue = m.get_proxy() >>> queue.put('hello')
Another client can also use it:
>>> from processing.managers import BaseManager, CreatorMethod >>> class QueueManager(BaseManager): ... get_proxy = CreatorMethod(typeid='get_proxy') ... >>> m = QueueManager(address=('foo.bar.org', 50000), authkey='none') >>> m.connect() >>> queue = m.get_proxy() >>> queue.get() 'hello'