There are certain guidelines and idioms which should be adhered to when using the processing package.
As far as possible one should try to avoid shifting large amounts of data between processes.
It is probably best to stick to using queues or pipes for communication between processes rather than using the lower level synchronization primitives from the threading module.
Do not use a proxy object from more than one thread unless you protect it with a lock.
Alternatively another copy of the proxy can be created using the copy.copy() function.
(There is never a problem with different processes using the 'same' proxy.)
On Unix a child process can make use of a shared resource created in a parent process using a global resource. However, it is better to pass the object as an argument to the constructor for the child process.
Apart from making the code (potentially) compatible with Windows this also ensures that as long as the child process is still alive the object will not be garbage collected in the parent process. This might be important if some resource is freed when the object is garbage collected in the parent process.
So for instance
from processing import Process, Lock def f(): ... do something using "lock" ... if __name__ == '__main__': lock = Lock() for i in range(10): Process(target=f).start()
should be rewritten as
from processing import Process, Lock def f(l): ... do something using "l" ... if __name__ == '__main__': lock = Lock() for i in range(10): Process(target=f, args=[lock]).start()
Since Windows lacks os.fork() it has a few extra restrictions:
More picklability:
Ensure that all arguments to Process.__init__() are picklable. This means, in particular, that bound or unbound methods cannot be used directly as the target argument on Windows --- just define a function and use that instead.
Also, if you subclass Process then make sure that instances will be picklable when the start() method is called.
Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that start() was called.
However, global variables which are just module level constants cause no problems.
Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).
For example, under Windows running the following module would recursively create new processes until you run out of memory or get a crash:
from processing import Process def foo(): print 'hello' p = Process(target=foo) p.start()
Instead one should protect creation of the new process by using if __name__ == '__main__': as follows:
from processing import Process def foo(): print 'hello' if __name__ == '__main__': p = Process(target=foo) p.start()
This allows the newly spawned Python interpreter to safely import the module and then run the module's foo() function.
Similar restrictions apply if a pool or manager is created in the main module.
One can produce Windows executables from a python program by using py2exe, PyInstaller, cx_Freeze etc. However, if the program uses processing then one needs to call freezeSupport() immediately after the if __name__ == '__main__': line of the main module. Otherwise one will probably get the same problems mentioned above concerning Safe importing. For example
from processing import Process, freezeSupport def foo(): print 'hello' if __name__ == '__main__': freezeSupport() p = Process(target=foo) p.start()
Note that calling freezeSupport() at some other point in the main module is likely to cause problems.