music21.common.parallel¶
Functions¶
-
music21.common.parallel.
cpus
()¶ Returns the number of CPUs or if >= 3, one less (to leave something out for multiprocessing)
-
music21.common.parallel.
runNonParallel
(iterable, parallelFunction, updateFunction=None, updateMultiply=3, unpackIterable=False)¶ This is intended to be a perfect drop in replacement for runParallel, except that it runs on one core only, and not in parallel.
Used, for instance, if we’re already in a parallel function.
-
music21.common.parallel.
runParallel
(iterable, parallelFunction, updateFunction=None, updateMultiply=3, unpackIterable=False)¶ runs parallelFunction over iterable in parallel, optionally calling updateFunction after each common.cpus * updateMultiply calls.
Setting updateMultiply too small can make it so that cores wait around when they could be working if one CPU has a particularly hard task. Setting it too high can make it seem like the job has hung.
updateFunction should take three arguments: the current position, the total to run, and the most recent results. It does not need to be pickleable, and in fact, a bound method might be very useful here. Or updateFunction can be “True” which just prints a generic message.
If unpackIterable is True then each element in iterable is considered a list or tuple of different arguments to delayFunction.
As of Python 2.7, partial functions are pickleable, so if you need to pass the same arguments to parallelFunction each time, make it a partial function before passing it to runParallel.
Note that parallelFunction, iterable’s contents, and the results of calling parallelFunction must all be pickleable, and that if pickling the contents or unpickling the results takes a lot of time, you won’t get nearly the speedup from this function as you might expect. The big culprit here is definitely music21 streams.