tkp.distribute – routines for running on distributed nodes

tkp.distribute implement various computation distribution methods. All sub modules should communicate with the other parts of TKP through the tkp.steps submodule.

class tkp.distribute.Runner(distributor, cores=0)[source]

Bases: object

Parameters:distributor – the name of the distribution method, example multiproc
get_func(func_name)[source]
map(func_name, iterable, args=[])[source]
Parameters:
  • func – The function to be called
  • iterable – a list of objects to iterate over
  • arguments – list of arguments to give to the function
Returns:

the results of all mapped functions
tkp.distribute.serial.map(func, iterable, arguments=[])[source]
tkp.distribute.serial.set_cores(cores=0)[source]

doesn’t do anything for serial

A computation distribution implementation using the build in multiprocessing module. the Pool.map function only accepts one argument, so we need to zip the iterable together with the arguments.

tkp.distribute.multiproc.map(func, iterable, args)[source]
tkp.distribute.multiproc.set_cores(cores=0)[source]

set the number of cores to use. 0 = autodetect