Package gridmap :: Module job

Module job

source code

This module provides wrappers that simplify submission and collection of jobs, in a more 'pythonic' fashion.


Authors:
Christian Widmer, Cheng Soon Ong, Dan Blanchard (dblanchard@ets.org)
Classes
  Job
Central entity that wraps a function and its data.
Functions
 
process_jobs(jobs, temp_dir=u'/scratch/', wait=True, white_list=None, quiet=True)
Take a list of jobs and process them on the cluster.
source code
 
grid_map(f, args_list, cleanup=True, mem_free=u'1G', name=u'gridmap_job', num_slots=1, temp_dir=u'/scratch/', white_list=None, queue=u'all.q', quiet=True)
Maps a function onto the cluster.
source code
 
pg_map(f, args_list, cleanup=True, mem_free=u'1G', name=u'gridmap_job', num_slots=1, temp_dir=u'/scratch/', white_list=None, queue=u'all.q', quiet=True) source code
Variables
  REDIS_DB = 2
  REDIS_PORT = 7272
  USE_MEM_FREE = False
  DEFAULT_QUEUE = u'all.q'
  __package__ = None
Function Details

process_jobs(jobs, temp_dir=u'/scratch/', wait=True, white_list=None, quiet=True)

source code 

Take a list of jobs and process them on the cluster.

Parameters:
  • temp_dir (basestring) - Local temporary directory for storing output for an individual job.
  • wait (bool) - Should we wait for jobs to finish? (Should only be false if the function you're running doesn't return anything)
  • white_list (list of basestring) - If specified, limit nodes used to only those in list.
  • quiet (bool) - When true, do not output information about the jobs that have been submitted.

grid_map(f, args_list, cleanup=True, mem_free=u'1G', name=u'gridmap_job', num_slots=1, temp_dir=u'/scratch/', white_list=None, queue=u'all.q', quiet=True)

source code 

Maps a function onto the cluster.

Parameters:
  • f (function) - The function to map on args_list
  • args_list (list) - List of arguments to pass to f
  • cleanup (bool) - Should we remove the stdout and stderr temporary files for each job when we're done? (They are left in place if there's an error.)
  • mem_free (basestring) - Estimate of how much memory each job will need (for scheduling). (Not currently used, because our cluster does not have that setting enabled.)
  • name (basestring) - Base name to give each job (will have a number add to end)
  • num_slots (int) - Number of slots each job should use.
  • temp_dir (basestring) - Local temporary directory for storing output for an individual job.
  • white_list (list of basestring) - If specified, limit nodes used to only those in list.
  • queue (basestring) - The SGE queue to use for scheduling.
  • quiet (bool) - When true, do not output information about the jobs that have been submitted.

Note: This can only be used with picklable functions (i.e., those that are defined at the module or class level).

pg_map(f, args_list, cleanup=True, mem_free=u'1G', name=u'gridmap_job', num_slots=1, temp_dir=u'/scratch/', white_list=None, queue=u'all.q', quiet=True)

source code 
Parameters:
  • f (function) - The function to map on args_list
  • args_list (list) - List of arguments to pass to f
  • cleanup (bool) - Should we remove the stdout and stderr temporary files for each job when we're done? (They are left in place if there's an error.)
  • mem_free (basestring) - Estimate of how much memory each job will need (for scheduling). (Not currently used, because our cluster does not have that setting enabled.)
  • name (basestring) - Base name to give each job (will have a number add to end)
  • num_slots (int) - Number of slots each job should use.
  • temp_dir (basestring) - Local temporary directory for storing output for an individual job.
  • white_list (list of basestring) - If specified, limit nodes used to only those in list.
  • queue (basestring) - The SGE queue to use for scheduling.
  • quiet (bool) - When true, do not output information about the jobs that have been submitted.

Deprecated: This function has been renamed grid_map.