SlipGURU Dipartimento di Informatica e Scienze dell'Informazione Università Degli Studi di Genova

job Package

job Package

Provides concrete implementation of job containers.

PPlusJob Module

Provides concrete implementation of the job container that uses PPlus library (https://bitbucket.org/slipguru/pplus). IMPORTANT NOTE: this job container works only with PPlus v0.5.2.

class kdvs.fw.impl.job.PPlusJob.PPlusJobContainer(**kwargs)

Bases: kdvs.fw.Job.JobContainer

Job container that uses PPlus v0.5.2. During instantiation, all parameters except ‘incrementID’, are passed directly to ‘pplus.PPlusConnection’. Refer to the PPlus documentation for more details.

See also

pplus.PPlusConnection

Parameters :

kwargs : dict

keyworded parameters passed directly to PPlusConnection, except ‘incrementID’ (if present)

start()

By default, provide some time for auto–discovery of Parallel Python to warm up (by sleeping for 5 secs).

See also

time.sleep

addJob(job, importable=False)

The job is added to Parallel Python queue and then executed based on internal Parallel Python scheduler. Once added, its status is changed to EXECUTING. The job is actually passed as ‘depfunc’, together with its name, found in proper scope on worker machine, and executed there. NOTE: earlier versions of PPlus will raise an exception here since they require submission of job as callable, not as name.

Parameters :

job : Job

job to be executed by this container

importable : boolean

the job may be self–contained (i.e. does not use any external libraries) or it can use the external code that is already installed (in the sense of importable Python module) on worker machine; in the first case, the job is said to be ‘not importable’, in the second case the job is ‘importable’; True if the job is importable, False if not; False by default

getJobInputKey(jobID)

Get PPlus key for the file that contains job raw input for the specific job.

Parameters :

jobID : string

identifier of already added job

Returns :

key : string

PPlus key for the requested file

Notes

See ‘File key’ in PPlus documentation

getJobOutputKey(jobID)

Get PPlus key for the file that contains job raw output for the specific job.

Parameters :

jobID : string

identifier of already added job

Returns :

key : string

PPlus key for the requested file

Notes

See ‘File key’ in PPlus documentation

getJobIDForKey(filekey)

Get identifier of the job for specific PPlus file key.

Parameters :

key : string

PPlus key for the requested file (input/output)

Returns :

jobID : string

identifier of already added job that is associated with requested file key

Notes

See ‘File key’ in PPlus documentation

close()

Collect all raw results from jobs already added. Blocking call.

Returns :

exceptions : tuple of tuples

tuple of the following tuples (‘PPlusJobs’, e), where ‘e’ is the instance of PPlusError generated during execution of jobs; the exception instance contains the details of the error, including identifier of the job that failed; note that some jobs may still finish normally, so the length of this tuple may vary

See also

PPlusConnection.collect, PPlusError

postClose(destPath, *args)

Perform the following operations AFTER the container has been close()d: copy ‘experiment.log’ to results sublocation specified with file system directory path ‘destPath’, and copy ‘master session’ log to the same sublocation.

Notes

See ‘PPlus logging’ in PPlus documentation

SimpleJob Module

Provides simple ‘null’ job container that executes jobs as ordinary callables in the order of submission; no parallel execution mechanisms are used. It requires no external libraries.

class kdvs.fw.impl.job.SimpleJob.SimpleJobContainer(**kwargs)

Bases: kdvs.fw.Job.JobContainer

Simple ‘null’ job container. It recognizes single parameter ‘incrementID’; if not present, it is assumed to be True.

Parameters :

kwargs : dict

actual parameters supplied during instantiation; they will be checked against reference ones

addJob(job, **kwargs)

The job is added to internal list.

Parameters :

job : Job

job to be executed by this container

kwargs : dict

any other arguments; not used

start()

Finish job submission stage and execute already added jobs, in the order of adding. Blocking call.

close()

Finish execution stage and return any exceptions raised during execution.

Returns :

exception : tuple of tuples

tuple of the following tuples: (jobID, e), where ‘jobID’ is the identifier of the failed job, and ‘e’ is an instance of Exception that was raised during execution; note that some jobs may still finish correctly so the length of this tuple may vary

postClose(destPath, *args)

Do nothing in post–closing stage.

class kdvs.fw.impl.job.SimpleJob.SimpleJobExecutor(jobs_to_execute)

Bases: object

Convenient wrapper for an iterable of jobs that creates simple job container which executes them.

Parameters :

jobs_to_execute : iterable of Job

jobs to be executed

Raises :

Error :

if iterable is incorrectly specified

run()

Create an instance of SimpleJobContainer, add requested jobs, and execute them.

close()

Close simple job container and return any exceptions raised during execution.

getJobResults()

Get iterable of all job results for executed jobs.

See also

JobContainer.getJobResults

Table Of Contents