queue Package

queue Package

queue_adapter Module

class fireworks.queue.queue_adapter.QueueAdapterBase

Bases: fireworks.utilities.fw_serializers.FWSerializable

The QueueAdapter is responsible for all interactions with a specific queue management system. This includes handling all details of queue script format as well as queue submission and management.

A user should extend this class with implementations that work on specific queue systems.

classmethod from_dict(m_dict)
get_njobs_in_queue(queue_params, username=None)

returns the number of jobs in the queue, probably using subprocess or shutil to call a command like ‘qstat’. returns None when the number of jobs cannot be determined.

Parameters:
  • queue_params – a QueueParams() instance
  • username – the username of the jobs to count (default is to autodetect)
get_script_str(queue_params, launch_dir)

returns a (multi-line) String representing the queue script, e.g. PBS script. All details of the queue script must be specified in the QueueParams().

Parameters:
  • queue_params – A QueueParams() instance
  • launch_dir – The directory the job will be launched in
submit_to_queue(queue_params, script_file)

submits the job to the queue, probably using subprocess or shutil :param queue_params: A QueueParams() instance :param script_file: name of the script file to use (String)

to_dict(*args, **kwargs)
class fireworks.queue.queue_adapter.QueueParams(queue_adapter, params, logging_dir='.')

Bases: fireworks.utilities.fw_serializers.FWSerializable

A QueueParams instance contains all the information needed to write a queue file and submit to a queue system. Details of the queue file format and queue submission commands should be included in the QueueAdapterBase object. Specific parameters used by the QueueAdapterBase should be included in the params variable.

__init__(queue_adapter, params, logging_dir='.')
Parameters:
  • queue_adapter – An implementation of QueueAdapterBase()
  • params – Additional parameters (dict) that the QueueAdapter might need.
  • logging_dir – Directory (String) to write logs to
classmethod from_dict(m_dict)

Note: The QueueAdapter is loaded based on its fw_name alone See the docs for to_dict() for more details

to_dict()

Note: the QueueAdapter is being serialized using the FW name alone This keeps the serialization compact and easy to edit by humans. The from_dict() will dynamically find the correct QueueAdapter using its fw_name.

queue_launcher Module

This module is used to submit jobs to a queue on a cluster. It can submit a single job, or if used in “rapid-fire” mode, can submit multiple jobs within a directory structure. The details of job submission and queue communication are handled using QueueParams, which specifies a QueueAdapter as well as desired properties of the submit script.

fireworks.queue.queue_launcher.launch_rocket_to_queue(queue_params, launcher_dir='.', strm_lvl=None, launchpad=None, fworker=None, reserve=False)

Submit a single job to the queue.

Parameters:
  • queue_params – A QueueParams instance
  • launcher_dir – The directory where to submit the job
fireworks.queue.queue_launcher.rapidfire(queue_params, launch_dir='.', njobs_queue=10, njobs_block=500, strm_lvl=None, nlaunches=0, sleep_time=60, launchpad=None, fworker=None, reserve=False)

Submit many jobs to the queue.

Parameters:
  • queue_params – A QueueParams instance
  • launch_dir – directory where we want to write the blocks
  • njobs_queue – stops submitting jobs when njobs_queue jobs are in the queue
  • njobs_block – automatically write a new block when njobs_block jobs are in a single block

Table Of Contents

This Page