core Package

firework Module

This module contains some of the most central FireWorks classes:

  • A Workflow is a sequence of FireWorks as a DAG (directed acyclic graph)
  • A FireWork defines a workflow step and contains one or more FireTasks along
with its Launches.
  • A Launch describes the run of a FireWork on a computing resource.
  • A FireTaskBase defines the contract for tasks that run within a FireWork (

FireTasks) - A FWAction encapsulates the output of a FireTask and tells FireWorks what to do next after a job completes

class fireworks.core.firework.FWAction(stored_data=None, exit=False, update_spec=None, mod_spec=None, additions=None, detours=None, defuse_children=False)

Bases: fireworks.utilities.fw_serializers.FWSerializable

A FWAction encapsulates the output of a FireTask (it is returned by a FireTask after the FireTask completes). The

FWAction allows a user to store rudimentary output data as well as return commands that alter the workflow.
__init__(stored_data=None, exit=False, update_spec=None, mod_spec=None, additions=None, detours=None, defuse_children=False)
Parameters:stored_data – (dict) data to store from the run. Does not

affect the operation of FireWorks. :param exit: (bool) if set to True, any remaining FireTasks within the same FireWork are skipped. :param update_spec: (dict) specifies how to update the child FW’s spec :param mod_spec: ([dict]) update the child FW’s spec using the DictMod language (more flexible than update_spec) :param additions: ([Workflow]) a list of WFs/FWs to add as children :param detours: ([Workflow]) a list of WFs/FWs to add as children ( they will inherit the current FW’s children) :param defuse_children: (bool) defuse all the original children of this FireWork

classmethod from_dict(*args, **kwargs)
skip_remaining_tasks

If the FWAction gives any dynamic action, we skip the subsequent FireTasks

Returns:(bool)
to_dict(*args, **kwargs)
class fireworks.core.firework.FireTaskBase(*args, **kwargs)

Bases: dict, fireworks.utilities.fw_serializers.FWSerializable

FireTaskBase is used like an abstract class that defines a computing task (FireTask). All FireTasks should inherit from FireTaskBase.

You can set parameters of a FireTask like you’d use a dict.

__init__(*args, **kwargs)
classmethod from_dict(*args, **kwargs)
required_params = []
run_task(fw_spec)

This method gets called when the FireTask is run. It can take in a FireWork spec, perform some task using that data, and then return an output in the form of a FWAction.

Parameters:fw_spec – (dict) a FireWork spec
Returns:(FWAction)
to_dict(*args, **kwargs)
class fireworks.core.firework.FireTaskMeta(name, bases, dct)

Bases: type

__init__(name, bases, dct)
class fireworks.core.firework.FireWork(tasks, spec=None, name=None, launches=None, archived_launches=None, state='WAITING', created_on=None, fw_id=-1)

Bases: fireworks.utilities.fw_serializers.FWSerializable

A FireWork is a workflow step and might be contain several FireTasks

STATE_RANKS = {'RUNNING': 5, 'ARCHIVED': -1, 'WAITING': 1, 'FIZZLED': 4, 'READY': 2, 'RESERVED': 3, 'COMPLETED': 7, 'DEFUSED': 0}
__init__(tasks, spec=None, name=None, launches=None, archived_launches=None, state='WAITING', created_on=None, fw_id=-1)
Parameters:
  • tasks – ([FireTask]) a list of FireTasks to run in sequence
  • spec – (dict) specification of the job to run. Used by the

FireTask :param launches: ([Launch]) a list of Launch objects of this FireWork :param archived_launches: ([Launch]) a list of archived Launch objects of this FireWork :param state: (str) the state of the FW (e.g. WAITING, RUNNING, COMPLETED, ARCHIVED) :param fw_id: (int) an identification number for this FireWork

classmethod from_dict(*args, **kwargs)
to_db_dict()
to_dict(*args, **kwargs)
class fireworks.core.firework.Launch(state, launch_dir, fworker=None, host=None, ip=None, trackers=None, action=None, state_history=None, launch_id=None, fw_id=None)

Bases: fireworks.utilities.fw_serializers.FWSerializable, object

A Launch encapsulates data about a specific run of a FireWork on a computing resource

__init__(state, launch_dir, fworker=None, host=None, ip=None, trackers=None, action=None, state_history=None, launch_id=None, fw_id=None)
Parameters:
  • state – (str) the state of the Launch (e.g. RUNNING, COMPLETED)
  • launch_dir – (str) the directory where the Launch takes place
  • fworker – (FWorker) The FireWorker running the Launch
  • host – (str) the hostname where the launch took place (set

automatically if None) :param ip: (str) the IP address where the launch took place (set automatically if None) :param trackers: ([Tracker]) File Trackers for this Launch :param action: (FWAction) the output of the Launch :param state_history: ([dict]) a history of all states of the Launch and when they occurred :param launch_id: (int) launch_id set by the LaunchPad :param fw_id: (int) id of the FireWork this Launch is running

classmethod from_dict(*args, **kwargs)
last_pinged
Returns:(datetime) the time the Launch last pinged a heartbeat that

it was still running

reservedtime_secs
Returns:(int) number of seconds the Launch was stuck as RESERVED in

a queue

runtime_secs
Returns:(int) the number of seconds that the Launch ran for
set_reservation_id(reservation_id)

Adds the job_id to the reservation

Parameters:reservation_id – (str) the id of the reservation (e.g.,

queue reservation)

state
Returns:(str) The current state of the Launch.
time_end
Returns:(datetime) the time the Launch was COMPLETED or FIZZLED
time_reserved
Returns:(datetime) the time the Launch was RESERVED in the queue
time_start
Returns:(datetime) the time the Launch started RUNNING
to_db_dict(*args, **kwargs)
to_dict(*args, **kwargs)
touch_history(update_time=None)

Updates the update_at field of the state history of a Launch. Used to ping that a Launch is still alive.

class fireworks.core.firework.Tracker(filename, nlines=25, content='')

Bases: fireworks.utilities.fw_serializers.FWSerializable, object

A Tracker monitors a file and returns the last N lines for updating the Launch object

MAX_TRACKER_LINES = 1000
__init__(filename, nlines=25, content='')
classmethod from_dict(m_dict)
to_dict()
track_file(launch_dir=None)

Reads the monitored file and returns back the last N lines :param launch_dir: directory where job was launched in case of relative filename :return:

class fireworks.core.firework.Workflow(fireworks, links_dict=None, name=None, metadata=None, created_on=None, updated_on=None)

Bases: fireworks.utilities.fw_serializers.FWSerializable

A Workflow connects a group of FireWorks in an execution order

Bases: dict, fireworks.utilities.fw_serializers.FWSerializable

An inner class for storing the DAG links between FireWorks

__init__(*args, **kwargs)
classmethod from_dict(m_dict)
nodes
to_db_dict()
to_dict()
Workflow.__init__(fireworks, links_dict=None, name=None, metadata=None, created_on=None, updated_on=None)
Parameters:
  • fireworks – ([FireWork]) - all FireWorks in this workflow
  • links_dict – (dict) links between the FWs as (parent_id):[(

child_id1, child_id2)] :param metadata: (dict) metadata for this Workflow

Workflow.apply_action(action, fw_id)

Apply a FWAction on a FireWork in the Workflow

Parameters:
  • action – (FWAction) action to apply
  • fw_id – (int) id of FireWork on which to apply the action
Returns:

([int]) list of FireWork ids that were updated or new

classmethod Workflow.from_FireWork(fw, name=None, metadata=None)
classmethod Workflow.from_dict(m_dict)
Workflow.fws
Workflow.leaf_fw_ids

Gets leaf FireWorks of this workflow (those with no children)

Returns:([int]) FireWork ids of leaf FWs
Workflow.refresh(fw_id, updated_ids=None)

Refreshes the state of a FireWork and any affected children.

Parameters:
  • fw_id – (int) id of the FireWork on which to perform the refresh
  • updated_ids – ([int])
Returns:

([int]) list of FireWork ids that were updated

Workflow.rerun_fw(fw_id, updated_ids=None)

Archives the launches of a FireWork so that it can be re-run. :param fw_id: (int) :return: ([int]) list of FireWork ids that were updated

Workflow.root_fw_ids

Gets root FireWorks of this workflow (those with no parents)

Returns:([int]) FireWork ids of root FWs
Workflow.state
Workflow.to_db_dict()
Workflow.to_dict()
Workflow.to_display_dict()

fw_config Module

A set of global constants for FireWorks (Python code as a config file)

fireworks.core.fw_config.FWConfig(*args, **kwargs)
fireworks.core.fw_config.FWData(*args, **kwargs)
fireworks.core.fw_config.singleton(class_)

fworker Module

This module contains FireWorker, which encapsulates information about a computing resource

class fireworks.core.fworker.FWorker(name='Automatically generated Worker', category='', query=None)

Bases: fireworks.utilities.fw_serializers.FWSerializable

__init__(name='Automatically generated Worker', category='', query=None)
Parameters:
  • name – the name of the resource, should be unique
  • category – a String describing the computing resource, does not need to be unique
  • query – a dict query that restricts the type of FireWork this resource will run
classmethod from_dict(*args, **kwargs)
query
to_dict(*args, **kwargs)

launchpad Module

The LaunchPad manages the FireWorks database.

class fireworks.core.launchpad.LaunchPad(host='localhost', port=27017, name='fireworks', username=None, password=None, logdir=None, strm_lvl=None, user_indices=None, wf_user_indices=None)

Bases: fireworks.utilities.fw_serializers.FWSerializable

The LaunchPad manages the FireWorks database.

__init__(host='localhost', port=27017, name='fireworks', username=None, password=None, logdir=None, strm_lvl=None, user_indices=None, wf_user_indices=None)
Parameters:
  • host
  • port
  • name
  • username
  • password
  • logdir
  • strm_lvl
  • user_indices
  • wf_user_indices
add_offline_run(launch_id, fw_id, name)
add_wf(wf, reassign_all=True)
Parameters:wf – a Workflow object.
archive_wf(fw_id)
classmethod auto_load()
checkout_fw(fworker, launch_dir, fw_id=None, host=None, ip=None)

(internal method) Finds a FireWork that’s ready to be run, marks it as running, and returns it to the caller. The caller is responsible for running the FireWork.

Parameters:
  • fworker – A FWorker instance
  • host – the host making the request (for creating a Launch object)
  • ip – the ip making the request (for creating a Launch object)
  • launch_dir – the dir the FW will be run in (for creating a Launch object)
Returns:

a FireWork, launch_id tuple

complete_launch(launch_id, action, state='COMPLETED')

(internal method) used to mark a FireWork’s Launch as completed. :param launch_id: :param action: the FWAction of what to do next

defuse_fw(fw_id)
defuse_wf(fw_id)
detect_lostruns(expiration_secs=14400, fizzle=False, rerun=False, max_runtime=None)
detect_unreserved(expiration_secs=1209600, rerun=False)
forget_offline(fw_id)
classmethod from_dict(d)
get_fw_by_id(fw_id)

Given a FireWork id, give back a FireWork object

Parameters:fw_id – FireWork id (int)
Returns:FireWork object
get_fw_ids(query=None, sort=None, limit=0, count_only=False)

Return all the fw ids that match a query, :param query: (dict) representing a Mongo query :param sort: [(str,str)] sort argument in Pymongo format :param limit: (int) limit the results :param count_only: (bool) only return the count rather than explicit ids

get_launch_by_id(launch_id)

Given a Launch id, return details of the Launch

Parameters:launch_id – launch id
Returns:Launch object
get_logdir()
get_new_fw_id()

Checkout the next FireWork id

get_new_launch_id()

Checkout the next Launch id

get_tracker_data(fw_id)
get_wf_by_fw_id(fw_id)

Given a FireWork id, give back the Workflow containing that FireWork :param fw_id: :return: A Workflow object

get_wf_ids(query=None, sort=None, limit=0, count_only=False)

Return one fw id for all workflows that match a query, :param query: (dict) representing a Mongo query :param sort: [(str,str)] sort argument in Pymongo format :param limit: (int) limit the results :param count_only: (bool) only return the count rather than explicit ids

maintain(infinite=True, maintain_interval=None)
mark_fizzled(launch_id)
ping_launch(launch_id, ptime=None)
recover_offline(launch_id, ignore_errors=False)
reignite_fw(fw_id)
reignite_wf(fw_id)
rerun_fw(fw_id)
reset(password, require_password=True)

Create a new FireWorks database. This will overwrite the existing FireWorks database! To safeguard against accidentally erasing an existing database, a password must be entered.

Parameters:
  • password – A String representing today’s date, e.g. ‘2012-12-31’
  • require_password – Whether a password is required to reset the DB. Highly recommended to leave this set to True, otherwise you are inviting disaster!
run_exists(fworker=None)

Checks to see if the database contains any FireWorks that are ready to run :return: (T/F)

set_priority(fw_id, priority)
to_dict()

Note: usernames/passwords are exported as unencrypted Strings!

tuneup()
unreserve(launch_id)

rocket Module

A Rocket fetches a FireWork from the database, runs the sequence of FireTasks inside, and then completes the Launch

class fireworks.core.rocket.Rocket(launchpad, fworker, fw_id)

The Rocket fetches a workflow step from the FireWorks database and executes it.

__init__(launchpad, fworker, fw_id)
Parameters:
  • launchpad – (LaunchPad) A LaunchPad object for interacting with the FW database. If none, reads FireWorks from FW.json and writes to FWAction.json
  • fworker – (FWorker) A FWorker object describing the computing resource
  • fw_id – (int) id of a specific FireWork to run (quit if it cannot be found)
run()

Run the rocket (check out a job from the database and execute it)

fireworks.core.rocket.do_ping(launchpad, launch_id)
fireworks.core.rocket.ping_launch(launchpad, launch_id, stop_event, master_thread)
fireworks.core.rocket.start_ping_launch(launch_id, lp)
fireworks.core.rocket.stop_ping_launch(ping_stop)

rocket_launcher Module

This module contains methods for launching Rockets, both singly and in rapid-fire mode

fireworks.core.rocket_launcher.launch_rocket(launchpad, fworker=None, fw_id=None, strm_lvl='INFO')

Run a single rocket in the current directory :param launchpad: (LaunchPad) :param fworker: (FWorker) :param fw_id: (int) if set, a particular FireWork to run :param strm_lvl: (str) level at which to output logs to stdout

fireworks.core.rocket_launcher.rapidfire(launchpad, fworker=None, m_dir=None, nlaunches=0, max_loops=-1, sleep_time=None, strm_lvl='INFO')

Keeps running Rockets in m_dir until we reach an error. Automatically creates subdirectories for each Rocket. Usually stops when we run out of FireWorks from the LaunchPad.

Parameters:
  • launchpad – (LaunchPad)
  • fworker – (FWorker object)
  • m_dir – (str) the directory in which to loop Rocket running
  • nlaunches – (int) 0 means ‘until completion’, -1 or “infinite” means to loop forever
  • max_loops – (int) maximum number of loops
  • sleep_time – (int) secs to sleep between rapidfire loop iterations
  • strm_lvl – (str) level at which to output logs to stdout

Table Of Contents

This Page