Bases: fireworks.utilities.fw_serializers.FWSerializable
A FWAction encapsulates the output of a FireTask (it is returned by a FireTask after the FireTask completes). The FWAction allows a user to store rudimentary output data as well as return commands that alter the workflow.
If the FWAction gives any dynamic action, we skip the subsequent FireTasks
Returns: | bool |
---|
Bases: collections.defaultdict, fireworks.utilities.fw_serializers.FWSerializable
FireTaskBase is used like an abstract class that defines a computing task (FireTask). All FireTasks should inherit from FireTaskBase.
You can set parameters of a FireTask like you’d use a dict.
This method gets called when the FireTask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.
Parameters: | fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources. |
---|---|
Returns: | (FWAction) |
Bases: abc.ABCMeta
Bases: fireworks.utilities.fw_serializers.FWSerializable
A Firework is a workflow step and might be contain several FireTasks.
Return firework dict with updated launches and state.
Bases: fireworks.utilities.fw_serializers.FWSerializable, object
A Launch encapsulates data about a specific run of a Firework on a computing resource.
**Returns* – datetime* – the time the Launch last pinged a heartbeat that it was still running
**Returns* – int* – number of seconds the Launch was stuck as RESERVED in a queue.
Adds the job_id to the reservation.
Parameters: | reservation_id (int or str) – the id of the reservation (e.g., queue reservation) |
---|
Bases: fireworks.utilities.fw_serializers.FWSerializable, object
A Tracker monitors a file and returns the last N lines for updating the Launch object.
Bases: fireworks.utilities.fw_serializers.FWSerializable
A Workflow connects a group of FireWorks in an execution order.
Bases: dict, fireworks.utilities.fw_serializers.FWSerializable
An inner class for storing the DAG links between FireWorks
Return list of all nodes
Return a dict of child and its parents.
Note: if performance of parent_links becomes an issue, override delitem/setitem to update parent_links
Convert to str form for Mongo, which cannot have int keys .
Returns: | dict |
---|
Convert to str form for Mongo, which cannot have int keys.
Returns: | dict |
---|
Method to add a workflow as a child to a Firework Note: detours must have children that have STATE_RANK that is WAITING or below
Parameters: | |
---|---|
Returns: | list of Firework ids that were updated or new |
Return type: | [int] |
Apply a FWAction on a Firework in the Workflow.
Parameters: |
|
---|---|
Returns: | list of Firework ids that were updated or new |
Return type: | [int] |
Return Workflow from the given Firework.
Parameters: | |
---|---|
Returns: | Workflow |
Return Workflow from its dict representation.
Parameters: | m_dict (dict) – either a Workflow dict or a Firework dict |
---|---|
Returns: | Workflow |
Create a fresh Workflow from an existing one.
Parameters: | wflow (Workflow) – |
---|---|
Returns: | Workflow |
Return list of all fireworks
Gets leaf FireWorks of this workflow (those with no children).
Returns: | Firework ids of leaf FWs |
---|---|
Return type: | [int] |
Refreshes the state of a Firework and any affected children.
Parameters: |
|
---|---|
Returns: | list of Firework ids that were updated |
Return type: | set(int) |
Archives the launches of a Firework so that it can be re-run.
Parameters: |
|
---|---|
Returns: | list of Firework ids that were updated |
Return type: | [int] |
Reset the states of all Fireworks in this workflow to ‘WAITING’.
Parameters: | reset_ids (bool) – if True, give each Firework a new id. |
---|
Gets root FireWorks of this workflow (those with no parents).
Returns: | Firework ids of root FWs |
---|---|
Return type: | [int] |
Bases: fireworks.utilities.fw_serializers.FWSerializable
Returns FWorker object from settings file(my_fworker.yaml).
Returns updated query dict.
Bases: fireworks.utilities.fw_serializers.FWSerializable
The LaunchPad manages the FireWorks database.
Add the launch and firework to the offline_run collection.
Parameters: |
---|
Add workflow(or firework) to the launchpad. The firework ids will be reassigned.
Parameters: | wf (Workflow/Firework) – |
---|---|
Returns: | mapping between old and new Firework ids |
Return type: | dict |
Append a new workflow on top of an existing workflow.
Parameters: |
|
---|
Archive the workflow containing the given firework id.
Parameters: | fw_id (int) – firework id |
---|
given the launch id, cancel the reservation and rerun the fireworks
Given the reservation id, cancel the reservation and rerun the corresponding fireworks.
Change the launch directory corresponding to the given launch id.
Parameters: |
---|
Checkout the next ready firework, mark it with the given state(RESERVED or RUNNING) and return it to the caller. The caller is responsible for running the Firework.
Parameters: |
|
---|---|
Returns: | firework and the new launch id |
Return type: | (Firework, int) |
Internal method used to mark a Firework’s Launch as completed.
Parameters: | |
---|---|
Returns: | updated launch |
Return type: | dict |
Given the firework id, defuse the firework and refresh the workflow.
Parameters: |
---|
Defuse the workflow containing the given firework id.
Parameters: |
---|
Delete the workflow containing firework with the given id.
Parameters: | fw_id (int) – Firework id |
---|
Detect lost runs i.e running fireworks that haven’t been updated within the specified time limit or running firework whose launch has been marked fizzed or completed.
Parameters: |
|
---|---|
Returns: |
|
Return type: | ([int], [int], [int]) |
Return the reserved launch ids that have not been updated for a while.
Parameters: |
|
---|---|
Returns: | list of expired lacunh ids |
Return type: | [int] |
Unmark the offline run for the given launch or firework id.
Parameters: |
---|
Given a Firework id, give back a Firework object.
Parameters: | fw_id (int) – Firework id. |
---|---|
Returns: | Firework object |
Given firework id, return firework dict.
Parameters: | fw_id (int) – firework id |
---|---|
Returns: | dict |
Return all the fw ids that match a query.
Parameters: | |
---|---|
Returns: | list of firework ids matching the query |
Return type: | list |
Given the reservation id, return the list of firework ids.
Parameters: | reservation_id (int) – |
---|---|
Returns: | list of firework ids. |
Return type: | [int] |
Given a Launch id, return details of the Launch.
Parameters: | launch_id (int) – launch id |
---|---|
Returns: | Launch object |
Return the log directory.
AJ: This is needed for job packing due to Proxy objects not being fully featured...
Checkout the next Firework id
Checkout the next Launch id
Given the firework id, return the reservation id
Parameters: | fw_id (id) – firework id |
---|---|
Returns: | list tracker dicts |
Return type: | [dict] |
Given a Firework id, give back the Workflow containing that Firework.
Parameters: | fw_id (int) – |
---|---|
Returns: | A Workflow object |
Given a FireWork id, give back the Workflow containing that FireWork.
Parameters: | fw_id (int) – |
---|---|
Returns: | A Workflow object |
Return one fw id for all workflows that match a query.
Parameters: | |
---|---|
Returns: | list of firework ids |
Return type: | list |
A much faster way to get summary information about a Workflow by querying only for needed information.
Parameters: | |
---|---|
Returns: | information about Workflow. |
Return type: | dict |
Perform launchpad maintenance: detect lost runs and unreserved RESERVE launches.
Parameters: |
|
---|
Mark the launch corresponding to the given id as FIZZLED.
Parameters: | launch_id (int) – launch id |
---|---|
Returns: | updated launch |
Return type: | dict |
Ping that a Launch is still alive: updates the ‘update_on ‘field of the state history of a Launch.
Parameters: |
---|
Update the launch state using the offline data in FW_offline.json file.
Parameters: | |
---|---|
Returns: | firework id if the recovering fails otherwise None |
Given the firework id, re-ignite(set state=WAITING) the defused firework.
Parameters: | fw_id (int) – firework id |
---|
Reignite the workflow containing the given firework id.
Parameters: | fw_id (int) – firework id |
---|
Rerun the firework corresponding to the given id.
Parameters: | |
---|---|
Returns: | list of firework ids that were rerun |
Return type: | [int] |
Rerun a fw at the task level.
Parameters: |
|
---|---|
Returns: | list of rerun firework ids. |
Return type: | [int] |
Checkout the next ready firework and mark the launch reserved.
Parameters: | |
---|---|
Returns: | the checked out firework and the new launch id |
Return type: | (Firework, int) |
Create a new FireWorks database. This will overwrite the existing FireWorks database! To safeguard against accidentally erasing an existing database, a password must be entered.
Parameters: |
|
---|
For the given launch id and firework id, restore the back up data.
Checks to see if the database contains any FireWorks that are ready to run.
Returns: | True if the database contains any FireWorks that are ready to run. |
---|---|
Return type: | bool |
Set priority to the firework with the given id.
Parameters: |
|
---|
Set reservation id to the launch corresponding to the given launch id.
Parameters: |
---|
Note: usernames/passwords are exported as unencrypted Strings!
Database tuneup: build indexes
Update fireworks with a spec. Sometimes you need to modify a firework in progress.
Parameters: |
|
---|
Bases: object
A LazyFirework only has the fw_id, and grabs other data just-in-time. This representation can speed up Workflow loading as only “important” FWs need to be fully loaded.
Bases: exceptions.ValueError
Error raised if the context manager WFLock can’t acquire the lock on the WF within the selected time interval (WFLOCK_EXPIRATION_SECS), if the killing of the lock is disabled (WFLOCK_EXPIRATION_KILL)
Bases: object
Lock a Workflow, i.e. for performing update operations Raises a LockedWorkflowError if the lock couldn’t be acquired withing expire_secs and kill==False. Calling functions are responsible for handling the error in order to avoid database inconsistencies.
The Rocket fetches a workflow step from the FireWorks database and executes it.
Run the rocket (check out a job from the database and execute it)
Run a single rocket in the current directory.
Parameters: | |
---|---|
Returns: | bool |
Keeps running Rockets in m_dir until we reach an error. Automatically creates subdirectories for each Rocket. Usually stops when we run out of FireWorks from the LaunchPad.
Parameters: |
|
---|