FireWorks comes with several built-in tasks for performing file I/O (writing, moving, and deleting files).
The FileWriteTask is a simple way to write files (a more powerful version based on templating is the TemplateWriterTask).
The FileDeleteTask lets you delete one or more files.
The FileTransferTask is a built-in FireTask for moving and copying files, perhaps to a remote server. Its usage is straightforward:
An example of a Firework that copies two files, file1.txt and file2.txt, to the home directory looks as follows:
spec:
_tasks:
- _fw_name: FileTransferTask
files:
- src: file1.txt
dest: ~/file1.txt
- src: file2.txt
dest: ~/file2.txt
mode: copy
In Python code, the same task would be defined as:
firetask1 = FileTransferTask({'files': [{'src': 'file1.txt', 'dest': '~/file1.txt'}, {'src': 'file2.txt', 'dest': '~/file2.txt'}], 'mode': 'copy'})
The files parameter must be specified, and is an array of dictionaries with src and dest keys. The default mode of operation is to move files from the source to destination; by changing the mode to copy, we will copy the files instead. Note that you can put as many files (or directories) in the files list as you want; the same mode will be applied to all of them. If you want to move some files and copy others, you’d need to include two different ``FileTransferTask``s in your Firework.
A more compact representation for File Transfers can be given if several files are being moved/copied to the same directory:
spec:
_tasks:
- _fw_name: FileTransferTask
dest: dest_dir
files:
- file1.txt
- file2.txt
mode: copy
In Python code, this representation would be defined as:
firetask1 = FileTransferTask({'files': ['file1.txt', 'file2.txt'], 'dest':'dest_dir', 'mode': 'copy'})
An example of the FileTransferTask in action is given in the FireTask tutorial.
Below are the various options that can be set for the FileTransferTask.
mode
The potential values are:
Note
With the exception of rtransfer, FileTransferTask is using Python’s shutil module to do the move/copy. You can read more about these modes in `Python’s *shutil* docs <http://docs.python.org/2/library/shutil.html`_.
ignore_errors
Either True or False (default=*False*). If True, a failed move/copy will just cause FileTransferTask to move to the next file in the files list. If False, a failed move/copy will raise an error.
shell_interpret
Either True or False (default=*True*). If False, the src and dest of files are taken literally. If True, the src and dest interpret environment variables and shortcuts like ~ and .
Note
In remote transfer mode, some shortcuts like . and ~ are not interpreted for the destination. However, environment variables will still be interpreted if shell_interpret is True.
Remote transfers are handled via SFTP using the paramiko Python library (make sure you’ve installed it via pip install paramiko). To do a remote transfer, you set files as before but:
If all this is configured properly, you should be able to transfer files to a remote machine via FileTransferTask. Some potential hiccups:
By default, the parameters for the FileTransferTask should be defined within the _task section of the spec corresponding to the FileTransferTask, not as a root key of the spec. If you’d like to instead specify the parameters in the root of the spec, you can set _use_global_spec to True within the _task section. Note that _use_global_spec can simplify querying and communication of parameters between FireWorks but can cause problems if you have multiple FileTransferTasks within the same Firework.
The CompressDir task allows you to compress each file in the current directory (e.g., via gzip).
(none)
The ArchiveDir task allows you to archive the current working directory into a single file.