Introduction
GC3Pie is a Python package for running large job campaigns on diverse
batch-oriented execution environments (for instance: a Sun/Oracle/Open
Grid Engine cluster, or the Swiss National Distributed Computing
Infrastructure SMSCG). It also provides facilities for implementing
command-line driver scripts, in the form of Python object classes
whose behavior can be customized by overriding specified object
methods.
GC3Pie is comprised of three main components:
- GC3Libs: A library of Python classes for controlling the life-cycle
of a Grid or batch computational job collections. GC3Libs provides
services for submitting computational jobs to Grids and batch
systems and controlling their execution, persisting job information,
and retrieving the final output.
- The GC3Apps software: A collection of driver scripts to run large job
campaigns. These are all “glue scripts” to control hundreds or
thousand job instances at once; they are used in production by
various research groups, but are also publicly available as examples
of GC3Libs coding.
- The GC3Utils software: This is a small set of low-level utilities to
perform various control and cleanup tasks, like querying a job’s
running state (gstat), forcibly terminating it
(gkill), re-submitting a job (gresub), etc.