Provides core low–level functionality of KDVS.
Provides version for the release of KDVS.
Major number for main changes. Minor number for new features. Release number for bug fixes and minor updates. Status = ‘alpha’/’beta’/None.
Provides specification for actions executed by execution environment.
Bases: object
Creates new action to be submitted to execution environment. Action specification encapsulates arbitrary Python callable and its arguments. In addition, if requested, execution environment can check if certain variables have been present and/or produced before and after the execution.
Parameters : | input_vars : list/tuple
output_vars : list/tuple
func : callable
args : iterable
kwargs : dict
|
---|
Return all the elements of action specification as tuple: (input_vars, output_vars, func, args, kwargs)
Provides mechanisms for configuration of KDVS. Configuration files are normal Python scripts that are evaluated with ‘execfile’. Since KDVS uses many options in many configuration files, this module provides utilities for evaluating and merging configurations.
Return absolute path of default configuration file for KDVS.
Return absolute path of default data directory for KDVS.
Return absolute path of default Gene Ontology (GO) data directory for KDVS.
Return absolute path of default visualization data directory for KDVS.
Merge parent end child config variables to produce final config variables, as follows. If the variable exists only in parent config or child config, write it to the final config. If the variable exists in both parent and child configs, retain the one from child config.
It is used to merge variables obtained from default configuration file and specified user configuration file.
Parameters : | parent_cfg : dict
child_cfg : dict
|
---|---|
Returns : | final_cfg : dict
|
See also
Evaluate default configuration file of KDVS and return its configuration variables.
Returns : | default_cfg_vars : dict
|
---|
See also
Evaluate specified user configuration file of KDVS, performs merge with default configuration, and returns final configuration variables.
Parameters : | cfg_file : string
ignore_default_cfg : bool
|
---|---|
Returns : | user_cfg : dict
|
See also
Provides layer for all DB operations performed by KDVS.
Bases: object
General manager of all DB operations performed by KDVS. It provides:
- automatic handling of meta–database that contains information of all used subordinated databases
- automated opening/closing of multiple subordinated databases
Parameters : | arbitrary_data_root : string/None
provider : DBProvider/None
rootdbid : string/None
|
---|
See also
Obtain connection for subordinated database with requested ID. If database does not exist, create it. If underlying provider accepts it, special ID ‘memdb’ may be used for single in–memory database.
Parameters : | db_id : string
|
---|---|
Returns : | handle : connection (depends on provider)
|
Obtain location for subordinated database with requested ID. If database with requested ID has not been yet created, return None. The meaning of ‘location’ depends on the provider. For instance, in case of SQLite3, single database is stored in single file, and ‘location’ is the absolute path to that file.
Parameters : | db_id : string
|
---|---|
Returns : | location : (depends on provider)
|
Closes requested subordinated database managed by this manager instance. If None is requested, then all databases are closed.
Parameters : | dbname : string/None
|
---|
Provides mechanism for dynamic verifying of dependences in KDVS.
Verify that requested module is importable from within KDVS.
Parameters : | modname : string
|
---|---|
Returns : | module_instance : module
|
Raises : | Error :
|
Provides functionality of encapsulated execution environment for small dependent computational tasks (dubbed actions). Actions are encapsulated Python callables that can be submitted to environment for execution. Actions are executed in submission order. Actions can communicate through shared variables that are kept within the environment instance. In addition, environment can verify if certain variables are present before or after the execution of specific action.
Bases: kdvs.core.util.Configurable
Implements basic execution environment, where set of actions is executed in order, and common set of environment variables is available for any action to store to and retrieve data from. The environment is configurable.
Parameters : | expected_cfg : dict
env_cfg : dict
|
---|---|
Raises : | Error :
|
Returns list of actions submitted for execution by this environment.
Returns : | actions : iterable of ActionSpec
|
---|
Submit instance of action specification (ActionSpec) for execution by this environment.
See also
Submit specified callable for execution by this environment.
Parameters : | action_callable : callable
action_kwargs : dict/None
action_input : list/tuple/None
action_output : list/tuple/None
|
---|
Clear all currently submitted actions.
Add new variable to execution environment, with possible replacement. Variables are managed in a dictionary, therefore all rules for adding items to standard dictionaries apply here as well.
Parameters : | varkey : object
var : object
replace : bool
|
---|---|
Raises : | ValueError :
|
Remove requested variable from execution environment. If variable does not exist, do nothing.
Parameters : | varkey : object
|
---|
Add all variables from requested dictionary to execution environment.
Parameters : | vardict : dict
replace : bool
|
---|---|
Raises : | ValueError :
|
See also
Retrieve value of variable present in execution environment.
Parameters : | varkey : object
|
---|---|
Returns : | var : object
|
Raises : | ValueError :
|
Retrieve keys of all existing variables present in execution environment.
Returns : | varkeys : iterable
|
---|
Execute all submitted actions so far in submission order (FIFO) and performs explicit garbage collection between each action run. When any action raises an exception during its run, the whole execution is stopped and diagnostic information is returned. If all actions finish without exception, also clear them.
Parameters : | verify_io : bool
dump_env : bool
|
---|---|
Returns : | diagnostic : tuple/None
|
See also
Notes
Dumping the environment that executes many long and complicated interlinked actions that use lots of environment variables produces a LOT OF diagnostic information.
Callback function called after each successful action execution. By default it does nothing.
Bases: kdvs.core.env.ExecutionEnvironment
Implements execution environment that utilizes logging.
See also
Parameters : | expected_cfg : dict
env_cfg : dict
|
---|---|
Raises : | Error :
|
Execute actions and log any diagnostic information obtained.
See also
Called after each successful action execution. Flushes pending logger output.
Provides unified error handling.
Bases: exceptions.Exception
Specific KDVS exception.
Bases: exceptions.Warning
Specific KDVS warning.
Provides logging utilities for KDVS.
Bases: object
Abstract logger for KDVS system.
Parameters : | name : string/None
level : integer
handler : Handler/None
formatter : Formatter/None
add_atts : dict/None
|
---|
See also
Return dictionary of standard attributes for the logger.
Return dictionary of associated custom attributes for the logger.
Bases: kdvs.core.log.Logger
Logger that uses rotating file mechanism. By design, the default logger that KDVS uses.
Parameters : | name : string/None
level : integer
path : string/None
maxBytes : integer
backupCount : integer
|
---|
See also
Bases: kdvs.core.log.Logger
Logger that directs messages to specified stream.
Parameters : | name : string/None
level : integer
stream : file
|
---|
See also
Contains set of useful providers for various generic classes of objects handled by KDVS.
Bases: object
Abstract class for providers of database services. All methods must be implemented in subclasses.
See also
Appropriately connect to specified database and return connection object.
Return appropriate low–level connection function.
Return appropriate instance of OperationalError.
Return appropriate type for DB column that contains unformatted text data.
Perform appropriate check if table is present in the database.
Get appropriate information about DB engine, as the following dictionary: {‘name’ : name, ‘version’ : version}.
Bases: kdvs.core.provider.DBProvider
Provider for SQLite3 database based on sqlite3.
Connect to specified database. All positional arguments are passed to sqlite3.connect() function. All keyworded arguments may be used by the user. Currently the following arguments are recognized:
- ‘unicode_strings’ (boolean)
if True, all strings returned by sqlite3 will be Unicode, and normal strings otherwise; sets global attribute sqlite3.text_factory appropriately; False by default (may be omitted)
Parameters : | args : iterable
kwargs : dict
|
---|---|
Raises : | Error :
|
Returns sqlite3.connect() instance.
Returns sqlite3.OperationalError instance.
Returns ‘TEXT’ as the default unformatted table content. See SQLite documentation for more details.
Check if specific table exists in given database. The check is performed as a query of ‘sqlite_master’ table. See SQLite documentation for more details.
Parameters : | conn : sqlite3.Connection
tablename : string
|
---|---|
Raises : | Error :
|
Get version information if the SQLite engine as dictionary {‘name’ : name, ‘version’ : version}. Version information is obtained as the result of SQLite function ‘sqlite_version’ executed from in–memory database. See SQLite documentation for more details.
Raises : | Error :
|
---|
Return opened file object suitable for use with context manager, regardless of file type. Provides transparent handling of compressed files.
Parameters : | filename : string
args : iterable
kwargs : dictionary
|
---|---|
Returns : | file_object : file_like
|
Bases: gzip.GzipFile
Wrapper class to allow opening gzip-ed files with context manager.
Constructor for the GzipFile class.
At least one of fileobj and filename must be given a non-trivial value.
The new class instance is based on fileobj, which can be a regular file, a StringIO object, or any other object which simulates a file. It defaults to None, in which case filename is opened to provide a file object.
When fileobj is not None, the filename argument is only used to be included in the gzip file header, which may includes the original filename of the uncompressed file. It defaults to the filename of fileobj, if discernible; otherwise, it defaults to the empty string, and in this case the original filename is not included in the header.
The mode argument can be any of ‘r’, ‘rb’, ‘a’, ‘ab’, ‘w’, or ‘wb’, depending on whether the file will be read or written. The default is the mode of fileobj if discernible; otherwise, the default is ‘rb’. Be aware that only the ‘rb’, ‘ab’, and ‘wb’ values should be used for cross-platform portability.
The compresslevel argument is an integer from 1 to 9 controlling the level of compression; 1 is fastest and produces the least compression, and 9 is slowest and produces the most compression. The default is 9.
The mtime argument is an optional numeric timestamp to be written to the stream when compressing. All gzip compressed streams are required to contain a timestamp. If omitted or None, the current time is used. This module ignores the timestamp when decompressing; however, some programs, such as gunzip, make use of it. The format of the timestamp is the same as that of the return value of time.time() and of the st_mtime member of the object returned by os.stat().
Bases: bz2.BZ2File
Wrapper class to allow opening bzip2-ed files with context manager.
File providers currently recognized by KDVS.
See also
Provides various utility classes and functions.
Surrounds requested string with given “quote” strings.
Parameters : | s : string
quote : string
|
---|---|
Returns : | quoted_str : string
|
Generator that discards strings prefixed with given comment string.
Parameters : | seq : iterable
comment : string/None
|
---|---|
Returns : | ncomseq : iterable
|
Return True if given object is an instance of list or tuple, and False otherwise.
Return True if given object is an instance of tuple, and False otherwise.
Return True if given object is an instance of integer number, and False otherwise.
Return class name of the object.
Generator that yields nothing.
Empty callable.
Internal function. Revert given slice object.
Bases: object
Wraps DBResult instance into a generator that can be passed to numpy.loadtxt() family of functions. It exposes appropriate methods accepted by numpy.loadtxt() depending on the numpy version: ‘readline’ (for numpy < 1.6.0), ‘next’ (for numpy >= 1.6.0).
Parameters : | dbresult : DBResult
delim : string
id_col_idx : integer/None
|
---|
See also
Produce representation of the object that is as serializable as possible (in the sense of pickle).
Parameters : | obj : object
|
---|---|
Returns : | ser_obj : object
|
Serialize given input object to opened file–like handle. Self–contained function, can be used as depfunc with PPlusJobContainer.
Parameters : | obj : object
out_fh : file-like
protocol : integer/None |
---|
Deserialize object from opened file–like handle. Self–contained function, can be used as depfunc with PPlusJobContainer.
Parameters : | in_fh : file-like
|
---|---|
Returns : | obj : object
|
Serialize given lines of text to opened file–like handle. Self–contained function, can be used as depfunc with PPlusJobContainer.
Parameters : | lines : iterable
out_fh : file-like
|
---|
Notes
Currently it just writes text lines with ‘writelines’. Override for more detailed control.
Pretty prints in sense of pprint.pprint(), the represenation of given input object, to opened file–like handle. Self–contained function, can be used as depfunc with PPlusJobContainer.
Parameters : | obj : object
out_fh : file-like
indent : integer
|
---|
Write given input object to opened file–like handle. Self–contained function, can be used as depfunc with PPlusJobContainer.
Parameters : | obj : object
out_fh : file-like
|
---|
Notes
Currently it just writes object with ‘write’. The exact behavior depends on the given input object and file-like object. The function does not check for any error. Override for more detailed control.
Return True if given directory is writable, and False otherwise.
Dynamically import given module.
Parameters : | qualified_name : string
qualifier : string
|
---|---|
Returns : | module : module
|
Notes
Importing is done relatively to ROOT_IMPORT_PATH only. This function is used for detailed dynamic imports from within KDVS.
Return filename component of given file path.
Internal helper function. Return correct indexes for given DSV instance.
Taken from itertools receipts. Split given iterable of elements into overlapping pairs of elements as follows: (s0,s1,s2,s3,...) -> (s0,s1),(s1,s2),(s2, s3),...
Bases: object
Abstract class that can be parametrized. During instantiation, given parameters can be compared to reference ones, and Error is thrown when they do not match.
Parameters : | ref_parameters : iterable
kwargs : dict
|
---|---|
Raises : | Error :
|
Bases: object
Abstract class that can be configured. During instantiation, given configuration can be compared to reference one, and Error is thrown when they do not match. Expected configuration is a dictionary of objects {‘par1’ : partype1obj, ‘par2’ : partype2obj, ...}, where partypeXobj is the example object of expected type. For instance, if partypeXobj is {}, it will be checked if the parameter value is a dictionary. Object types are checked with isinstance().
Parameters : | expected_cfg : dict
actual_cfg : dict
|
---|---|
Raises : | Error :
Error :
|
Return all keys of actual supplied configuration.
Bases: object
Class that represents string constants.
Parameters : | srepr : object
|
---|
Notes
In current implementation, it represents ‘text’ with ‘<text>’.