The htsql_ctl package

htsql_ctl

This package implements the htsql-ctl script.

class htsql_ctl.HTSQL_CTL(stdin, stdout, stderr)

Implements the htsql-ctl script.

Usage:

ctl = HTSQL_CTL(stdin, stdout, stderr)
exit_code = ctl.main(argv)

htsql_ctl.default

This module implements the default routine.

class htsql_ctl.default.DefaultRoutine(ctl, attributes)

Implements the default routine.

The default routine is executed when no explicit routine name is given.

htsql_ctl.error

This module implements exceptions raised by command-line scripts.

exception htsql_ctl.error.ScriptError(detail)

A fatal application error.

detail
The error message.

htsql_ctl.extension

This module implements the extension routine.

htsql_ctl.help

This module implements the help routine.

class htsql_ctl.help.HelpRoutine(ctl, attributes)

Implements the help routine.

When called without any parameters, it describes the application. When called with a routine name, it describes the routine.

htsql_ctl.option

This module defines script options.

class htsql_ctl.option.Option(attribute, short_name=None, long_name=None, with_value=False, value_name=None, validator=None, default=None, is_list=False, hint=None)

Describes a script option.

attribute (a string)
The name of the routine attribute. When the routine is initialized, the value of the option is assigned to the attribute.
short_name (a string or None)
The abbrebiated form of the option (a dash + a character).
long_name (a string or None)
The full form of the option (two dashes + the option name).
with_value (Boolean)
If set, the option requires a parameter.
value_name (a string or None)
The name of the option parameter.
validator (htsql.validator.Validator or None)
The validator for the option parameter.
default
The default value of the parameter; None if not set.
is_list (Boolean)
If set, the option could be specified more than one time. The combined values are stored as a list; the default value for a list option is [].
hint (a string or None)
A one-line description of the option.

When an option does not require a parameter, the value of the option is either False or True. If the option is provided in the command line, the value is True; otherwise, the value is False.

When an option requires a parameter, the value of the option is determined by the parameter and the attributes validator and default. If the option is not provided, its value is equal to default. If the option is provided, its value is the value of the option parameter normalized by application of validator.

get_hint()

Returns a short one-line description of the option.

get_signature()

Returns the signature of the option parameters.

htsql_ctl.regress

This module implements the regress routine.

class htsql_ctl.regress.TermStringIO(buf, output)

A readable file-like object with an “echo”. Whenever some content is read from it, the same content is echoed to the specified output stream.

Use TermStringIO to preserve the content of interactive sessions with pre-recorded input. Assign:

sys.stdout = StringIO.StringIO()
sys.stdin = TermStringIO(input, sys.stdout)

where input contains the pre-recorded input data. After the session is done, the content of sys.stdout will be the same as if the session was performed on a real terminal with echo enabled.

buf (a string)
The content of the stream.
output (a writable file-like object)
A stream that records data being read.
class htsql_ctl.regress.Field(attribute, val, default=<object object at 0x8b33b80>, hint=None)

Describes a parameter of test data.

attribute (a string)
The name of the attribute that contains the field value.
val (htsql.validator.Validator)
The validator for the field values.
default
The default value of the field. If not provided, the field cannot be omitted. The is_mandatory attribute indicates if the default value is provided.
hint (a string or None)
A short one-line description of the field.
get_hint()

Returns short one-line description of the field.

get_signature()

Returns the field name.

class htsql_ctl.regress.AnyField

Indicates that test data may contain extra fields.

Add AnyField() to the fields list to indicate that YAML representation of test data may contain some attributes not described by other fields. These extra attributes will be silently ignored.

class htsql_ctl.regress.TestData(routine, case_class, attributes, location=None)

Represents input or output data of a test case.

This is an abstract class. Create a subclass of TestData to describe input or output data for a specific test kind. You need to specify the format of test data using the fields class attribute.

The fields attribute is a list of Field instances. Each field describes an attribute of test data.

Instances if TestData are YAML-serializable. A instance of a TestData subclass is represented as a mapping YAML node. The sets of keys and the format of the values come from the fields list. Add an AnyField instance to fields to indicate that the mapping node may contain some extra fields (which are to be ignored).

The constructor of TestData accepts the following arguments:

routine (RegressRoutine)
The routine that started the testing.
case_class (a subclass of TestCase)
A test type. The object being constructed is an instance of either case_class.Input or case_class.Output.
attributes (a dictionary)
A dictionary of attributes and their values. The set of attributes is declared using the fields class variable.
location (a string or None)
When the test data is loaded from a YAML file, location indicates the location of the corresponding YAML node.
init_attributes()

Normalize field values.

class htsql_ctl.regress.TestCase(routine, state, input, output)

Describes a test type.

This an abstract class. Create a subclass of TestCase to describe a new type of test case. When subclassing, define the following class attributes:

name (a string)
The name of the test.
hint (a string)
Short one-line description of the test.
help (a string)
Long description of the test.
Input (a subclass of TestData)
The format of the test input.
Output (a subclass of TestData or None)
The format of the test output.

You also need to override methods verify() and train() to specify how to execute the test case in a normal and in a train mode.

The constructor of TestCase takes the following arguments:

routine (RegressRoutine)
The routine that started the testing.
state
An object keeping the mutable testing state.
input (an instance of Input)
Input test data.
output (an instance of Output or None)
Expected output test data.
classmethod get_hint()

Returns short one-line description of the test case.

classmethod get_help()

Returns long description of the test case.

classmethod matches(input, output)

Checks if the given input and output records belong to the same test case.

Note that we assume that both test input and test output have a field with the same attribute name. This attribute is called the key attribute. Input data matches output data when the values of their key attribute are equal.

get_suites()

For container test cases, returns a set of test suites that belong to the test case; otherwise returns an empty set.

out(*values, **options)

Print values to the standard output stream.

out() supports the same options as htsql.ctl.script.Script.out() and an extra option:

indent
A number of spaces to print before the first value, default is 0.
ask(message, choices)

Asks the user a question; returns the reply.

message (a string)
The question.
choices (a list of strings)
The list of valid replies.

Typically the question has the form:

Press ENTER to perform <the default action>,
      'x'+ENTER to perform <another action>,
      'y'+ENTER to perform <another action>,
      'z'+ENTER to perform <another action>.

In this case, choices should be equal to:

['', 'x', 'y', 'z']

The reply is stripped of leading and trailing whitespaces and translated to the lower case.

ask_halt()

Ask if the user wants to halt the tests.

Returns DO_HALT or DO_CONTINUE.

ask_record()

Ask if the user wants to remember the new output of a test case.

Returns DO_RECORD, DO_SKIP, or DO_HALT.

ask_save()

Ask if the user wants to save the updated output data.

Returns DO_SAVE or DO_DISCARD.

out_exception(exc_info)

Prints an exception traceback.

out_sep(sep='-', length=72)

Prints a separator: a long line of dashes.

out_header()

Prints a nice header describing the test case.

halted(message=None)

Indicate that the test case failed and stop the tests.

failed(message=None)

Indicate that the test case failed; stop the tests unless --force or --train flags are set.

updated(message=None)

Indicate that the output of the test case has been updated.

passed(message=None)

Indicate that the test case passed.

verify()

Executes the test case.

This method runs the test case with the given input data. If the test completed without errors, compare the produced output with the given expected output.

The test case fails if

  • the test failed to complete without errors;
  • or the expected test output is not provided;
  • or the expected test output is not equal to the actual test output.

Some test cases may not generate output; in this case the test passes if it is completed without errors.

train()

Executes the test case in the training mode; returns the output data.

In the train mode, when the expected test output is not equal to the actual test output, the user is given a choice to update the expected test output.

Note that when the output has not been changed or the user refused to update it, the method must return the original output data, self.output.

class htsql_ctl.regress.SkipTestCase(routine, state, input, output)

Implements a skippable test case.

This is an abstract mixin class; subclasses should call skipped() to check if the test case is enabled or not.

skipped()

Checks if the test is disabled.

class htsql_ctl.regress.DefineTestCase(routine, state, input, output)

Activates a named toggle.

class htsql_ctl.regress.RunAndCompareTestCase(routine, state, input, output)

Implements common methods for a broad category of test cases.

This class implements common scenario: run the test, get the output and compare it with the expected output.

This is an abstract class; create a subclass to implement a concrete test case. The following methods has to be overridden: execute(), render() and differs().

out_lines(lines, indent=0)

Prints the lines with the specified identation.

out_diff(old_output, new_output)

Prints the delta between two test outputs.

render(output)

Converts the output data to a list of lines.

execute()

Runs the test case; returns the produced output.

Returns None if an error occured when running the test case.

differs(old_output, new_output)

Checks if the actual test output differs from the expected test output.

class htsql_ctl.regress.AppTestCase(routine, state, input, output)

Configures the HTSQL application.

class htsql_ctl.regress.LoadAppTestCase(routine, state, input, output)

Loads an existing configuration of an HTSQL application.

class htsql_ctl.regress.IncludeTestCase(routine, state, input, output)

Loads input test data from a file.

class htsql_ctl.regress.SuiteTestCase(routine, state, input, output)

Implements a container of test cases.

class htsql_ctl.regress.QueryTestCase(routine, state, input, output)

Performs an HTSQL query.

class htsql_ctl.regress.CtlTestCase(routine, state, input, output)

Executes a script routine.

class htsql_ctl.regress.Fork(process, temp_path)

Keeps information on the started processes.

Class attributes:

active_forks
The global list of active processes.
is_atexit_registered
Indicates whether an atexit.atexit() callable was registered. The callable is called when the script is about to finish and kills any remaining active processes.

Attributes:

process (an instance of subprocess.Popen)
The wrapped process.
temp_path (a string)
A directory containing two files: input and output, which keeps the content of the standard input and the standard output respectively.
classmethod start(executable, arguments, input)

Starts a new process.

executable
The path to the executable.
arguments
The list of arguments (not including the executable).
input
The content of the standard input.

Returns a new Fork instance.

end()

Ends the process.

Returns the content of the standard output.

class htsql_ctl.regress.StartCtlTestCase(routine, state, input, output)

Starts a long-running routine.

class htsql_ctl.regress.EndCtlTestCase(routine, state, input, output)

Terminates a long-running routine.

class htsql_ctl.regress.PythonCodeTestCase(routine, state, input, output)

Executes arbitrary Python code.

class htsql_ctl.regress.PythonCodeIncludeTestCase(routine, state, input, output)

Executes arbitrary Python code loaded from a file.

class htsql_ctl.regress.SQLTestCase(routine, state, input, output)

Executes a SQL query.

load()

Returns the SQL data to execute.

class htsql_ctl.regress.SQLIncludeTestCase(routine, state, input, output)

Loads SQL queries from a file and executes them.

class htsql_ctl.regress.WriteToFileTestCase(routine, state, input, output)

Writes some data to a file.

class htsql_ctl.regress.ReadFromFileTestCase(routine, state, input, output)

Reads the file content.

class htsql_ctl.regress.RemoveFilesTestCase(routine, state, input, output)

Removes the specified files.

class htsql_ctl.regress.MakeDirTestCase(routine, state, input, output)

Creates a directory.

class htsql_ctl.regress.RemoveDirTestCase(routine, state, input, output)

Removes a directory.

class htsql_ctl.regress.TestState(app=None, forks=None, toggles=None, saves=None, with_all_suites=False, passed=0, failed=0, updated=0, is_exiting=False)

Keeps the mutable state of the testing process.

app
The current HTSQL application.
forks
A mapping from command-line parameters to Fork instances; contains long-running applications.
toggles
A set of active named toggles.
saves
A mapping of named application configurations.
with_all_suites
Indicates that the current suite or one of its ancestors was explicitly selected by the user.
passed
The current number of passed tests.
failed
The current number of failed tests.
updated
The current number of updated tests.
is_exiting
Indicates whether the user asked to halt the testing.
push(other)

Push the state data to a derived state.

other (TestState)
A derived state, the state created by a suite for the suite test cases.
pull(other)

Pull the state from a derived state.

Note that only statistical information is pulled from the derived state.

other (TestState)
A derived state, the state created by a suite for the suite test cases.
class htsql_ctl.regress.RegressYAMLLoader(routine, with_input, with_output, stream)

Loads test data from a YAML file.

routine (RegressRoutine)
The testing engine.
with_input (Boolean)
Indicates that the YAML file contains input records.
with_output (Boolean)
Indicates that the YAML file contains output records.
stream (a file or a file-like object)
The YAML stream.
load()

Loads test data from the YAML stream.

class htsql_ctl.regress.RegressYAMLDumper(routine, with_input, with_output, stream, **keywords)

Dumps test data to a YAML file.

routine (RegressRoutine)
The testing engine.
with_input (Boolean)
Indicates that the YAML file will contain input records.
with_output (Boolean)
Indicates that the YAML file will contain output records.
stream (a file or a file-like object)
The stream where the YAML document is written.
dump(data)

Dumps the data to the YAML stream.

class htsql_ctl.regress.RegressRoutine(ctl, attributes)

Implements the regress routine.

state_class

alias of TestState

classmethod get_help(**substitutes)

Returns a long description of the routine.

classmethod get_feature(name)

Finds the test case by name.

htsql_ctl.request

This module implements the get and post routines.

class htsql_ctl.request.Request(environ)

Represents a WSGI request.

environ
A WSGI environ dictionary.
classmethod prepare(method, query, remote_user=None, content_type=None, content_body=None, extra_headers=None)

Produces a Request object from the given parameters.

method ('GET' or 'POST')
The HTTP request method.
query (a string)
The path and the query parts of the URI.
remote_user (a string or None)
The name of the authenticated user.
content_type (a string or None)
The content type of the POST data, used only when method is 'POST'. If not provided, guessed from the file name of the content_body stream. If that fails, 'application/octet-stream' is used.
content_body (a string, a file or a file-like object or None)
The body of the HTTP request, used only when method is 'POST'.
extra_headers (a dictionary or None)
A dictionary of HTTP headers.
execute(app)

Executes the request against the given WSGI application.

app
A WSGI application.

Returns a Response object.

class htsql_ctl.request.Response

Represents a response to a WSGI request.

status (a string)
The HTTP status line.
headers (a list of pairs)
The HTTP headers.
body (a string)
The HTTP body.
exc_info (a tuple (type, value, traceback) or None)
Any exception occured when the request was executed.
set(**attributes)

Updates the response parameters.

complete()

Returns True if the response is complete; False otherwise.

The response is considered valid if the HTTP status, headers and body are set and valid and no exception occured during the execution of the request.

dump(stream, with_headers=False)

Writes the response to the output stream.

stream (a file or a file-like object)
The stream where to write the response.
with_headers
Indicates whether the status line and the headers should also be written.
class htsql_ctl.request.GetPostBaseRoutine(ctl, attributes)

Implements the common methods for the get and post routines.

Both routines take a connection URI and an HTSQL query as arguments and execute an HTTP request.

class htsql_ctl.request.GetRoutine(ctl, attributes)

Implements the get routine.

The routine executes an HTSQL query over the specified database.

class htsql_ctl.request.PostRoutine(ctl, attributes)

Implements the post routine.

The routine executes an HTSQL query with POST data over the specified database.

htsql_ctl.routine

This module defines basic classes for implementing script routines.

class htsql_ctl.routine.Argument(attribute, validator, default=<object object at 0x8b33770>, is_list=False, hint=None)

Describes an argument of a script routine.

attribute (a string)
The name of the routine attribute. When the routine is initialized, the value of the argument is assigned to the attribute.
validator (htsql.validator.Validator)
The validator for the argument value.
default
The default value of the argument. If default is not provided, the argument value is always required. The is_mandatory attribute indicates if the default value is omitted.
is_list (Boolean)
If set, the argument may accept more than one parameter. In this case, the argument value is a list of parameters.
hint (a string or None)
A short one-line description of the argument.
get_hint()

Returns a short one-line description of the argument.

get_signature()

Returns the argument signature.

class htsql_ctl.routine.Routine(ctl, attributes)

Describes a script routine.

Routine is a base abstract class for implementing a script routine. To create a concrete routine, subclass Routine, declare the routine name, arguments and options, and override run().

The following class attributes should be overridden.

name (a string)
The name of the routine. The name must be unique across all routines of the script. If equal to '', the routine is called when the script is executed without any parameters.
aliases (a list of strings)
The list of alternative routine names.
arguments (a list of Argument)
The list of routine arguments.
options (a list of htsql.ctl.option.Option)
The list of routine options.
hint (a string or None)
A one-line description of the routine.
help (a string or None)
A long description of the routine. Keep the line width at 72 characters.

The constructor of Routine accepts the following arguments:

ctl
The script; an instance of htsql.ctl.script.Script or of its subclass.
attributes
A dictionary mapping attribute names to attribute values. The attribute names correspond to the routine arguments and options; the values are obtained by parsing the command-line parameters.
classmethod get_hint()

Returns a short one-line description of the routine.

classmethod get_help(**substitutes)

Returns a long description of the routine.

classmethod get_signature()

Returns the routine signature.

classmethod get_feature(name)

Finds some routine feature by name.

run()

Executes the routine. Returns an exit code.

htsql_ctl.script

This module implements a command-line application with subcommands.

class htsql_ctl.script.Script(stdin, stdout, stderr)

Implements a command-line application with the following interface:

ctl <routine> <options> <arguments>
ctl
The name of the script.
<routine>
The name of the subcommand.
<options>

A list of routine options. The set of supported options is routine-specific.

Each option may be written using either an abbreviated form (a dash + a character) or a full form (two dashes + the option name). For instance, the quiet option may be written as -q or as --quiet.

Some options expect a parameter, which should then follow the option name. For instance, the input option expects a file name parameter, which could be passed using one of the following forms:

  • -iFILE
  • -i FILE
  • --input=FILE
  • --input FILE

Two or more options written using an abbreviated form could share the leading dash. For instance, -q -iFILE could also be written as -qiFILE. When this form is used, only the last option in the list could accept a parameter.

Note that the order and the position of options in the command line is not fixed. In particular, an option could preceed the routine name or follow a routine argument. Use special parameter -- to indicate that there are no more options in the list of the remaining parameters. This is useful when a routine argument starts with a dash.

<arguments>
A list of parameters of the routine. The number and the form of the arguments are routine-specific.

Script is the base abstract class implementing a command-line application with subcommands. To create a concrete application, do the following:

  1. Create a subclass of Script overriding the class attributes routines, hint, help, and copyright.
  2. The attribute routines should be a list of htsql.ctl.routine.Routine subclasses. Each subclass implements an application routine.
  3. The attributes hint, help, and copyright should provide a short one-line description of the application, a longer description or the application, and a copyright notice respectively. They are displayed when the application is called without any parameters.

The constructor of Script accepts the following arguments:

stdin
A file or a file-like object representing the standard input stream.
stdout
A file or a file-like object representing the standard output stream.
stderr
A file or a file-like object representing the standard error stream.
main(argv)

Execute the application.

argv (a list of strings)
The first element of the list is the path to the application. The remaining elements of the list are the application parameters.

Returns an exit status suitable for passing to sys.exit(). The exit status could be None, an integer value, or an exception object. Note that when a non-integer value is passed to sys.exit(), the value is printed and the system exit code is set to one.

out(*values, **options)

Print the values to the standard output stream.

Supported options are:

sep
A string to print between values, default is ' '.
end
A string to print after the last value, default is '\n'.
file
A file or a file-like object, default is stdout.
err(*values, **options)

Print the values to the standard error stream.

Supported options are:

sep
A string to print between values, default is ' '.
end
A string to print after the last value, default is '\n'.
file
A file or a file-like object, default is stderr.
get_hint()

Returns a short one-line description of the application.

get_help(**substitutes)

Returns a long description of the application.

Returns a copyright notice.

htsql_ctl.server

This module implements the server routine.

class htsql_ctl.server.ServerRoutine(ctl, attributes)

Implements the server routine.

The routine starts an HTTP server that serves HTSQL requests over the specified database.

htsql_ctl.shell

This module implements the shell routine.

class htsql_ctl.shell.Cmd(routine, argument)

Describes a shell command.

This is the base abtract class for all shell commands. To create a concrete command, subclass Cmd, declare the command name and other parameters, and override execute().

The following class attributes could be overridden.

name (a string)
The name of the command. Name equal to an empty string ('') means this command is executed when no explicit command name is provided. The name must be unique across all commands.
aliases (a list of strings)
The list of alternative command names.
signature (a string or None)
Declaration of the command name and the arguments.
hint (a string or None)
A short one-line description of the command.
help (a string or None)
A long description of the command. Keep the line width at 72 characters.

Instances of Cmd have the following attributes.

routine (ShellRoutine)
The routine that executed the command.
ctl (htsql.ctl.script.Script)
The script that started the shell routine.
state (ShellState)
The shell state.
argument (a string)
The argument of the command.
classmethod get_signature()

Returns an (informal) signature of the command.

classmethod get_hint()

Returns a short one-line description of the command.

classmethod get_help()

Returns a long description of the command.

execute()

Executes the command.

The normal return value is None; any other value causes the shell to exit.

class htsql_ctl.shell.HelpCmd(routine, argument)

Implements the help command.

class htsql_ctl.shell.ExitCmd(routine, argument)

Implements the exit command.

class htsql_ctl.shell.UserCmd(routine, argument)

Implements the user command.

class htsql_ctl.shell.HeadersCmd(routine, argument)

Implements the headers command.

class htsql_ctl.shell.PagerCmd(routine, argument)

Implements the pager command.

class htsql_ctl.shell.GetPostBaseCmd(routine, argument)

Implements the common methods of get and post commands.

class htsql_ctl.shell.GetCmd(routine, argument)

Implements the get command.

class htsql_ctl.shell.PostCmd(routine, argument)

Implements the post command.

class htsql_ctl.shell.RunCmd(routine, argument)

Implements the run command.

class htsql_ctl.shell.ShellState(app=None, with_headers=False, remote_user=None, with_pager=True, completer=None, completer_delims=None, completions=())

Holds mutable shell parameters.

app
The current HTSQL application.
with_headers (Boolean)
Indicates whether to display the status line and the headers of an HTTP response.
remote_user (a string or None)
The WSGI remote user.
with_pager (Boolean)
Indicates whether the pager is enabled.
class htsql_ctl.shell.ShellRoutine(ctl, attributes)

Implements the shell routine.

classmethod get_intro()

Returns a notice to display when the shell is started.

classmethod get_usage()

Returns the shell description used by the help command.

classmethod get_help(**substitutes)

Returns a long description of the routine.

classmethod get_feature(name)

Returns the shell command by name.

htsql_ctl.version

This module implements the version routine.

class htsql_ctl.version.VersionRoutine(ctl, attributes)

Implements the version routine.

The routine displays the version of the htsql package.