Usage

We briefly describe the use of cygenja. Basically, you register some filters, file extensions and actions before triggering the translation. In the Examples section, you can see cygenja in action as we detail its use to generate the CySparse library.

The Generator class

The Generator class is the main class of the cygenja library. It also is the only class you need to interact with. Its constructor is really simple:

from cygenja.generator import Generator
...

logger = ...
env = ...
engine = Generator('root_directory', env, logger, True)

You provide a root_directory directory, a Jinja2 environment, a logger (from the standard logging library) and decide if warnings must raise Exceptions or not (default: False). The first two arguments are mandatory while the last two are optional. You don’t have to provide a logging engine. We describe the root directory a little further in The root directory, refer the reader to jinja2.Environment for more about the env argument and discuss the two last arguments in the next corresponding subsections.

Logging

A logging engine can be used but is not mandatory. If you don’t want to log cygenja‘s behavior, simply pass None as the value of the logger argument in the constructor (this is the default). The logging engine is an object from Python’s logging library.

from cygenja.generator import Generator
import logging

logger = logging.getLogger('Logger name')
engine = Generator('root_directory', logger, True)

Three message logging helpers are provided:

def log_info(self, msg)
def log_warning(self, msg)
def log_error(self, msg)

Their names and signatures are quite self-explanatory.

Raising exceptions on Warnings

Errors always trigger RuntimeErrors while warnings may or may not trigger RuntimeErrors. To raise exceptions on warnings, set the raise_exception_on_warning to True in the constructor:

engine = Generator('root_directory', logger=logger, raise_exception_on_warning=True)

By default, raise_exception_on_warning is set to False.

Patterns

There are only two types of patterns:

  • fnmatch patterns for file names and
  • glob patterns for directory names.

This is a general rule for the whole library. When you register an action though, you must provide a directory name, not a directory name pattern.

We encourage the reader to (re)read the specifications of these two libraries.

The root directory

The root directory is really the main working directory: all file generations can only be done inside subdirectories of this directory.

This is so important that it is worth a warning:

Warning

File generations can only be done inside subdirectories of the root directory.

This directory is given as the first parameter of Generator‘s constructor and can be absolute or relative. At any moment, you can retrieve this directory as an absolute path:

engine = Generator('root_directory', ...)

absolute_root_directory = engine.root_directory()

Filters

Filters are simply Jinja2 filters. These filters are registered:

def my_jinja2_filter(filter_argument):
    ...
    return filter_result

engine = Generator(...)
engine.register_filter('my_filter_name', my_jinja2_filter)

where 'my_filter_name' is the name of the filter used inside your Jinja2 template files and my_jinja2_filter is a reference to the actual filter.

The signature of register_filter is:

register_filter(self, filter_name, filter_ref, force=False)

allowing you to register a new filter under an already existing filter name. If you keep force set to False, a warning is triggered each time you try to register a new filter under an already existing filter name and this new filter is disregarded.

You also can register several filters at once with a dictionary of filters:

engine = Generator(...)
filters = { 'f1' : filter1,
            'f2' : filter2}

engine.register_filters(filters, force=False)

At any time, you can list the registered filters:

engine = Generator(...)
print engine.filters_list()

This list also includes predefined Jinja2 filters (see builtin filter). If you only want the filters you registered, invoke:

engine.registered_filters_list()

File extensions

cygenja uses a correspondance table between template files and generated files. This table defines a correspondance between file extensions. For instance, to have *.cpd templates generate *.pxd files:

engine = Generator(...)
engine.register_extension(`.cpd`, `.pxd`)

Again, we use a force switch to force the redefinition of such a correspondance. By default, this switch is set to False and if you try to redefine an association with a given template extension, you will trigger a warning and this new correspondance will be disregarded.

You can use a dict to register several extensions at once:

engine = Generator(...)
ext_correspondance = { '.cpd' : '.pxd',
                       '.cpx' : 'pyx'}
engine.register_extensions(ext_correspondance, force=False):

As with filters, you can retrieve the registered extensions:

engine.registered_extensions_list()

Files with extensions registered as template file extensions are systematically parsed, i.e. you cannot use these extensions for files that are not templates because cygenja will try to parse them. What about generated file extensions? Files with these extensions can peacefully coexist with generated files, i.e. existing files, regardless of their extensions, can coexist with generated files and will not be plagued by cyjenja. This means that you can safely delete files: only generated files will be deleted [1].

Note

Only generated files are deleted. You can thus safely delete files with cygenja.

Actions

Actions (defined in the GeneratorAction class) are really the core concept of cygenja: an action correspond to a translation rule. This translation rule makes a correspondance between a subdirectory and a file pattern and a user callback. Here is the signature of the register_action method:

def register_action(self, relative_directory, file_pattern, action_function)

The relative_directory argument holds the name of a relative directory from the root directory. The separator is OS dependent. For instance, under linux, you can register the following:

engine = Generator(...)

def action_function(...):
    ...
    return ...

engine.register_action('cysparse/sparse/utils', 'find*.cpy', action_function)

This means that all files corresponding to the 'find*.cpy' fnmatch pattern inside the cysparse/sparse/utils directory can be dealt with the action_function.

Contrary to filters and file extensions, you cannot ask for a list of registered actions. But you can ask cygenja to perform a dry session: cygenja outputs what it would normaly do but without taking any action [2].

User callback

The action_function() is a user-defined callback without argument. It returns a file suffix with a corresponding Jinja2 variables dict (this is a simple Python dict). Let’s illustrate this by an example:

GENERAL_CONTEXT = {...}
INDEX_TYPES = ['INT32', 'INT64']
ELEMENT_TYPES = ['FLOAT32', 'FLOAT64']

def generate_following_index_and_type():
    """

    """
    for index in INDEX_TYPES:
        GENERAL_CONTEXT['index'] = index
        for type in ELEMENT_TYPES:
            GENERAL_CONTEXT['type'] = type
            yield '_%s_%s' % (index, type), GENERAL_CONTEXT

The user-defined callback generate_following_index_and_type() doesn’t take any input argument and returns the '_%s_%s' suffix string together with the variables dict GENERAL_CONTEXT. This function allows cygenja to create files with this suffix from any matching template file. The GENERAL_CONTEXT is given to Jinja2 for the appropriate translation.

For instance, let’s use the ext_correspondance extensions dict discussed earlier (see File extensions):

ext_correspondance = { '.cpd' : '.pxd',
                       '.cpx' : 'pyx'}

Any template file with a .cpd or .cpx extension will be translated into a _index_type.pxd or _index_type.pyx file respectively. For instance, the template file my_template_code_file.cpd will be translated to:

  • my_template_code_file_INT32_FLOAT32.cpd
  • my_template_code_file_INT32_FLOAT64.cpd
  • my_template_code_file_INT64_FLOAT32.cpd
  • my_template_code_file_INT64_FLOAT64.cpd

As this function is defined by the user, you have total control on what you want to generate or not. In our example, we redefine GENERAL_CONTEXT['index'] and GENERAL_CONTEXT['type'] for each index and element types.

We use generators (yield) but you could return a list if you prefer.

Incompatible actions

You could register incompatible actions, i.e. register competing actions that would translate a file in different ways. Our approach is to only use the first compatible action and to disregard all the other actions, regardless if they could be applied or not. So the order in which you register your actions is important. A file will be dealt with the first compatible action found. This is worth a warning:

Warning

A template is translated with the first compatible action found and only that action.

Default action

cygenja allows to define one default action that will be triggered when no other compatible action is found for a given template file that corresponds to a fnmatch pattern:

engine = Generator(...)

def default_action():
    return ...

engine.register_default_action('*.*',  default_action)

Be careful when defining a default action. This action is applied to all template files (corresponding to the fnmatch pattern) for which no compatible action is found. You might want to prefer declaring explicit actions than relying on this implicit default action. That said, if you have lots of default cases, this default action can be very convenient and avoid lots of unnecessary action declarations.

File generation

To generate the files from template files, there is only one method to invoke: generate(). Its signature is:

def generate(self, dir_pattern, file_pattern, action_ch='g',
             recursively=False, force=False)

dir_pattern is a glob pattern taken from the root directory and it is only used for directories while file_pattern is a fnmatch pattern taken from all matching directories and is only used for files. The action_ch is a character that triggers different behaviours:

  • g: Generate all files that match both directory and file patterns. This is the default behavior.
  • d: Same as g but with doing anything, i.e. dry run.
  • c: Same as g but erasing the generated files instead, i.e. clean.

These actions can be done in a given directory or in all its corresponding subdirectories. To choose between these two options, use the recursively switch. Finally, by default, files are only generated if they are outdated, i.e. if they are older than the template they were originated from. You can force the generation with the force switch.

Footnotes

[1]The user is responsible to not to define a translation rule that overwrites any existing files.
[2]

You also have access to the internal TreeMap object:

engine = Generator(...)

treemap = engine.registered_actions_treemap()

and thus you have access to all its methods. One interesting method is to_string(). It gives you a representation of all involved subdirectories.