Metadata-Version: 2.4
Name: scrippy-core
Version: 3.0.0
Summary: "Module principal du cadriciel Scrippy"
Home-page: https://codeberg.org/scrippy/scrippy-core
Author: Michael Costa, Florent Chevalier
Author-email: michael.costa@mcos.nc, florent.chevalier.nc@gmail.com
License: MIT
Project-URL: Bug Tracker, https://codeberg.org/scrippy/scrippy-core/issues
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Environment :: Console
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: coloredlogs>=10.0
Requires-Dist: prettytable>=0.7.2
Requires-Dist: filelock>=3.0.12
Requires-Dist: PyYAML>=6.0.3
Requires-Dist: rich>=10.16.2
Provides-Extra: dev
Requires-Dist: twine; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: wheel>=0.37.0; extra == "dev"
Requires-Dist: flake8>=4.0.1; extra == "dev"
Requires-Dist: flake8-breakpoint>=1.1.0; extra == "dev"
Requires-Dist: flake8-builtins>=1.5.3; extra == "dev"
Requires-Dist: flake8-print>=4.0.0; extra == "dev"
Requires-Dist: flake8-return>=1.1.3; extra == "dev"
Requires-Dist: pep8-naming>=0.8.2; extra == "dev"
Requires-Dist: pytest>=6.2.2; extra == "dev"
Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
Requires-Dist: bump2version>=1.0.1; extra == "dev"
Requires-Dist: setuptools>=60.9.2; extra == "dev"
Requires-Dist: pypirc-chappers>=1.0.8; extra == "dev"
Requires-Dist: pylint>=2.12.2; extra == "dev"
Dynamic: license-file

![Build Status](https://drone-ext.mcos.nc/api/badges/scrippy/scrippy-core/status.svg) ![License](https://img.shields.io/static/v1?label=license&color=orange&message=MIT) ![Language](https://img.shields.io/static/v1?label=language&color=informational&message=Python)


![Scrippy, my scrangourou friend](./scrippy-core.png "Scrippy, my scrangourou friend")

# `scrippy_core`

`scrippy_core` is the main _Python3_ module of the [`Scrippy`](https://codeberg.org/scrippy) framework for the normalization of _Python_ scripts writing.

This module provides all the basic functionalities such as configuration file management, log and execution history management, script access control, concurrent execution management, etc.

## Prerequisites

### System

#### Debian and derivatives

- python3
- python3-pip
- python-dev
- build-essential

### Python modules

#### List of required modules

The modules listed below will be installed automatically.

- prettytable
- coloredlogs
- filelock
- PyYAML
- rich

_Python_ 3.8 or later is required.

## Installation


### Complete **Scrippy** framework installation and configuration

The **Scrippy** framework and all its packages can be installed with the [**Scrippy** installer and configuration helper](https://codeberg.org/scrippy/scrippy).

```shell
pip install scrippy
scrippy install
```
The commands listed above will install the **Scrippy** scripting framework with default values in the user home directory.

### With `pip`

```bash
sudo pip install scrippy-core
```

### Manual installation from source

```bash
git clone https://codeberg.org/scrippy/scrippy-core.git
cd scrippy-core
python -m pip install -r requirements.txt
make install
```

### Environment configuration details

At the start of each script, _Scrippy_ looks for its global configuration file at the following locations in the following order:

1. `/etc/scrippy/scrippy.yml`, allows package mainteners to provide a default _Scrippy_ configuration (i.e: _vendor config_)
2. `~/.config/scrippy/scrippy.yml`, allows users to configure _Scrippy_ without requiring administrator permissions
3. `/usr/local/etc/scrippy/scrippy.yml`, allows the system administrator to configure _Scrippy_.

When multiple configuration files exist within the same system, the configuration files are merged so that a user can override a default configuration without being able to override a configuration set up by the system administrator.

The _Scrippy_ configuration file must define a number of directories that will be useful to scripts based on _Scrippy_.

| Key                   | Purpose                                                                 |
| --------------------- | ----------------------------------------------------------------------- |
| `env.logdir`         | Directory of execution logs of scripts based on _Scrippy_              |
| `env.histdir`        | Directory of execution history files                                   |
| `env.reportdir`      | Directory of report files                                              |
| `env.tmpdir`         | Temporary directory for scripts based on _Scrippy_                     |
| `env.templatedir`    | Directory of template files                                            |
| `env.confdir`        | Directory of configuration files for scripts based on _Scrippy_        |
| `env.datadir`        | Directory of data used by scripts based on _Scrippy_                   |

Model of _Scrippy_ execution environment configuration file:

```yaml
env:
  logdir: /var/log/scrippy
  histdir: /var/scrippy/hist
  reportdir: /var/scrippy/reports
  tmpdir: /var/tmp/scrippy
  datadir: /var/scrippy/data
  templatedir: /var/scrippy/templates
  confdir: /usr/local/etc/scrippy/conf
```

#### Scripts executed with elevated privileges (i.e: `sudo`)

Configuration files are merged **key by key**: each file only overrides the keys it defines, and the system administrator configuration file always has the last word on the keys it defines.

When a script is allowed to be executed with elevated privileges, it is up to the system administrator to ensure that the user executing the script cannot alter its environment:

- The system administrator configuration file (`/usr/local/etc/scrippy/scrippy.yml`) should define **all** the `env` keys. Any key left undefined may be provided by the user configuration file.
- The `HOME` environment variable should not be preserved (i.e: no `sudo -E`, no `env_keep` of `HOME`, `Defaults always_set_home` in _sudoers_). Otherwise, the user configuration file (`~/.config/scrippy/scrippy.yml`) of the calling user is read by the privileged script.
- The directories defined in the `env` section should not be writable by other users than the ones allowed to execute the scripts.

##### Creation of the directories

`logdir`, `histdir` and `tmpdir` are automatically created at the first execution of a script. The other directories (`confdir`, `reportdir`, `datadir` and `templatedir`) must be created beforehand.

Python script to create all the directories defined in a _Scrippy_ configuration file (set `conf_file` to the configuration file actually used):

```python
import os
import yaml

conf_file = "/usr/local/etc/scrippy/scrippy.yml"
with open(conf_file, "r") as conf:
  scrippy_conf = yaml.safe_load(conf)
  for rep in scrippy_conf["env"]:
    os.makedirs(scrippy_conf["env"][rep], 0o0775, exist_ok=True)
```

---

## Formalism

Scripts using the `scrippy_core` module must follow a certain formalism in order to ensure standardization of their format while facilitating the implementation of certain advanced functionalities such as the control of the validity of the configuration or the management of optional parameters.

Thus each script must include in its [_doc string_](https://www.python.org/dev/peps/pep-0257/) a declarative header and a set of predefined _sections_.


### Basic model

The basic model below can be used as a code snippet:

```python
#!/bin/env python3
"""
--------------------------------------------------------------------------------
  @author         : <Author>
  @date           : <Date of current script version>
  @version        : <X.Y.Z>
  @description    : <Brief description (one line) of the utility of the script>

--------------------------------------------------------------------------------
  Update:
  <X.Y.Z>  <Date> - <Author> - <Reason>: <Description of update>

--------------------------------------------------------------------------------
  List of authorized users or group:
    @user:<username>
    @group:<group name>

--------------------------------------------------------------------------------
  Concurrent executions:
    @max_instance: <INT>
    @timeout: <INT>
    @exit_on_wait: <BOOL>
    @exit_on_timeout: <BOOL>

--------------------------------------------------------------------------------
  List of mandatory configuration parameters:
    @conf:<section>|<key>|<type>|<secret>

--------------------------------------------------------------------------------
  List of execution options and argument parameters:
    @args:<name>|<type>|<help>|<number of arguments>|<default value>|<conflicts>|<implies>|<required>|<secret>

--------------------------------------------------------------------------------
  Functioning:
  ---------------
    <Detailed description of the script>
...
"""
#-------------------------------------------------------------------------------
#  Initialization of the environment
#-------------------------------------------------------------------------------
import scrippy_core
from scrippy_core import logger


#-------------------------------------------------------------------------------
#  Definition of functions and classes
#-------------------------------------------------------------------------------
class <Class>(<object>):
  def __init__(self, [<param>, ...]):
  [...]

def <function>([<param>, ...]):
  [...]

#-------------------------------------------------------------------------------
#  Main processing
#-------------------------------------------------------------------------------

def main():
  with scrippy_core.ScriptContext() as _context:
    # Retrieve config if necessary
    config = _context.config
    args = _context.args

    [...]

#-------------------------------------------------------------------------------
#  Entry point
#-------------------------------------------------------------------------------

if __name__ == '__main__':
  main()
```

A `main()` function **must**:

- Be defined in the `Main processing` section
- Be called in the `Entry point` section
- Directly enclose its contents by `with scrippy_core.ScriptContext() as _context:` which manages and activates all the functionalities of scripts written from the `scrippy_core` module.

```python
def main():
  with scrippy_core.ScriptContext() as _context:
    # Script environement is accessed through `_context.env`
    # Script arguments are accessed through `_context.args`
    # Script configuration are accessed through `_context.config`

if __name__ == '__main__':
  main()
```

### Header elements

The `author`, `date`, `version`, `description` elements are **mandatory** and will be automatically displayed by the `--help` option.

#### Version

A script's version number is in the format X.Y.Z with:

- `X`, the major version identifier
- `Y` is the minor version identifier
- `Z`, the correction version identifier

**Major version X**: It is worth "0" during development, the script is considered invalid and should not be called by other scripts or used in production.

Once the script is validated, the version should be 1.0.0 (first stable version).

`X` should be incremented if changes in the code no longer guarantee backward compatibility.

The minor version identifiers `Y` and the correction version identifiers `Z` must be reset to zero when the major version identifier `X` is incremented.

**Minor version Y**: Must be incremented when adding new functionalities or improving the code that has no impact on backward compatibility.

The correction version identifier `Z` must be reset to zero when the minor version identifier is incremented.

**Correction version Z**: Must be incremented if only retro-compatible corrections are introduced.

A correction is defined as an internal change that corrects incorrect behavior (Bug).

`Z` can be incremented for typographical or grammatical correction.


#### Update:

In addition to the version number, modification date, and modification author, each line of the script history must indicate the reason for the modification.

`<Reason>` can take one of the following values:

- `cre`: Script creation
- `evo`: Script evolution (adding functionality, improving code, etc.)
- `bugfix`: Correction of unexpected behavior (bug)
- `typo`: Typo correction, adding comments, and any action that does not modify the code.

## Authorized Users and Groups

The `scrippy_core` module adds a verification layer to the script execution to ensure that a script is executed by a specific user or a user belonging to a specific group.

Placed in the header, a line such as `@user:harry.fink` will prevent execution by any user other than `harry.fink`.

It is possible to define several authorized users by multiplying the declarations:

```
@user:harry.fink
@user:luiggi.vercotti
```

It is also possible to authorize groups of users with a line such as `@group:monty` which ensures that only a user from the `monty` group executes the script.

In the same way as for users, it is possible to multiply group declarations:

```
@group:monty
@group:python
```

In the absence of `@user` and `@group` declarative lines, file permissions take precedence, and in all cases, file permissions have priority.

**Attention:** If a group and a user are declared, **both conditions must be met** for the user to be authorized to execute the script.

## Management of Concurrent Executions

The optional declarations `@max_instance`, `@timeout`, `@exit_on_wait`, and `@exit_on_timeout` make it possible to define the number of concurrent executions of the same script as well as the behavior of the script if necessary:

| Declaration        | Type                        | Utility                                                                                                          | Default value |
| ------------------ | --------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------- |
| `@max_instance`    | Integer                     | Maximum number of parallel executions                                                                            | 0 (infinite)  |
| `@timeout`         | Integer                     | Maximum waiting time expressed in seconds if `@exit_on_timeout` is set to true. Ignored if `@exit_on_timeout` is set to false: the script waits indefinitely for an execution slot | 0             |
| `@exit_on_timeout` | Boolean (`true`, `1`, `on`) | Exits the script with an error when the waiting time is reached                                                  | False         |
| `@exit_on_wait`    | Boolean (`true`, `1`, `on`) | Immediately exits the script with an error in case of waiting, even if the waiting time is not reached           | False         |

The waiting occurrences are executed sequentially in the order in which they are recorded in the execution queue.

The availability of an execution slot is checked every 3 seconds: the actual waiting time is the largest multiple of 3 lower than `@timeout` (i.e: 9 seconds for `@timeout: 10`, no waiting at all for `@timeout` lower than or equal to 3).

In the following example, two occurrences of the script are allowed. A third execution will be put on hold for 9 seconds, after which the script will exit with an error if it cannot obtain a execution slot.

```
  @max_instance: 2
  @timeout: 10
  @exit_on_wait: false
  @exit_on_timeout: true
```

## Management and Verification of Mandatory Configuration Parameters

Any script may have an optional configuration file.

This configuration file will be automatically loaded at script start if:
- located in the directory defined by the `env::confdir` variable set by the [**Scrippy** main configuration file](#environment-configuration-details)
- has the same name as the script that must load it, stripped of its extension and suffixed with the `.conf` extension.

This way the script `exp_test_logs.py` will automatically load the configuration file `exp_test_logs.conf`.

A script configuration file is a simple _ini_ file divided into as many sections as necessary:

```ini
[log]
  level = ERROR
[database]
  host = srv.flying.circus
  port = 5432
  user = arthur.pewtey
  base = ministry_of_silly_walks
  password = parrot
# The section name may hashave spaces
[my section]
  my variable = my value
```

In such a file:

- Indentation is possible but optional
- A line beginning with `#` or `;` is considered a comment, even if it is indented.
- All values are strings:
  - It is up to the developer to convert the variable value to the desired type during processing (see [**_Retrieving a Value of a Particular Type_**](#retrieving-a-value-of-a-particular-type)).
  - Spaces are accepted in the section name, key name, or value.


### Validating the configuration file

In order to validate the configuration file, the script must contain a set of lines starting with `@conf` and describing the configuration file in its [_docstring_](https://www.python.org/dev/peps/pep-0257/).

The declaration lines of the configuration format must respect the following format:

```
@conf:<section>|<key>|<value_type>|<secret>
```

All fields are **mandatory** and can not be empty.

`<value_type>` must be one of the following recognized types:

- `str` (string of characters)
- `int` (integer)
- `float` (floating point number)
- `bool` (boolean)

`<secret>` must have the value `true` or `false`.

If `<secret>` has the value `true`, the configuration parameter value will be considered a _secret_ and will be redacted in the logging files.

Any invalid declaration (missing, empty or extra field, unknown type, invalid `<secret>` value, duplicate declaration) is reported with the faulty declaration and the script exits with an error.

Example:

From the following declaration:

```
@conf:log|level|str|false
@conf:database|port|int|false
@conf:sql|verbose|bool|false
@conf:sql|database|str|false
@conf:sql|password|str|true
```

The following configuration file will be checked:

```
[log]
  level = error
[database]
  port = 5432
[sql]
  verbose = True
  database = monty
  password = d34dp4rr0t
```

All occurrences of the password `d34dp4rr0t` will be replaced by `*******` in the logging files and on the standard output.

No control over the values of the parameters is performed, so there is no need to indicate them. Only the structure of the configuration and the type of the keys are checked.

When the `--debug` option is used, the value of each key declared with `@conf` is displayed on the standard output and in the log during the configuration check (secret values are redacted). Setting the log level to `debug` in the configuration file does not display them, as this log level is applied after the configuration check.

```python
"""
@conf:database|port|int|false
@conf:database|base|str|false
@conf:database|host|str|false
@conf:database|password|str|true
@conf:database|user|str|false
@conf:local|dir|str|false
@conf:src|port|int|false
@conf:src|host|str|false
@conf:src|dir|str|false
@conf:src|user|str|false
@conf:dst|port|int|false
@conf:dst|host|str|false
@conf:dst|dir|str|false
@conf:dst|user|str|false
"""
import scrippy_core
from scrippy_core import logger

with scrippy_core.ScriptContext() as _context:
  # retrieving the configuration
  config = _context.config
```

If one of the sections or keys described by `@conf` is absent from the configuration file or if the type described for a key does not match the type found in the configuration file for that key, a critical error is raised and the script immediately exits with an error.

Extra sections or keys found in the configuration file that are not declared will simply be ignored during validation but will remain available to the script.

Thus in the above example, the `[mail]` section not being defined in `@conf` neither its presence nor its validity will be checked.


### Retrieving a configuration parameter value

Retrieving a parameter value from the configuration file is done through the `_context.config.get()` method.

```python
"""
@conf:database|port|int|false
@conf:database|base|str|false
@conf:database|host|str|false
@conf:database|password|str|true
@conf:database|user|str|false
"""
import logging
import scrippy_core
from scrippy_core import logger

with scrippy_core.ScriptContext() as _context:
  config = _context.config
  logger.info(config.get('database', 'host'))
```

In the above example, the value of the `host` key in the `database` section will be displayed on the screen.

If the section or key does not exist, an error is raised and **the script will immediately raise a critical error**.


#### Retrieving a value of a particular type

Unless the 'param_type' parameter is set to one of the authorized values (`str` (default), `int`, `float` or `bool`), the returned type is always a string.

Calling the `Config.get()` method with the wrong type will raise an error and **the script will immediately raise a critical error**.

```python
"""
@conf:log|level|str|false
@conf:database|port|int|false
@conf:database|base|str|false
@conf:database|host|str|false
@conf:database|password|str|true
@conf:database|user|str|false
"""
import scrippy_core
from scrippy_core import logger

with scrippy_core.ScriptContext() as _context:
  config = _context.config
  logger.info(config.get('database', 'port', 'int'))
```


### Reserved Sections and Keys:

The following key is automatically read and applied when entering the script context (`with scrippy_core.ScriptContext() as _context:`).

This configuration key is optional, just like the configuration file.

- Log level:

```ini
[log]
  level = <str>
```

More details in the [**Execution log management**](#execution-log-management) section.

### Examples

All examples in this documentation are based on all or part of the following configuration file:

```ini
[log]
  level = info
[database]
  host = srv.flying.circus
  port = 5432
  user = arthur.pewtey
  base = ministry_of_silly_walks
  password = dead parrot
[local]
  dir = /tmp/transfert
[src]
  host = srv.source.circus
  port = 22
  user = harry.fink
  dir = /home/harry.fink/data
[dst]
  host = srv.destination.circus
  port = 22
  user = luigi.vercotti
  dir = /home/luigi.vercotti/received
[mail]
  host = srv.mail.circus
  port = 25
  from = Luiggi Vercotti
  from_addr = luiggi.vercotti@circus.com
  to = Harry Fink
  to_addr = harry.fink@circus.com
  subject =  Execution Report
```

## Execution options management

Script options can be managed through declarations in the [_docstring_](https://www.python.org/dev/peps/pep-0257/).

The declaration of an option is composed of 9 fields:

```
@args:<name>|<type>|<help>|<number of arguments>|<default value>|<conflicts>|<implies>|<required>|<secret>
```

with:

- **name**: The name of the option (required). Allowed characters are letters, digits, `-` and `_`. The value of an option named `my-option` is available as `args.my_option`.
- **type**: One of the following values: `str`, `int`, `float`, `choice`, `bool` (required). If the type is `choice`, the list of possible choices must be entered in the **default value** field as a list of words separated by commas.
- **help**: A string summarising the purpose of this option (required).
- **number of arguments**: The number of arguments taken by the option. This field is required for all types except `bool`, where the declared number of arguments is ignored. Its value is usually an integer greater than or equal to 1 but can take the value `+` when the number of arguments is greater than 1 but is not known in advance, or `?` when the number of arguments can be equal to zero. For `choice` options, each value must be one of the accepted choices.
- **default value**: The default value of the option (optional). For `choice` options, this field is required and contains the list of accepted choices, not a default value: a `choice` option that is not used on the command line is `None`. With several arguments, default values are separated by commas. For `bool` options, the default value is `false` when not set: the option is then `true` when used. A `bool` option whose default value is `true` is `false` when used.
- **conflicts**: The list of options that conflict with the current option (optional, list of options separated by commas).
- **implies**: The list of options induced by the current option (optional, list of options separated by commas).
- **required**: A Boolean (`true` or `false`) indicating whether the option is mandatory or not (required).
- **secret**: A Boolean (`true` or `false`) indicating whether the option value should be kept secret (required). A `bool` option can not be secret.

Any invalid declaration (missing, empty or extra field, unknown type, invalid value, duplicate option, unknown option in **conflicts** or **implies**...) is reported with the faulty declaration and the script exits with an error.

The value of an option whose _secret_ attribute is set to `true` will be automatically masked in log files as well as in standard output.

The declaration of the options will automatically generate the help of the script accessible with the `--help` option.

The following options will also be automatically generated:

- `--version`: Displays the script version number from the information contained in the header.
- `--source-code`: Displays the source code of the script.
- `--hist [NB_EXECUTION (default:10)]`: Displays the execution history of the script.
- `--log [SESSION]`: Displays the content of the log file corresponding to the session whose identifier is passed as an argument. By default, the latest session is displayed.
- `--debug`: Forces the logging level to DEBUG (Changes in log level during execution are then ignored).
- `--no-stdout`: Disable log output to standard output except for *error* and *critical* messages.
- `--no-log-file`: Prevents the creation of log and history files.

The `--help`, `--version`, `--source-code`, `--hist`, `--log`, `--debug`, `--no-stdout` and `--no-log-file` options are reserved: declaring them is an error.

### Examples

The following script can be called with the following options and arguments:

- `--env`: Mandatory. Accepts one of the following values: qualif, preprod or prod
- `--appname`: Optional. A free string with a default value of "aviva"
- `--now`: Boolean option. If used, its value will be _true_ (false by default) and the `--date` option cannot be used.
- `--date`: A sequence of 3 integers from which a date will be created (i.e. 24 02 2019). This option conflicts with the `--now` option.
- `--email`: A free string. This option is mandatory if the `--date` option is used.

```python
#!/bin/env python3
"""
--------------------------------------------------------------------------------
  @author         : Florent Chevalier
  @date           : 2019-07-27
  @version        : 1.0.0
  @description    : test arguments

--------------------------------------------------------------------------------
  Update :
  1.0.0  2019-07-27   - Florent Chevalier   - Cre : Production
--------------------------------------------------------------------------------
    @args:env|choice|Target environment|1|qualif,preprod,prod|||true|false
    @args:appname|str|Application name|1|aviva|||false|false
    @args:date|int|Planning date(day, month, year)|3||now|email|false|false
    @args:email|str|Notification email address|1||||false|false
    @args:now|bool|Apply immediately||false|date||false|false
--------------------------------------------------------------------------------
"""
import datetime
import scrippy_core
from scrippy_core import logger

def print_contact(email):
  logger.info(" - Contact: {}".format(email))

def print_appname(appname):
  logger.info(" - Application: {}".format(appname))

def print_date(date):
  logger.info(" - Date: {}".format(date))

def main():
  with scrippy_core.ScriptContext() as _context:
    args = _context.args
    date = "not planned"
    if args.date:
      date = "{}/{}/{}".format(args.date[0], args.date[1], args.date[2])
    if args.now:
      date = datetime.datetime.now().strftime('%d/%m/%Y')
    logger.info("[+] Report:")
    logger.info(" - Environment: {}".format(args.env))
    print_date(date)
    print_appname(args.appname)
    print_contact(args.email)

if __name__ == '__main__':
  main()
```


## Execution log management

Execution logging is done using the `scrippy_core.logger` object.

Two types of logs are simultaneously available:

- **Standard output**: Display towards `sys.stdout`, colored when the standard output is a terminal whose `TERM` environment variable is one of `linux`, `xterm`, `xterm-256color`, `screen`, `screen-256color`, `tmux` or `tmux-256color`
- **Log file**: A file located in log directory defined by the `env::logdir` variable set by the [**Scrippy** main configuration file](#environment-configuration-details).

The log file is located in a subdirectory named after the script: `<logdir>/<script_name>/<script_name>_<timestamp>_<pid>.log`.

If a configuration file exists for the script and it contains a `[log]` section indicating a log level with the `level` key, then the indicated log level is automatically applied.

The `--no-log-file` option disables the log file (see below).


### Defining the log level through the configuration file:

```
[log]
  level = warning
```

The value of the log level in the configuration file is case-insensitive.

The available log levels, from least verbose to most verbose, are [the logging levels of the standard logging module](https://docs.python.org/3/library/logging.html#logging-levels)

- `critical`
- `error`
- `warning`
- `info`
- `debug`

Note that the `debug` log level displays details that could prove to be a source of information leakage. It is not recommended to use it in production.

All scripts written using the `scrippy_core` module have the following logging options:

- `--no-log-file`: When this option is used, no execution log is recorded on disk. This option does not prevent output on the screen.
- `--debug`: When this option is used, the log level is forced to `DEBUG`. In this case, the script does not take into account a possible configuration parameter indicating the opposite.

These two options can be combined. An invalid log level in the configuration file is reported as an error even when the `--debug` option is used.


## Error management

The **Scrippy** framework is intended to be used in production environments where debug trace or even stack traces should be avoided.

For the above reason, if an exception is not handled and propagates up to the *script context*, a `sys.exit(1)` will be triggered with a brief error message **with no stack trace**.

To display the stack trace, the log level must be set to `debug`.

Please, debug your scripts in testing environment !


## Execution history

The execution history file is located in the history directory defined by the `env::histdir` variable set by the [**Scrippy** main configuration file](#environment-configuration-details)

The execution history file will be created and named `<script_name>.db` at the first script execution.

This file is an _sqlite3_ database that lists all executions of a script.

For each execution the following information are stored:
- session identifier
- Start date of the execution
- End date of the execution
- Duration of the execution
- User who initiated the execution
- User who actually ran the script (in case of _sudo_)
- Return code of the execution (0 if OK, other value if KO)
- Set of parameters passed as arguments to the script
- Name of the error if the execution did not end correctly (0 otherwise)

If the history file already exists at execution time, it will be updated with the parameters of the new execution.

History is automatically enabled by the declaration `with scrippy_core.ScriptContext() as _context:` enclosing the content of the `main()` function.

```python
def main():
  with scrippy_core.ScriptContext() as _context:
    <...>
```

Each script execution is assigned a session allowing each execution to be uniquely identified.

This session is composed of:

- a _timestamp_ representing the execution time
- the process identifier (PID)


Exemple:
  ```txt
  1568975414.6954327_10580
  ```

This session identifier is reported in the `Session` column of the history and allows the corresponding log to be found (See the `--log` option in [**_Execution options management_**](#execution-log-management)).


### Retention

The number of executions kept in the history file is **50** by default.

It is possible to override this value by specifying the desired retention number using the declaration `with scrippy_core.ScriptContext(retention=100) as _context:`

```python
with scrippy_core.ScriptContext(retention=100) as _context:
  config = _context.config
  args = _context.args
  <...>
```

### Displaying execution history

All scripts based on the `scrippy_core` module have an `--hist` option that allows you to display the last executions.

```shell
exp_test_script.py --hist
```

The number of executions to display can be specified by passing an integer parameter to the `--hist` option.

```shell
exp_test_script.py --hist 2
```

For each script execution, the following columns are displayed, from the most recent execution to the oldest:

| Column     | Content                                                                                                     |
| ---------- | ----------------------------------------------------------------------------------------------------------- |
| `Session`  | The unique session identifier (`<timestamp>_<pid>`), to be used with the `--log` option                     |
| `Start`    | The start date of the execution (`DD/MM/YYYY HH:MM:SS`)                                                     |
| `End`      | The end date of the execution (`None` while the execution is running or if it has been killed)              |
| `Duration` | The duration of the execution (i.e: `1 min 12 sec`)                                                         |
| `Origin`   | The user who initiated the execution (the original user in case of _sudo_)                                  |
| `Exec`     | The user who actually ran the script (i.e: `root` in case of _sudo_)                                        |
| `Code`     | The exit status of the execution (0 if OK, other value if KO)                                               |
| `Params`   | The list of options and arguments passed to the script, secret values being redacted                        |
| `Exit`     | The name of the error if the execution did not end correctly (i.e: `ValueError`, `SystemExit`), 0 otherwise |

## Temporary workspace

The declaration `with scrippy_core.ScriptContext() as _context:` automatically creates a temporary workspace that will **automatically be destroyed with its contents at the end of the script**.

The workspace is located in the temporary directory defined by the `env::tmpdir` variable set by the [**Scrippy** main configuration file].

The exact path to this workspace can be obtained using the `workspace_path` attribute of the execution context `_context`.

```python
with scrippy_core.ScriptContext() as _context:
  workspace_path = _context.workspace_path
  ...
```

In the previous example, the `workspace_path` variable will contain the path to the temporary working directory, whose name will be constructed as follows: `<tmpdir>/<SCRIPT NAME>_<SESSION ID>`

Exemple:

```bash
/var/tmp/scrippy/exp_transfert_ftp_1574391960.6696503_102257
```

This temporary workspace, which will be **automatically destroyed with its contents at the end of the script**, is a directory that can be used to create temporary files used by the script.

```python
#!/bin/env python3
import os
import scrippy_core
from scrippy_core import logger

def create_file(workspace_path):
  tmp_file = "fichier.tmp"
  logger.info("[+] Creating temporary file: {}".format(os.path.join(workspace_path, tmp_file)))
  with open(os.path.join(workspace_path, tmp_file), 'a') as tmpfile:
    logger.info("[+] Writing in temporary file")
    tmpfile.write("Nobody expects the Spanish inquisition !")

def main():
  with scrippy_core.ScriptContext() as _context:
    config = _context.config
    create_file(_context.workspace_path)

if __name__ == '__main__':
  main()
```

## Accessing the environment

The environment is accessible through the `_context.env` dictionary.

It contains the directories defined in the `env` section of the [**Scrippy** main configuration file](#environment-configuration-details) (`logdir`, `histdir`, `reportdir`, `tmpdir`, `datadir`, `templatedir`, `confdir`).

```python
#!/bin/env python3
import os
import scrippy_core
from scrippy_core import logger

def create_report(reportdir, filename):
  report_filename = os.path.join(reportdir, filename)
  logger.info(f"[+] Creating report file: {report_filename}")
  with open(report_filename, 'a') as report:
    logger.info("[+] Writing report file")
    report.write("Nobody expects the Spanish inquisition !")

def main():
  with scrippy_core.ScriptContext() as _context:
    config = _context.config
    create_report(_context.env.get("reportdir"), "inquisition.txt")

if __name__ == '__main__':
  main()
```

## Tips and guidelines

- A log is **never too verbose**
- **Priority should be given to readability** and maintainability rather than compactness and technicality
- Break down the code into **small functions**
- **Each function should log its entry point** and where possible, the parameters it receives
- **Maximize variable naming** and transfer as many variables as possible to the configuration file or run-time options (no hard coded values)
- **Simplify the main algorithm** of the program to its simplest expression
- **Handle errors as finely as possible** and as close as possible
- **Avoid global variables at all cost**
- **Minimize information leaks** in log files by using the `secret` attributes of options and arguments whose values are sensitive information (login, password, connection server, etc.)
- **Create a `main()` function** containing the script's main algorithm
- Create the environment (objects and variables used at the global level) in the `Entry point` section.

## Additional modules

The _Scrippy_ framework, of which `scrippy-core` is the core, has modules that facilitate the writing of advanced _Python_ scripts in accordance with Scrippy's basic principles.

| Module             | Utility                                                       |
| ------------------ | ------------------------------------------------------------- |
| `scrippy-template` | Template file management (based on _Jinja2_)                  |
| `scrippy-remote`   | Implementation of _SSH/SFTP_ and _FTP_ protocols               |
| `scrippy-mail`     | Implementation of _SMTP_, _POP_ and _Spamassassin_ protocols   |
| `scrippy-git`      | Git repository management                                     |
| `scrippy-db`       | Database management (_PostgreSQL_ and _Oracle_)                |
| `scrippy-api`      | Using _REST API_ (based on _requests_)                         |
| `scrippy-snmp`     | Basic _SNMP_ capabilities                                     |

All available modules are available at [**https://codeberg.org/scrippy**](https://codeberg.org/scrippy)
